šŸŽ Checkout my Learn React by Making a Game course and get 1 month Free on Skillshare!

TensorflowJs detect multiple objects from an image with COCO-SSD – PART 1

We have seen some while ago how to use a trained TensorflowJs model to recognize the main object from an image.

Today we will do something similar, but with an upgrade. While our initial example was only able to recognize just one single object, in today's example we will use TensorflowJs to recognize multiple objects from a given image.

The final example will also draw a rectangle around the objects it detects. The output will be like in the image below:
TensorflowJs detect multiple objects from an image with COCO-SSD

Given that this example will be a bit more complex, I've decided to split this into two parts. You can check out the second part of this tutorial here, and if you want to skip ahead here is the full code and working example.

You can see in the video below the workflow of the example:

Let's start coding!

Setting up the initial HTML and Javascript

We will start with the below HTML:

<button disabled id="btn">Detect objects</button>
<input disabled id="inp" type='file' accept='image/*' capture='camera'/>
<img id="img" />
<canvas id="can" width="400" height="300">

The scope of the file input is to load the image, and the role of the button is to trigger the TensorflowJs model object identification.

However, you may wonder why we need both an img element and a canvas. The reason for this is that we will need the canvas to draw the rectangles around the identified objects. We can do it directly on the img element as well, but it's easier and more flexible to do it on an HTML canvas.

Please note that we will have a CSS rule that will hide the img element:

img {
  display: none;
}

The initial Javascript contains just the setup for the initial references to the UI elements:

const fileInput = document.querySelector('#inp')
const canvas = document.querySelector('#can')
const button = document.querySelector('#btn')
const ctx = canvas.getContext('2d')
let model = null

For the moment, the TensorflowJs model will be set to null and we will load it during the next step.

Loading the COCO-SSD model in TensorflowJs

The model that we will use today is COCO-SSD, a model trained to identify multiple objects from an image.

It can take as inputs any browser-based image elements (<img>, <canvas>, and even <video>) and returns an array with the bounding boxes for the recognized images.

As usual, we will use Codepen for this example. So, we will need to load both TensorflowJs and COCO-SSD:
load COCO-SSD and TensorflowJs in Codepen

By the way, you can also load them via script tags:

<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs"> </script>
<!-- Load the coco-ssd model. -->
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/coco-ssd"> </script>

With this in place we can now proceed to load the COCO-SSD model. Keep in mind that this is an asynchronous operation:

cocoSsd.load({base: 'mobilenet_v2'}).then(enableFileChooser)

Given that we cannot do anything until the model is fully loaded, both the button and the file chooser will be initially disabled.

So, after the loading is done, we can go and enable the file chooser so that we can upload the image, and instantiate the model:

const enableFileChooser = (m)=> {
  model = m
  fileInput.disabled = false 
}

Uploading the image and drawing it to the canvas

The next step is to upload the image and draw that image to the canvas.

I've written two more in-depth articles about how to upload an image and how to draw an image on an HTML canvas, so I will not go too much into details about this one:

const uploadImg = () => {
  img.src = window.URL.createObjectURL(fileInput.files[0])
  img.onload = ()=> {
    ctx.drawImage(img, 0, 0)
    // will need to enable the detect button here
  }
}

const enableFileChooser = (m)=> {
  model = m
  fileInput.disabled = false
  fileInput.addEventListener('change', uploadImg)  
}

At this point, our app will just upload the image and shows its preview in the canvas.
TensorflowJs detect multiple objects from a given image

In the next part of this tutorial, we will see to use COCO-SSD and TensorflowJs to detect the objects in the image and also how to draw the bounding rectangles around them.

šŸ“– 50 Javascript, React and NextJs Projects

Learn by doing with this FREE ebook! Not sure what to build? Dive in with 50 projects with project briefs and wireframes! Choose from 8 project categories and get started right away.

šŸ“– 50 Javascript, React and NextJs Projects

Learn by doing with this FREE ebook! Not sure what to build? Dive in with 50 projects with project briefs and wireframes! Choose from 8 project categories and get started right away.


Leave a Reply

Your email address will not be published. Required fields are marked *