TensorflowJs detect multiple objects from an image with COCO-SSD – PART 2

DanielApril 6, 2023

By the end of the first part of this tutorial about how to use TensorflowJs to detect multiple objects from a given image we reached the point where we loaded the COCO-SSD model and drew the uploaded image on a canvas.

Next, let's see how to use the model to identify the objects.

Identify the objects from an image with Javascript and TensorflowJs

This is the most important step in our application. Fortunately, it's also very easy to do image object recognition when you have a pertained machine learning model made for this.

If you remember in the initial setup of the app, the Detect objects button was disabled:

In the following code, we will enable the button and also call the detect() method of the model that will trigger the actual object identification:

const drawResults = (predictions)=> {
  console.log(predictions)
}

const detectObjects =  () => {
  model.detect(img).then(drawResults)
}

const uploadImg = () => {
  img.src = window.URL.createObjectURL(fileInput.files[0])
  img.onload = ()=> {
    ctx.drawImage(img, 0, 0)
    enableDetectButton()
  }
}

The COCO-SSD model has an async method named detect() that takes as input any image format (and also video) and returns an array with objects containing data about the identified objects.

For example, an output may look like this:

[
  {bbox: [58.02872180938721, 51.45532786846161, 206.44893646240234, 216.78188145160675],
  class: "person",
  score: 0.9879968762397766},
  {bbox: [239.37225341796875, 125.25517344474792, 137.3770236968994, 144.3488359451294],
  class: "dog",
  score: 0.9753953814506531}
]

It tells us that the model has identified 2 objects in the given image and that it can provide the following info about these objects:

• bbox is an array of 4 values containing the coordinates for the object in the image
• class is the object in the image. You can see here a list of all the objects the COCO-SSD model recognizes.
• score how sure the model is about that prediction. Remember that the way how machine learning works is by building estimations for functions. So, everything becomes a probabilistic game.

Drawing the bounding rectangles for identified objects on the canvas

Let's take again a look at the final flow of this example:

Ar this point we just log the predictions, but we can also draw rectangles around the identified objects.

You may remember that during the initial setup, we added an HTML canvas. This is where it will become very handy.

We have access to the context of this canvas:

const ctx = canvas.getContext('2d')

So, we can use this context to paint the location of the identified objects in the image:

const randomColor = () => `hsl(${~~(360 * Math.random())}, 100%, 40%)`;

const drawResult = ([x, y, w, h], label, confidence) => {
  const colour = randomColor();
  const text = `${label} ${confidence.toFixed(2)}`
  ctx.beginPath();
  ctx.font = '12px Arial';
  ctx.strokeStyle = colour;
  ctx.lineWidth = 2;
  ctx.strokeRect(x, y, w, h);
  ctx.fillStyle = colour;
  const textSize = ctx.measureText(text).width;
  ctx.rect(x + 1, y - 20, textSize + 12, 20);
  ctx.fill();
  ctx.fillStyle = 'white';
  ctx.fillText(text, x + 5, y - 5);
  ctx.closePath();
};

const drawResults = (predictions)=> {
  console.log(predictions)
  predictions.map (
    p => drawResult(p.bbox, p.class, p.score)
  )
}

We iterate through each prediction and using the drawResult() method we add a bounding rectangle around it.

All the positioning data comes from the bbox array. The drawResult() method used a mix of Javascript array destructuring with renaming to avoid a long list of arguments:

// method declaration
const drawResult = ([x, y, w, h], label, confidence) => {}

// method call
drawResult(p.bbox, p.class, p.score)

Each rectangle has a different color generated by the randomColor() method. By the way, this method uses the double tilde ~~ operator Javascript syntax.

And there you have it! The output of our app will look like this:

If you want to take a look at the full code or play with this example here is the link for the codepen.

Happy coding!