Object Recognition Robot

Object Sense has learned to recognise 80 kinds of everyday thing, from cups, bottles and bananas to books, scissors and teddy bears. In this project you hold something up to your computer's webcam, Object Sense names it, and MIEO scrolls the name across its display while the sprite on the stage says it. The recognising happens on your computer, with no internet. You'll also find out why AI sometimes gets things wrong, and how to help it.

• What object recognition is: naming the things in a picture
• Why an AI is only sure about the kinds of thing it has learned
• Remembering the last answer, so MIEO only speaks up when something changes
• Sending text from the computer to MIEO's display

• Mieo board
• Battery
• 3D-printed mountings
• Screws and screwdriver
• A webcam; most laptops have one built in
• Some everyday objects: a cup, a banana, a book, a bottle
• A USB cable, or Bluetooth on your computer
• The Robolab app (free)
• A webcam (most laptops have one built in)

1. Fit the battery to the Mieo board if it isn't there already, and switch MIEO on.
2. Connect MIEO to your computer with a USB cable or over Bluetooth, and stand it beside the screen.
3. Gather a few objects to test, and sit facing the webcam in a well-lit room.

Want MIEO Bot to guide you?

Robolab's built-in tutor has a lesson on the same idea, “Treasure Detector”. Open MIEO Bot in Robolab, pick Explorer, then Mieo, and it shows you the blocks one at a time and checks your work as you go.

About MIEO Bot's lessons

1. Open Robolab, choose Mieo, connect and set the switch at the top right to Arena. Add the Object Sense extension, and make two variables, thing and last.
2. Start with “when green flag clicked”, “turn camera on with 0 % see-through”, “show boxes around objects”, “set last to nothing” and “show think”.
3. Add a “forever” loop with “look for objects in camera” inside, then “if … then” with “number of objects found > 0”.
4. Inside it, add “set thing to (name of object 1)”, then “if not thing = last then”.
5. Inside that, add “set last to (thing)”, “say (thing)” and “show … with a … at speed fast” with the thing variable in its text gap, so MIEO scrolls the name.
6. Click the green flag, allow the camera, and hold an object up close to it.

Arena mode: the program runs on your computer and talks to MIEO live over USB or Bluetooth, so keep MIEO switched on and connected.

when green flag clicked
turn camera [on v] with (0) % see-through
[show v] boxes around objects
set [last v] to (nothing)
show [think v]
forever
  look for objects in [camera v]
  if <(number of objects found) > (0)> then
    set [thing v] to ([name v] of object [1 v])
    if <not <(thing) = (last)>> then
      set [last v] to (thing)
      say (thing)
      show (thing) with a (#0CF) at speed [fast v]
    end
  end
end

Click the green flag: the camera picture fills the stage and MIEO looks thoughtful. Hold a cup up close and a box appears round it; the Mieo sprite says “cup” and MIEO scrolls “cup” across its display. Swap to a banana and it says and scrolls “banana”. Holding the same thing still doesn't repeat the name; it waits until something new appears.

If it doesn't work

• It keeps saying person? That's you! Hold the object closer, so it fills more of the picture than you do.
• It names the wrong thing? Object Sense only knows 80 kinds of thing. Try a plain, clear object with nothing behind it.
• Nothing happens? Check the camera is allowed in your computer's settings, and that the room isn't too dark.

1. Easy: show a face as well as the name, such as happy for a banana and surprise for scissors.
2. Medium: play a game of “find me a cup”: MIEO plays its win sound only when the thing is cup.
3. Hard: make a list of everything MIEO has seen today, adding each new name to it.