Machine Learning with Microsoft CustomVision

By Matt Tank

To complement my introduction to Machine Learning from 2017, I thought I’d try something different. Late last year, I gave a presentation at my company about Microsoft Custom Vision. I thought I’d outline that presentation here, and show the real-life side of how a business might use machine learning to help you as a customer, or to improve some of their internal processes.

Note that while Microsoft’s product is featured here, there are other technologies and services in the market that can perform similar tasks. This isn’t an endorsement of Custom Vision, and if you are looking to introduce this capability into your business, you should evaluate a range of products against your requirements and budget.

First off, what is Custom Vision? Part of the Microsoft Cognitive Services suite of Cloud-based AI products, its job it to pull useful information out of images. It developed out of another product called Computer Vision (which you can try out by clicking the link). To properly put these two products in perspective, I will quickly explain Computer Vision.

Computer Vision is a ready-to-use image classification service. You can upload an image, and it will tell you what’s in it (with varying degrees of success, as you can see below):


It can also give you a description of the image. In this case, something along the lines of “A dog with a ball getting chased by a panda(!?) in the ocean”. This technology has a wide range of uses. You’ve probably already seen it in use, like in your smartphone’s photo app.

This is fine if you want to identify something generally, but what if IDs like “Dog” or “Ball” aren’t specific enough? Computer Vision doesn’t go more detailed than that, so you can’t identify the “soccer ball” for example. This is where Custom Vision comes in.

Custom Vision is your own, fully-customisable, image classifier. It can be used to look at an image, and identify things based on what you teach it to recognise. First up, you log into the Custom Vision Console (using a Microsoft or Office 365 account) and create a Project. After that, you create tags (the things you want to recognise), and upload images, assigning one or more tags to each image. For the purpose of my demonstration, I chose birds:


Here, I’ve uploaded a number of images of a kookaburra, with a wide variety of angles and positions, then assigned the “Kookaburra” tag to them. Keep in mind that Computer Vision, and other technologies like it, can’t ID a kookaburra, just a “bird”, so this is why we have to use Custom Vision. This step is all about teaching the system what a kookaburra looks like.

Once we’ve added an acceptable number of images for each bird (each tag you create needs at least 5 images, but the more the better), we can click the Train button and the system analyzes everything we have uploaded, and creates its identification model.

Next up is to test it. You can run a test straight from Custom Vision itself, using the Quick Test button. For this we pick a type of bird that we’ve previously tagged, but make sure we use a new image (one we didn’t use as part of the training process):


As you can see on the right, Custom Vision recognised the grace and beauty of the bird in this image, and is 99.9% sure it’s the Australian White Ibis (or maybe it just recognised the bin).

The more images you train the service with, the better the it can pick up subtle differences and make the correct ID. However, it can still make mistakes, especially if the objects you are trying to distinguish between are very similar (such as the Australian Magpie, vs the Murray Magpie, below):


(For what it’s worth, Custom Vision still managed to pick the right bird on both occasions with about 85% confidence).

Once you’re happy enough with the result, you can build the service into a mobile app, website, or something similar, and start to use it. The best part? Your classification model can continue to improve. Every image that is submitted to the service is available to be tagged, and then you can retrain your model. You could do this manually, or you could build it into your application. For example, one application user could upload a blurry image of a magpie to the service, which Custom Vision misidentifies as a kookaburra. Along comes another application user, who happens to be a magpie expert. He corrects the identification, and in the back end, the application correctly tags the image, which is incorporated into the model in the next training run.

A good example of exactly this is the iNaturalist website and app, where a user uploads a plant or animal image, and the “community” identifies it. Every confirmed (“Research Grade”) identification feeds into their machine learning model. You can see a demo of their machine learning technology (which I need to stress is not based on Microsoft Custom Vision) at

So how is all this useful? Here are some examples:

  • Quality Control – for example, a fruit producer could have photos taken of their fruit on the production line. A Custom Vision project could be trained to identify bad (diseased, misshapen, unripe) fruit, and they could be automatically discarded later down the line.
  • Product support – Let’s say Lego wants to make it easy to order specific replacement bricks. You take a photo of the brick you want to order more of. Custom Vision recognises it based on size, colour and shape, and redirects you to the online ordering page for that exact brick.
  • Reference – You’re out fishing somewhere, and need to know whether you can keep your catch. Your local fisheries department could develop an app where you take a photo of your catch and upload it. Custom Vision, pre-trained with images of local species, will identify your fish, and feed back size limits, take limits, and so on, as below:


All of these examples use specialised classifications, which are not available in generic services like Computer Vision, so they are great candidates for Custom Vision.

It’s surprisingly easy to jump in and get started with this service, so if you’re interested, grab your Microsoft (Hotmail) account details and log into [EDIT] You no longer have the option to create a standalone project. It must be connected to a Microsoft Azure subscription. See the comments section for advice on setting this up for free.

You will be able to play around without having a lot of technical knowledge. However, if you’re looking to take the next step, and integrate Custom Vision (or any of the other Cognitive Services products) into your business systems, by all means get in touch.

(Full-resolution version of title image © Steve Doyle. All other images are the property of their respective copyright holders. Please contact me for image attribution)


One thought on “Machine Learning with Microsoft CustomVision

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s