January 9, 2019

How to Implement Custom Image Recognition Into E-commerce Apps

Table of Contents:

Learn how to implement image classification for e-commerce apps in a few steps. Thanks to Nick Engmann for this Clarifai feature tutorial!

Computers are doing just about everything these days, from powering backflipping robots to emulating the human voice. Given these technological advances, it should not surprise us that image recognition APIs can tell the minute differences between a ‘Pink Lady’ and a ‘Red Delicious’ apple. By leveraging the power of Clarifai’s image classification tools you can train your apps to recognize just about anything. Welcome to the world of super-powered computer vision hacks!

Clarifai has already trained its general image recognition model to classify over 11,000 concepts including streets, vehicles, and foliage. However, using the power of Clarifai’s Custom Training platform users can create their own custom models and concepts.

Recently, a team at HackMerced demoed their winning application SnapShop, leveraging the power of Clarifai’s general image classification model. Their application allows users to upload items to a virtual storefront and automatically have items labeled with the topics Clarifai finds. 

null

 We loved their application but noticed that the tags that come back from the general model are, as expected, general. We wanted to use Clarifai’s Custom Training platform to see if we could make the results more robust. Our goal is to let users upload laptops and have the Custom Training platform label with its brand.

Sign up for your free Clarifai account here!

 

Getting Started with SnapShop’s Code

For this tutorial, we are going to build on top of SnapShop’s codebase but change it so that we can identify specific laptops by using Clarifai’s Custom Training platform. So first let's go ahead and download both SnapShop’s GitLab backend repo and frontend repo. You can follow their readmes for details on how to deploy the application.

Setting Up the Frontend

null

 

Setting Up the Backend

null

 

Setting up Clarifai

Next, we’ll need to set up a Clarifai account. You can follow the direction here to create a new application. Select ‘General’ as your basic workflow and type ‘laptops’ as your App name.

null

Next, you’ll want to click on ‘API Keys’. You’ll find your information in the dropdown below. Make sure to take down your API Key, and don’t share this string with anyone else! It will be used to verify that you’re the developer accessing the API.

Add Your Details to SnapShop’s Code

First, we’ll need to fill in our Clarifai API credentials in Snapshop’s backend code. Take the API key that you copied from the Clarifai application and paste it over the YOUR_API_KEY placeholder inside the tags.py file.

Filepath: hackmerced-shop-server/tags.py

Change the frontend Sell.js file to point at your local backend:

Filepath:hackmerced-shop-client/src/components/routes/Sell.js

Let’s Test Things Out!

Now that our own credentials are in place, we can run SnapShop’s code locally.

On the client side run:

$ npm start
null

 

On the server side run:

  $ python snap.py 
null

 

Let’s take a picture of a Dell-XPS and upload it to SnapShop.

null

 

Not too shabby! The general Clarifai model classified the Dell-XPS image we gave it as “Internet”, “Computer”, “Laptop”, “Screen”, and “Technology”. Now it’s time to leverage the power of Clarifai’s Custom Training model to see if we can get SnapShop to classify this same image as a Dell-XPS.

null

 

Add Images to Your Clarifai Application

Custom models are built by training on your own data. To help our application identify our custom concepts let’s upload pictures of the concept we want it to recognize. You can either upload image files to your Clarifai application or you can drag and drop image URLs to the application.

Here is a handy Github gist for you that has a list of images of a Dell-XPS, not including our previous test image. Let’s copy and paste these links in!

null

 

Adding a Concept

Now that the images are inside the web application we need to tell Clarifai what they are. In the browse screen, we can select these recently uploaded images and hit the add concept button and give these images the label “dell-xps”.

null

 

Adding Negative Concepts

To make a robust concept it’s important to tell our model what is NOT a Dell-XPS as well. Luckily this is a super simple process! To start, go ahead and copy the links from this gist of examples of Dell-Inspirons. And just like before we are going to add these image URLs to Clarifai.

null

 

Finally, similar to the positive examples, highlight the negative examples and hit add concept. This time, though, we double-click the dell-xps concept to show the cancel icon.

null

 

Training Our Model Based on Our Photos

Now you are a few clicks away from properly training the model. Go ahead and click on the ellipses icon beside the ‘laptops’ model and click ‘Train Model.’ You will know when training is finished by the dialog on the far right screen displaying “Model trained successfully.”

null

 

Pulling it all together!

Next let’s configure the backend to use both the general model you were using before and this new custom laptops model we are developing. Copy the code below and replace your tags.py file in your SnapShop backend. Remember to include your API key!

Go ahead and paste in your API key on lines 5 and 9. Then you have to add your model version id in on line 17. This way the model knows which version to use during the backend call. You can find this model version id on the models page on the right side.

null

 

Testing

Time to test things out! Go ahead and fire up the backend and the frontend. Then upload the picture of the Dell-XPS laptop used before and boom-bam-pow specific image characterization.

null

 

Now that you know how to use Clarifai’s custom training platform, you can add more images and concepts to your model to make it even more versatile. Check out Clarifai’s Custom Training Overview for a guide on how to further expand your image classification applications!

New call-to-action