This blog is a sequel to my first blog on building faAi, an AI system which supposed to assist my wife to choose an outfit to wear. In the previous blog, I shared how I built the Clothes Diary, the first module of faAi which automatically record a snapshot of what my wife wears everyday, tagged with weather condition.
In this blog, I will share how I built the second module, an automatic Clothes Cataloguing system which identify clothes captured in the Clothes Diary. Why do we need to identify clothes, you probably ask? Because, being able to identify a clothing is a very crucial step towards being able to make recommendations.
So, the goal here is quite clear, to build a cataloging system which automatically assigns an ID to each unique outfit seen by the Clothes Diary with additional tagging such as skirt, pants, shorts, vest, etc. As you definitely have more than one of each kind, the final id will be more like skirt-1, skirt-2, pants-1, shorts-1, shorts-2, etc.
In order to uniquely identify clothes, we need to know what makes an outfit different from others. First, the structural shape, e.g. pants is longer than shorts, skirts is short but has a wider bottom, etc. Second, the texture (including colour), e.g. wool, cotton, red, yellow, etc. So, I need a system which can locate all outfits in an image and identify each one by their structural shape and texture.
One way to achieve this is to build a multi-class object detection. However, training an object detection to locate and identify unique skirt, pants and shorts, etc requires massive effort in labelling the training set. You can see that this approach does not scale well as I don’t want to manually label each outfit she owned and every outfit she buys in the future!
Instead, I decided to go with this idea of two stage process. First to use an object detection to locate and identify the structural shape of clothes (skirt, shorts, pants, vest, dress, etc). Next, to use a gram matrix to generate a texture finger print of each detected outfit. The type of cloth returned by the object detection together with its finger print will be passed on to a search module to find a matching outfit (skirt-1, skirt-2, pants-1, etc) from our clothes catalogue if existed, otherwise a new entry will be added. I will explain my…