Targeted advertisement for Advertima

Advertima, a Computer Vision and Machine Learning company, visually interprets human behavior in the physical world, enabling smart spaces of the future to interact with people in seamless and meaningful ways. Their real-time and easily accessible data stream, called the ‘Human Data Layer’, powers their smart retail solutions, including Autonomous Store and Smart Signage. These next-generation solutions empower companies to create the customer experience of tomorrow: frictionless, relevant and highly-targeted. Advertima is fully compliant with the EU General Data Privacy Regulation (GDPR).

Sarah Kurmulis, Olena Levchun, and Matthieu Bornet were tasked with processing, cleaning, and engineering anonymous metadata from multiple stores and building models using Machine Learning and Deep Learning to understand consumers’ walking paths around the store monitors. Knowing who is currently present in the screen’s display area and paying attention is a prerequisite for showing relevant ads. More precisely, the team predicted whether a given consumer will be present in the display area over the next few seconds. A subsequent improvement allowed the team to predict the consumer’s position within less than a meter deviation. By analysing the head orientation, they were also able to predict whether someone will be looking at the monitor, a proxy for paying attention.

The project’s goal was to investigate improvement opportunities of Advertima’s smart targeting technology. Presently, Advertima extracts several features of the customers walking in front of their screens - amongst them is their age and gender. These are inferred by an advanced real-time computer vision technology. Images are never stored; only the extracted features are kept and properly anonymized. Knowing the age and gender allows for personalization (ex: showing shaving products to adult men or cosmetic products for young women). Since selecting, loading and displaying a concrete advertisement can range between some milliseconds and, in some cases, up to 3 seconds, it is important to know who is going to be in front of the monitor in advance.

To this end, the team analyzed the moving patterns of a large collection of customers. The second by second trajectories of people walking in the viewing areas of monitors from different installations were made available by Advertima. The data also contained other movement-relevant information, such as head position with respect to the screen. From the positional data, the team built a Machine Learning model to predict the position of a customer given his or her past trajectory in the immediate future (1, 2 and 5 seconds). The model performed very well, predicting a customer’s position with an average precision of less than 1 meter.

A subsequent iteration improved the model to also predict the head orientation. In this way, targeted advertisements can be shown not only to people close to the screen, but also to those looking in its direction.

As part of the project the students performed extensive data exploration and feature engineering, which allowed them to also report some insightful visualizations. In particular, they produced maps of hotspots where people tend to walk predominantly. These so-called attention zones can be used to optimize the store layout and the specific position of the screen.

By completing this project, Advertima was supported in creating more accurate analyses for their personalized advertising. Potential next steps for the team of the three bootcamp graduates were defined as follows:
 

Their models reached high levels of accuracy for predicting presence (86% mean accuracy at 5 seconds in the future) and position in stores (8 cm mean position accuracy for a 1s forecast, 112 cm for a 5s forecast), and valuable insights into zones of attention to the screen. These results will enable the company to further improve its targeted ad system.