How data science can improve retail

<h1 id="intro">Intro</h1> The strides in ecommerce represent an entire paradigm shift in retail. Despite only making up around 10% of all retail purchases, ecommerce accounts for more than $2 trillion dollars in sales. Mapping the interactions between the offline and online world seems like an arduous task, but when we focus on each customer and their purchasing paths, it becomes something that can be broken up into a few different paths. We’re going to take a look at a few surprising ways that data science can increase your sales, both offline and online. Do you know who you are selling to? You have quite a few different systems for gathering information about your client. They scattered about and do not take them into consideration. You have loyalty information from in-store purchases because your front line is methodical asking but your online purchase history does not take this information into account. <blockquote>More precisely, groceries and big-box stores optimize separately for online and offline. We lose value of our marketing endeavors if we don’t take a wider look at our data and search for insights.</blockquote> Amazon is a great example of this, with its anticipatory shipping. Products are shipped before customers even order them. Past behavior and predicted purchases are used to plan on decreasing next day shipping to next hour shipping. Is it crazy? Of course it is! Products are being shipped without having anyone to receive them. But that doesn’t matter. Once these products are out of the warehouse and in a given area, they can be marketed to others at a discounted rate or kept at the final hub. This is more of a logistic miracle than an e commerce one, but shows how forward thinking you have to be if you want to lead the future. I would wager that you are not Amazon, but with such a behemoth so far ahead, it makes sense to try your hand at staying competitive in your niche. It is certainly working for Amazon, with over $2 billion in profits last year. <img src="/blog-old/assets/article_images/2017-07-20-improve-retail/amazon-distribution.png" alt="An image from amazon's anticipatory shipping patent illustrating the logistic paths" /> But how does this actually work? There is some machine learning that goes into predicting client behavior. Machine learning takes in data to train a model. Training is the process of feeding data into a model for it to apply statistical weights allowing the model to automatically recognise future purchases. For example, John purchases a new book every two or three weeks. Based on this behavior, we know what to expect from him. We do not use all of our data, but divide it into train and test data. If we use too much of our data as test set, then we could end up overfitting; a situation when our model identifies artifacts in the data that do not exist. This a simplified example that isn’t representative of the insights pulled out of millions of clients’ purchase history. These behaviors are then juxtaposed with each other to segment clients into various cohorts that overlap and vary. Machine learning methods can be used for a variety different use cases such as product recommendations, churn predictions, logistics planning and automatic personalized marketing. There are quite a few more use cases that I’ll let your imagination come up with. <img src="/blog-old/assets/article_images/2017-07-20-improve-retail/supervised-machine-learning.png" alt="A high level diagram of supervised machine learning process" /> Let’s get started and build some interesting applications of data science. First off, we need to agglomerate all of the relevant data we have on our clients. Certain regions have legal restrictions on what sensitive information you can store. There is a way around this; I am not advocating for breaking the law but rather, to anonymize your data. Many times this can be done with a common user ID across every system you use. I also want to be clear and point out that there are certain types of data that we simply can not collect. Always ask a lawyer about your options when dealing with sensitive information. <img src="/blog-old/assets/article_images/2017-07-20-improve-retail/privacy-map.jpg" alt="Map of global data protection. It is a bit out of date but the overall message is the same" /> We should also consider a few large steps that need executing, namely, creating a data lake and preferably stream processing as well. You may be wondering why we need a data lake as opposed to a data warehouse. There are a few ways in how they differ but the biggest advantages of data lakes over data warehouses are <ul><li><strong>Format</strong>: We can be agnostic about what data we let in, it all goes in and doesn’t need structure</li><li><strong>Flexible</strong>: Can be configured as needed and highly flexible to our needs</li><li><strong>Cost</strong>: We can use servers adaptively</li><li><strong>Processing</strong>: Raw data can be loaded without any assigned model</li></ul> Flexibility is exactly what we need when doing data science; the entire structure of a data warehouse would need changing whenever we want to try something new. This may seem amusing, but is only the tip of what insights lie below the surface. Stream processing lets us react near instantaneously to changes in our client’s behavior. For example, they walk into our store and purchase an item on our site at a lower rate. We lost money here because our dynamic pricing did not take into account the client’s location. This may not always be possible but should get those gears spinning as to when else we can achieve. <h2 id="use-cases">Use Cases</h2> Retail and e-commerce both generate a surprisingly large amount of data in recent years. But without the proper tools in place, most of this data illustrates what you already know. I want to point out a few common uses of data science in retail. There are quite a few more that we could discuss, but not every company is ready for them. What I mean is that they are trying to skip steps that need to be done before we start completely automating processes. At Appsilon, we help companies at any level of data science maturity take the next step. <h3 id="product-recommendations">Product Recommendations</h3> Traditionally, retailers would have consultants in store that would use their own judgement to help customers find what they need. There are also endcaps that present the highest bidder to customers without taking into consideration their interests or needs. Online, this would be based on the past revenue and would try and recommend the highest selling products. This didn’t really make any sense, as these products were doing quite well selling themselves and skewed the rest of the catalogue. There are a few ways for recommendations to reach your clients. The first most basic example is where we do not know our client at all. They are literally a ghost and we don’t even know where they are browsing from. Without using data science we can recommend a high margin item or the most common purchase our clients make as the first thing our new visitor sees. Your recommendations are based on intuition but are not correct. As an example let suppose we have 3 products: <ul><li>A) margin: $10 probability of purchase: 10% = expected margin is $1</li><li>B) margin: $5 probability of purchase: 30% = expected margin is $1.5</li><li>C) margin: $1 probability of purchase: 60% = expected margin is $0.6%</li></ul> The best product to recommend is B, even though it is not most common (less than C) and not with the highest margin (less than A). We then take a client who we recognize and can tailor our offering. Based on their activity, in combination with our other client data, we know what we expect our client to purchase. Not only that, but we have information about what items go hand in hand with our client. Let’s say that they have a grill in their shopping cart and are proceeding to checkout. This would be a perfect time to remind our client about the need for a spatula, grill scrubber, lighter fluid or a thermometer. This helps increase sales but also makes sure that our client is not unpleasantly surprised when his item doesn’t include batteries. We can always get data as well <ul><li>weekday, time of a day</li><li>IP address -&gt; geolocation</li><li>browser data (mobile or desktop? which internet browser?)</li><li>behavioral data (how do they browse our website)</li></ul> The most interesting and complex way to help our customers is by taking their aesthetics into account. Taking into consideration not only historical purchases and recent activity, but also online behavior and social media gives us a much deeper purview of their interests, preferences and the kinds of designs that could interest. It is at the level of abstraction where we truly appeal to our client individually. Furniture is an segment that aptly highlights where this can provide immense value. We know that our client is looking for a chair. Based on their activity, they are quite interested in tech and sci fi. Therefore, a cushy leather chair should not be our first product recommendation, but rather, an ergonomic chair with sharp lines. <h3 id="assortment-optimization">Assortment Optimization</h3> Long tails: the bane of inventory and catalogue optimization; what if we got rid of them? There are obviously items we need to keep to satisfy our niche customers, but overall, we should take a deep look at what we keep in stock. By analysing our product demand and the types of products our clients usually by, we can more accurately ascertain the need for certain products to be in stock. There may be items that are very niche, but that pull in a large cart because this item has a lot of associated cross products. The margins on other long tail products may be so high, that even with only a few annual purchases, they are still worth including. Our competitors are a very good source of information for the assortment we may want to include. Taking into account a full view of our competitors inventory can illuminate a few blind spots we had, a few products that we should have included a long time ago. In relation to the personalized product recommendations above, this works when creating new products or updating our seasonal line. We take the aggregate aesthetics of our proposed segments and can see the types of designs that could bring in the highest number of new clients. It may turn out that a neon purple carpet with pterodactyls are exactly what expecting parents want in their bedroom. I don’t think this is the case, but that is the issue we are addressing. Our intuition pales in comparison to the types of insights we can pull from our data. If we are diligent in our data science approach and streamline our inventory to a near optimal level, savings start popping up all over the board. Be it from decreased production times, to minimizing the amount of maintenance that is needed. Customer service and sales can also focus on increasing their product expertise so as to better serve customers. <h3 id="pricing">Pricing</h3> The market will pay whatever the market can support. The price of fruit obviously change depending on their country of origin, the season and weather conditions. Those are supply side changes, but what about demand? Various customers are prepared to purchase different items at drastically different prices. A taxi driver needs oil changes or car filters much more than a student who only drives home every other weekend. Scale also plays into pricing. Buying in bulk takes this into consideration but does not include recurring purchases. It makes sense to discount a given item if we expect our client to purchase the same item every month. Also, adaptive pricing that reacts to deals that may be lost could benefit from a discount at the point of purchase. The best pricing decisions also take into account a larger scope of data that includes the weather, location, time and day of purchase and other economic factors. <h3 id="personalized-marketing">Personalized marketing</h3> This is especially beneficial for large retailers. Let’s take a look at an example that holds quite a bit of promise for routine advertising. This can apply to digital and offline equally, it is easier for digital. A monthly newsletter with information about new products, discounts, and promotions going on is often sent our entire list of customers without any regard for who they are. But they should be personalized, at least to the cohort level. Here, we can take into advantage of the product recommendations, assortment optimization and dynamic pricing, as they all have an effect on what we include in our communications. A few years ago, there was quite a bit of buzz around Target, in particular, detecting pregnant customers. These customers are proven to be extremely tired and not have any desire to do their groceries in multiple places. As such, it was in Target’s interest to get these customers through the door for formula or diapers because they would then do their entire grocery shopping for the week. The life time value of these clients is very high, as there is low churn among them for a few years following pregnancy. But the hype was over glorified. The story itself revolves around a 16 year old girl, her father, a Target manager and a flier. The father had received fliers and coupons for maternal vitamins, diapers, etc. and came in to complain to the manager. The manager apologized and proceeded to call the father a few weeks later, but it turned out that the daughter was in fact pregnant. While the exact story <a href="http://www.kdnuggets.com/2014/05/target-predict-teen-pregnancy-inside-story.html" target="_blank" rel="noopener noreferrer">never happened</a>, target did find a way to predict pregnancies. It turns out that there are very specific items purchased when an individual is in their second trimester. Over 20 items taken into consideration were enough to give every customer a pregnancy chance score. This example illustrates the kinds of insights that can be discovered but only scratched the surface of what is possible. <h2 id="summary">Summary</h2> We’ve looked at four examples of how data science and machine learning can be used in online and offline retail. While the number of possibilities is almost unlimited, the most important points I want to reiterate are the benefits of personalization and the true impact of trusting in data. It is wasteful to collect data and not take advantage of it. <a href="mailto:konrad.pabianczyk@wordpress.appsilon.com">Get in touch with us</a> if you want to see how your data can give you a competitive advantage.