The @WalmartLabs Social Media Analytics project

Firstly, let me wish you all a happy new year. 2012 was definitely very exciting for us all. While scientists at CERN were sifting through a whopping 200 petabytes of data analyzing 800 trillion collision events to detect the Higgs boson, we at @WalmartLabs started out on a little hunt of our own. I’m talking about an insights-mining project we started working on last year.

Social Media Analytics is all about mining retail-related insights from social channels, a perilous and personally exciting task to us. When our team spent the 22nd of November feverishly following the social retail pulse on Black Friday, we knew the world wasn’t preparing for an apocalypse.

As we watched the incredible surge in Walmart related social buzz on that day, we were only gently reminded another time of the promise that lay hidden deep within the treasure of the social data goldmine – the promise of social media analytics that is only emphasized all the time today, almost to the point of a cliché. The potential itself is nevertheless, still largely untapped. We are only barely beginning to scratch the surface of all the great tales that the data has to tell us.

Social buzz typically precedes retail buzz. People are constantly expressing about upcoming stuff on social apps - the hot new video game whose trailer just released, the cool gadget about to be launched, or a new upgrade to that toy that your child always loved. There are good things said, and bad things too. And thus, social media is really a direct real-time feedback channel to us from our many customers. I am still only stating the obvious.

Our goal is to tap into this social buzz and help Walmart with decision making on aspects like inventory and assortment. As an example, the figure below shows a reasonable spike in social activity about Sony's new Android phone Xperia Z, few days ahead of its actual launch. Such insights can help our merchants make smarter decisions ahead of time.

Social data mining comes with incredible challenges, which only makes it all the more exciting for our super smart engineers to come to work every day. Data volume is formidably huge. We are talking about petabytes here. Real-time social data processing requires sophisticated data stores and blazingly fast algorithms. The noise levels are exorbitant, the language used in social forums is heavily informal, unstructured and often ungrammatical, and filtering out that helpful insight out of the huge amount of noise is super hard. Just consider algorithmically parsing - “OMG!!! dis is sooo coool! i luv ma new fone. i cant believ ma luck 4 chosin this! #wellwhatdoyathink”. Popular text analytics and natural language processing techniques based on standard language models simply fail. We need altogether different techniques to filter out and focus on social data that is relevant to us, which in itself is a daunting task. The next step is to map this to meaningful retail products. All of these are difficult tasks. As a quick sneak-peek, a new technique we are trying out today is to look for any of several hand-verified n-grams around brands in a large time window. Several more schemes are to follow. It is only after conquering all of these multifold challenges that meaningful recommendations can be made.

Today, our social media analytics project operates on top of a searchable index of 60 billion social documents and helps merchants at Walmart monitor sentiments and popular interests real-time, or inquire into trends in the past. One can also see geographical variations of social sentiments and buzz levels. There are also tools that marry search trends on walmart.com, sales trends in our brick-and-mortar stores and social buzz all in one place, to help make correlations. Together, these tools provide powerful social insights today.

As we step into another fantastic year, we are excited to be taking up more audacious goals. On one hand, we aspire to improve the quality of our insights and work with our merchants to expedite them effectively. On the other, we aim to map our interest trends to demand levels for actual products and come up with insights for assortment and inventory management. And all of this, well ahead of time, while we can make a difference. 

It is going to be an exciting year indeed.

The Community is pushing the Mobile Web Platform

At @WalmartLabs, our focus is to develop technologies that provide an innovative shopping experience to millions of users around the world. Our customers spend their time on a variety of platforms of their choosing, so we aim to be where they are. One of the key platforms is the Web, and much effort has been spent over the last few years to ensure that users visit Web sites via rich Web runtimes. The mobile Web used to be about feature phones getting minimal content, but that world has changed quickly over the last few years and the mobile device in your hand has an amazing engine that has the power to do as much as your desktop computer.

There have been a number of efforts to standardize features for the mobile Web, but none of them have taken off. Partly, this is due to the fast evolution and change from the world of WAP to the world of HTML5 on the phone. Today, it has reached a point of critical mass where people are looking to push development of the mobile Web platform. This push won't happen with only certain perspectives, and needs broad participation across the industry.

We have been talking to Facebook about the issues surrounding the mobile Web for quite some time, and I am very excited to have @WalmartLabs be a part of a new community effort, the forming of W3C Core Mobile Web Platform Community Group whose goals are clear:

Accelerate the adoption of the Mobile Web as a compelling platform for the development of modern mobile Web applications.

The Community Group will bring developers, equipment manufacturers, browser vendors, operators and other relevant members of the industry together to agree on core features developers can depend on; create related conformance test suites; and provide to W3C (and non-W3C) groups use cases, scenarios, and other input to enable successful mobile Web app development.

A critical topic to address is for most important specs to be supported across a wide range of browsers. But, support is only half the battle. The quality and performance of a feature is crucial as well. This is why we are looking forward to seeing rich test suites that go into that side of things too.

We love the Web, but users -- and thus developers -- demand that the platform evolves faster than it has to date. As we push the boundaries of the mobile Web at @WalmartLabs, we look forward to the opportunity to discuss the needs of users and developers as we work with community members to make the Web a better place.

Join us!