Firstly, let me wish you all a happy new year. 2012
was definitely very exciting for us all. While scientists at CERN were sifting
through a whopping 200 petabytes of data analyzing 800 trillion collision
events to detect the Higgs boson, we at @WalmartLabs started out on a little
hunt of our own. I’m talking about an insights-mining project we started
working on last year.
Social Media Analytics is all about mining retail-related insights from social channels, a perilous and personally exciting task to us. When our team spent the 22nd of November feverishly following the social retail pulse on Black Friday, we knew the world wasn’t preparing for an apocalypse.
Social buzz typically
precedes retail buzz. People are constantly expressing about upcoming stuff on
social apps - the hot new video game whose trailer just released, the cool
gadget about to be launched, or a new upgrade to that toy that your child
always loved. There are good things said, and bad things
too. And thus, social media is really a direct real-time feedback channel
to us from our many customers. I am still only stating the obvious.
Social Media Analytics is all about mining retail-related insights from social channels, a perilous and personally exciting task to us. When our team spent the 22nd of November feverishly following the social retail pulse on Black Friday, we knew the world wasn’t preparing for an apocalypse.
As we watched the incredible surge in Walmart related
social buzz on that day, we were only gently reminded another time of the
promise that lay hidden deep within the treasure of the social data goldmine – the
promise of social media analytics that is only emphasized all the time today,
almost to the point of a cliché. The potential itself is nevertheless, still
largely untapped. We are only barely beginning to scratch the surface of all the
great tales that the data has to tell us.
Our goal is to tap into this social buzz and
help Walmart with decision making on aspects like inventory and assortment. As
an example, the figure below shows a reasonable spike in social activity about Sony's new Android phone Xperia Z, few days ahead of its actual launch. Such insights can help our
merchants make smarter decisions ahead of time.
Social data mining comes with incredible
challenges, which only makes it all the more exciting for our super smart
engineers to come to work every day. Data volume is formidably huge. We are
talking about petabytes here. Real-time social data processing requires
sophisticated data stores and blazingly fast algorithms. The noise levels are
exorbitant, the language used in social forums is heavily informal,
unstructured and often ungrammatical, and filtering out that helpful insight
out of the huge amount of noise is super hard. Just consider algorithmically
parsing - “OMG!!! dis is sooo coool! i luv ma new fone. i cant believ ma
luck 4 chosin this! #wellwhatdoyathink”. Popular text analytics and natural
language processing techniques based on standard language models simply fail.
We need altogether different techniques to filter out and focus on social data
that is relevant to us, which in itself is a daunting task. The next step is to
map this to meaningful retail products. All of these are difficult tasks. As a
quick sneak-peek, a new technique we are trying out today is to look for any of
several hand-verified n-grams around brands in a large time window. Several
more schemes are to follow. It is only after conquering all of these multifold
challenges that meaningful recommendations can be made.
Today, our social media analytics project operates on top of a searchable index of
60 billion social documents and helps merchants at Walmart monitor sentiments
and popular interests real-time, or inquire into trends in the past. One can
also see geographical variations of social sentiments and buzz levels. There
are also tools that marry search trends on walmart.com, sales trends in our
brick-and-mortar stores and social buzz all in one place, to help make
correlations. Together, these tools provide powerful social insights today.
As we step into another fantastic year, we are
excited to be taking up more audacious goals. On one hand, we aspire to improve
the quality of our insights and work with our merchants to expedite them effectively.
On the other, we aim to map our interest trends to demand levels for actual
products and come up with insights for assortment and inventory management. And
all of this, well ahead of time, while we can make a difference.
It is going to be an exciting year indeed.
