The DataWeave Blog

Tag: Expert Speak

DataWeave is home to thought leaders in retail, data science, artificial intelligence, and more. Sneak a peek into their minds.

  • AI-powered Product Matching: The Key to Competitive Pricing Intelligence in eCommerce

    AI-powered Product Matching: The Key to Competitive Pricing Intelligence in eCommerce

    With thousands of products and hundreds of online retailers to choose from, the average modern-day shopper usually compares prices across several e-commerce sites effortlessly before often settling for the lowest priced option. As a result, retailers today are forced to execute millions of price changes per day in a never-ending race to be the lowest priced – without losing out on any potential margin.

    Identifying, classifying, and matching products is the first step to comparing prices across websites. However, there is no standardization in the way products are represented across e-commerce websites, causing this process to be fairly complex.

    Here’s an example:

    What’s needed is a pricing intelligence solution that first matches products across several websites swiftly and accurately, and then enables automated tracking of competitor pricing data on an ongoing basis.

    Pricing intelligence solutions already exist. What’s wrong with using them?

    There are several challenges with the incumbent solutions in the market – the biggest one being that they don’t work in a timely manner. In essence, it’s like deferring the process of finding actionable information that helps retailers acquire a competitive advantage, and instead doing it in hindsight. Like an autopsy of sorts.

    Here are the various solution types we have in the market today:

    • Internally developed systems – Solutions developed by retailers themselves often rely on heavy manual data aggregation and have poor product matching capabilities. Since these solutions have been developed by professionals not attuned to building data crunching machines, they pose significant operational challenges in the form of maintenance, updates, etc.
    • Web scraping solutions – These solutions have no data normalization or product matching capabilities, and lack the power to deliver relevant actionable insights. What’s more, it’s a struggle to scale them up to accommodate massive volumes of data during peak times such as promotional campaigns.
    • DIY solutions – These solutions require manual research and entry of data. It goes without saying that due to the level of human intervention and effort required, they’re expensive, difficult to scale, slow, and of questionable accuracy.

    As common as it is nowadays, AI has the answer

    DataWeave’s competitive pricing intelligence solution is designed to help retailers achieve precisely the competitive advantage they need by providing them with accurate, timely, and actionable pricing insights enabled by matching products at scale. We provide retailers with access to detailed pricing information on millions of products across competitors, as frequently as they need it.

    Our technology stack broadly consists of the following.

    1. Data Aggregation

    At DataWeave, we can aggregate data from diverse web sources across complex web environments – consistently and at a very high accuracy. Having been in the industry for close to a decade, we’re sitting on a lot of data that we can use to train our product matching platform.

    Our datasets include data points from tens of millions of products and have been collected from numerous geographies and verticals in retail. The datasets contain hierarchically arranged information based on retail taxonomy. At the root level, there’s information such as category and subcategory, and at the top level, we have product details such as title, description, and other <attribute, value> relationships. Our machine learning architectures and semi-automated training data building systems, augmented by the skills of a strong QA team, help us annotate the necessary information and create labeled datasets using proprietary tools.

    2. AI for Product Matching

    Product matching at DataWeave is done via a unified platform that uses both text and image recognition capabilities to accurately identify similar SKUs across thousands of e-commerce stores and millions of products. We use an ensemble deep learning architectures tailored to NLP and Computer Vision problems specific to us and heuristics pertinent to the Retail domain. Products are also classified based on their features, and a normalization layer is designed based on various text/image-based attributes.

    Our semantics layer, while technically an integral part of the product matching process, deserves particular mention due to its powerful capabilities.

    The text data processing consists of internal, deep pre-trained word embeddings. We use state-of-the-art, customized word representation techniques such as ELMO, BERT, and Transformer to capture deeply contextualized text with improved accuracy. A self-attention/intra-attention mechanism learns the correlation between the word in question and a previous part of the description.

    Image data processing starts with object detection to identify the region of interest of a given product (for example, the upper body of a fashion model displaying a shirt). We then leverage deep learning architectures such as VggNet, Inception-V3, and ResNet, which we have trained using millions of labeled images. Next, we apply multiple pre-processing techniques such as variable background removal, face removal, skin removal, and image quality enhancing and extract image signatures via deep learning and machine learning-based algorithms to uniquely identify products across billions of indexed products.

    Finally, we efficiently distribute billions of images across multiple stores for fast access, and to facilitate searches at a massive scale (in a matter of milliseconds, without the slightest compromise on accuracy) using our image matching engine.

    3. Human Intelligence in the Loop

    In scenarios where the confidence scores of the machine-driven matches are low, we have a team of Quality Assurance (QA) specialists who verify the output.

    This team does three things:

    • Find out why the confidence score is low
    • Confirm the right product matches
    • Figure out a way to encode this knowledge into a rule and feed it back to the algorithm

    In this way, we’ve built a self-improving feedback loop which, by its very nature, performs better over time. This system has accumulated knowledge over the 8 years of our operations, which is going to be hard for anyone to replicate. Essentially, this process enables us to match products at massive scale quickly and at very high levels of accuracy (usually over 95%).

    4. Actionable Insights Via Data Visualization

    Once the matching process is completed, the prices are aggregated at any frequency, enabling retailers to optimize their prices on an ongoing basis. Pricing insights are typically consumed via our SaaS-based web-portal, which consists of dashboards, reports, and visualizations.

    Alternatively, we can integrate with internal analytics platforms through APIs or generate and deliver spreadsheet reports on a regular basis, depending on the preferences of our customers.

    To summarize

    The benefits of our solution are many. Detailed price improvement opportunity-related insights generated in a timely manner empower retailers to significantly enhance their competitive positioning across categories, product types, and brands, as well as ability to influence their price perception among consumers. These insights, when leveraged at a higher granularity over the long term, can help maximize revenue through price optimization at a large scale.

    Our solution also helps drive process-based as well as operational optimizations for retailers. Such modifications help them better align themselves to effectively adopt a data-driven approach to pricing, in turn helping them achieve much smarter retail operations across the board.

    All of this wouldn’t be possible if the product matching process, inherent to this system, was unreliable, expensive, or time-consuming.

    If you would like to learn more about DataWeave’s proprietary product matching platform and the benefits it offers to eCommerce businesses and brands, talk to us now!

  • [INFOGRAPHIC] 2020: The Year the World Navigated Uncertainty Together

    [INFOGRAPHIC] 2020: The Year the World Navigated Uncertainty Together

    The start of 2020 brought with it the promise of global economic growth. Markets in the US were on a steady rise we also witnessed demand from brands and retailers in Europe and the Middle East. All seemed to be on track to make it a year of plenty.

    Out of nowhere, the end of the first quarter saw the world coming to a grinding halt. The world was held hostage by a global pandemic and the force with which we were hit, was unprecedented.

    From February to mid-May we saw things come to a sharp halt. We at DataWeave seized this intermittent downtime to bolster our product offerings.

    On the flip side, when the world did start opening May onwards, we saw completely new categories take center stage digitally. With new habits and trends taking shape, the pandemic single-handedly caused exceptional growth in the Food and Grocery Delivery intermediaries. Predictably, the rest of the world followed. Our existing customers saw the competition rise steeply with everyone coming online. We invested substantially in our Digital Shelf Analytics solutions after noticing that e-commerce was seeing a boom. 2020 saw brands making their online presence the new norm. This meant that small, medium and large enterprises had to now divert their spending to analytics and e-commerce. 

    It is interesting to note that the rise in the food and grocery delivery segment gave brands another channel to focus on vis a vis their presence. Brands that were available on these sites focused on how they could optimize their sales on these channels, which proved to be the front runners during the height of the pandemic. While the challenges and opportunities for both these segments overlapped and seemed similar, our solutions helped measure and optimize brand performance across all online channels. Some of the in-demand solutions and analytics we saw our customers use were; share of search, content audit, assortment and availability, pricing and promotions, and ratings and reviews. 

    There were mixed emotions in the market, with regard to the best use of marketing spends. Human resource and client cutbacks happened across the board. At DataWeave however, we had the pleasure of onboarding 25 new clients including retailers and brands ranging from food and grocery delivery, home improvement from across multiple geographies.

    Infographics

    Throughout the year, the work never ceased at DataWeave. The team showed incredible resilience while working remotely, making sure our deliverables were being taken care of, at all times. Due to the e-commerce boom and immense pressure from existing and new entrants in the digital space, our clients saw a need to gather more insights. With the given uptick, we are happy to report that our stellar 95%+ accuracy record for in-depth insights at scale, was maintained through the course of all the work done.

    Looking forward to the year 2021:

    In the US, the adoption of e-commerce accelerated as traditional brick and mortar stores shut down and pivoted. To put things into perspective, e-commerce adoption grew only by 4.3% from 2014 to 2019. In just three months in 2020, e-commerce adoption grew at 4.3%! Add to that, with approved vaccines making their way slowly to the public, we do anticipate the travel sector to open up and we look forward to working with new clients.

    Nike’s Chief Executive, John Donahoe recently said, ” We know that digital is the new normal. The consumer today is digitally grounded and simply will not revert back…the shift to online sales could be a permanent trend.” We could not agree more! With online sales here to stay, brand and retailers’ requirements to keep their competitive edge will only continue to grow. We at DataWeave, look forward to delivering the results they want in this new year, and for the years to come.

  • Coronavirus Outbreak: Impact on E-Commerce Retailers and Consumer Brands

    Coronavirus Outbreak: Impact on E-Commerce Retailers and Consumer Brands

    The Coronavirus, otherwise known as COVID-19, has made landfall on U.S. shores. At the time of writing this article, there are over 230 confirmed cases in the country and 12 deaths. The growing unease about the virus, which has quickly accumulated 95,000+ confirmed cases globally, has, among other things, adversely affected businesses and stock markets the world over.

    In the wake of this outbreak, U.S. based retailers and brands would be prudent to brace themselves and plan ahead to minimize disruptions as much as possible.

    Businesses and consumers in China, the global epicenter of the epidemic, have been dealing with these challenges over the last couple of months. It’s likely that some of the trends observed in China would be mimicked in the U.S. as well, something that domestic retailers and brands would do well to study and prepare for.

    The Inadvertent E-commerce Wave

    When the outbreak happened in China, it caused an uptick in e-commerce adoption as shoppers were reluctant to step out of their homes and instead, opted to shop for their goods online.

    Reports indicate that Chinese online retailer JD.com’s online grocery sales grew 215% YoY over a 10-day period between late January and early February. Similarly, Carrefour’s vegetable deliveries grew by 600% YoY during the Lunar New Year period. Online sales of Dettol, a disinfectant produced by Reckitt Benckiser, rose 643% YoY between 10 February and 13 February on China’s Suning.com.

    In Singapore, another region affected by the virus more recently than in China, Lazada’s grocery arm, RedMart, and Supermarket chain, NTUC FairPrice, both reported an unprecedented surge in demand, which tested their delivery capabilities to the limit.

    This bump in online sales isn’t just restricted to grocery, but other categories as well. Jean-Paul Agon, CEO of L’Oréal, recently said that online sales of the brand’s beauty products increased in China in February.

    Given such a consistent shift in shopping behavior across coronavirus-affected regions, it’s logical to expect that a similar trend would be followed in the U.S. – in fact, it might already be underway.

    A recent survey by Coresight Research indicated that 27.5% of U.S. respondents are avoiding public areas at least to some extent, and 58% plan to if the outbreak worsens. Of those who have altered their routines, more than 40% say they are “avoiding or limiting visits to shopping centers/ malls” and more than 30% are avoiding stores in general. The survey also found consumers will likely begin to avoid restaurants, movie theaters, sporting events and other entertainment venues.

    Therefore, it’s essential for U.S. retailers and brands to swiftly energize their e-commerce readiness and be fully prepared to cater to the circumstances-induced shift in shopping behavior, inclined toward online.

    A Logistical Nightmare

    The most obvious area of impact for retailers and brands is in their supply chain and order fulfilment operations.

    A large portion of consumer product manufacturers rely to some extent on China, and the potential impact of the virus on supply chain processes is inescapable. Chinese factories have been operating at partial capacity, impacting supply chains globally. This has largely affected highly popular e-commerce categories like consumer electronics, fashion and furniture.

    Shares in the U.S. of furniture e-commerce retailer, Wayfair, fell as much as 26% toward the end of February, according to a Bloomberg report. The is particularly revealing, as the online retailer reportedly relies on China for half of its merchandise.

    Retailers struggling to cope with this stress in their supply chain systems would do well to warn their customers beforehand about delays in deliveries, like AliExpress has just done.

    For categories like CPG, as consumers increasingly shop online, retailers that offer Buy Online Pick Up In Store (BOPIS), should expect a surge in its adoption, and reinforce their online infrastructure and in-store operations to cater to the rising demand.

    In addition to disruptions in the supply chain, several other mission-critical areas are likely to get affected too.

    Keeping Up With The Online Surge

    As with any event of this magnitude, the business implications reach far and wide. The following are a few areas that we’ve identified as critical, based on our experience working with retailers and brands. Being aware of and focusing on these issues are likely to alleviate some of the issues faced by consumers today.

    Fair pricing: There have been several reports of price gouging on e-commerce platforms. Examples include 2-ounce Purell bottles being sold for $400 and face masks for up to $20. While these prices have mostly been set by third party merchants, brands are likely to face the flak from consumers. A recent Bloomberg article reported that online retailers still rely partly on employees to manually monitor these items. This approach has obvious limitations, such as products quickly reappearing on the website after being de-listed. Brands and e-commerce platforms will need to explore automated ways of controlling their online pricing practices at large scale.

    3P merchant and counterfeit management: Often, unauthorized third-party merchants selling an original manufacturer’s goods are the ones who unreasonably inflate prices. These merchants tend to test the markets on online marketplaces with their pricing, which adversely affects the brand image of the manufacturer. Further still, they sometimes list counterfeit or fake goods that make incorrect or extravagant claims. Brands will need to swiftly identify and de-list these merchants from online marketplaces.

    Ensuring stock availability: During times like these, it’s a common sight to see empty aisles at supermarkets selling items like canned food, water, paper products and personal care products. Consumers will benefit from brands monitoring their stock availability at stores, which will help them better align their supply chain operations to the rapidly changing demand patterns across the U.S. map. This way, efforts can be more targeted at regions with severe shortages.

    Content compliance: Helium 10, a technology provider for Amazon sellers, reported that since 26 February, 90% of searches on Amazon are coronavirus related, and searches for hand sanitizers spiked to 1.5 million searches in February compared to 90,000 in November. As a result, to arrest exploitative practices, some online marketplaces have announced policy guidelines on product content claiming health benefits. Words like ‘Coronavirus‘, ‘COVID-19‘, ‘Virus‘ and ‘epidemic’ are, in fact, prohibited.  Amazon has already de-listed several merchants claiming fraudulent cures. Ebay has gone as far as to ban all new listings for face masks, hand sanitizers, and disinfecting wipes, due to regulatory restrictions. In this context, retailers and brands will benefit from deploying tracking mechanisms that quickly identify offenders.

    The areas of business presented above are by no means a comprehensive list for retailers and brands to rely on during this time. Still, these are critical impact areas for them to address, even as huge efforts are made toward managing highly stressed supply chains.

    DataWeave Offers Support

    The coronavirus outbreak is likely to get worse before it gets better. As we enter unchartered territories, DataWeave is offering to contribute in small ways, pro bono, by leveraging our expert talent and competitive intelligence technology platform, to address some of the challenges faced by retailers and brands.

    We’re announcing a limited-time, no-cost offer to detect and report on price gouging, the presence of unauthorized third-party merchants, as well as stock availability across U.S. ZIP-codes. This offer will be valid for 4-6 weeks (timeline will be flexible based on how the outbreak develops) and limited to monitoring the top 10 U.S. online marketplaces, as well as critical product categories such as medicinal and hygiene-related products, emergency food items, survival-related products, fuel, etc.

    Reach out to us for further details.

  • [INFOGRAPHIC] 2019 at DataWeave: Blazing New Trails

    [INFOGRAPHIC] 2019 at DataWeave: Blazing New Trails

    As another year comes to a close, we look back at 2019 with fond memories and look forward to the exciting new prospects of 2020. Take a trip with us as we highlight some of DataWeave’s milestones of the last twelve months.

    Over the course of the year, DataWeave’s success has gone hand in hand with the evolution of retail and e-commerce, reinforcing the relevance of our technology platform.

    Our rapid growth in the North American market is a reflection of how intense competition in the region is triggering the need for accurate, timely, and actionable competitive and market insights, as well as other avenues for retailers and brands to gain a competitive edge.

    Last year, we saw a resurgence of big-box (omnichannel) retailers as they adopted innovative approaches to play to their strengths (their offline stores). Offering buy online, pick up in store (BOPIS) or click-and-collect options, rolling out price match guarantee programs, and expanding their partnerships with delivery services like Instacart, enabled these retailers to leverage the best of both the online and offline worlds to compete with e-commerce firms.

    Amazon continues to dominate e-commerce with a daunting 38% share in the US. Still, the partnerships between brands and Amazon are increasingly being tested. Nike and Ikea recently joined the likes of Swatch and Birkenstock to sever ties with the retail behemoth. This seemingly growing trend is largely due to counterfeits continuing to leak through the system.

    Brands that used to de-prioritize their focus on their eCommerce channel (as it often was only a small portion of their revenues) have come to realize that consumers use large marketplaces like Amazon not just to shop for products but also to perform product research. As a result, how these brands are represented and sold online impact their offline sales. And with the onset of BOPIS and click-and-collect initiatives, brands can now analyze this correlation even at a hyperlocal (ZIP-code) level.

    Large marketplaces, for their part, have started taking advantage of the increasingly brand-agnostic shopping behavior of consumers by launching ad-platforms for brands and manufacturers, enabling them to boost their visibility online.

    Due to such sweeping transformations to the market landscape, brands and retailers are increasingly looking more toward intelligent tech-based solutions to help them gain a competitive edge.

    In order to effectively serve the growing need for competitive and market insights, we’ve pushed our platform to its limits and beyond. It’s our constant endeavor to innovate and improve. This is evident with the launch of a host of new features on our product suite, especially Brand Analytics – designed to enable consumer brands to protect their brand equity and optimize e-commerce performance.

    One of the key factors that enabled us to achieve all the milestones we did is the aggressive hiring of some of the most skilled talent in the tech industry. Our team grew by 44% in 2019, giving us additional confidence to raise the bar on our capabilities and offer 95% accuracy in our data and insights to our customers consistently.

    We’re encouraged by the fact that we’ve more than doubled as a business, year-over-year, for the past several years, without depending solely on growing the team, but also by consolidating our technology stack, optimizing our processes, and scaling our products.

    Here’s a sneak peek into our performance in 2019:

    2020 Vision

    The upcoming year promises to be an exciting one for the retail industry and the consumer brand space at large. We plan to be at the helm and increase our footprint all around. There’s a strong focus to expand our US team and consequently, the business. While we continue to strengthen our roots in India, we will look toward other mature markets like the UK, Germany and the Middle East as well.

    On other fronts, we’re gathering steam on new partnership engagements – consulting firms, ad tech firms, marketing agencies and complementary technologies. We will also expand our foray into the travel and delivery services verticals.

    With our diversifying portfolio, we haven’t lost sight of one of the most important aspects of any successful company – its employees. We will continue to keep our employees engaged, motivated, and satisfied by providing vertical and horizontal career growth opportunities, conducting personalized training programs, organizing hackathons, fostering cross-team collaboration and learning, and encouraging everyone to periodically blow off some steam at company retreats and the ferociously fought in-house sports tournaments.

    Here’s to a stellar 2020 of empowered retailers and brands. We wish them well as they navigate the dense competitive landscape, knowing that they have an ally in their corner with DataWeave.

  • 3 Common Problems Brands Face in eCommerce | DataWeave

    3 Common Problems Brands Face in eCommerce | DataWeave

    Over the last three years, I have helped deploy eCommerce analytics solutions for several brands and manufacturers globally. During this time, I have conversed with day-to-day users up through C-Suite executives of some of the world’s most successful brands, while also working with the founders of startup brands who were simply trying to find their place in the world of commerce.

    As I look back on my time to date, I have noticed a few themes emerge from my diverse client conversations with brands, which are indicative of an ecosystem that’s only now coming to terms with retailers and consumers moving online. Here are three fundamental problems I’ve seen brands often run into as they adapt to the world of eCommerce:

    1. “We have no idea what we are doing”

    My favorite part about being an analytics solution provider is the introduction session with a new client. I always entered these conversations with a few key questions:

    – What are your top three eCommerce initiatives for the next 12 months?

    – How does your team and other internal resources align with these initiatives

    – How do you envision using this type of tool to help you succeed with your goals? What made you choose ours?

    Early in my career, what always amazed me was that these enormous brands – wildly successful brands – entered into a partnership without a clear plan to execute. Many would fumble through what I thought were very basic questions. After a few of these conversations, I came to the realization that most brands have a limited understanding of what they are doing in eCommerce.

    How could this be possible?

    I remember a conversation with a large CPG brand executive. He said, “Keep in mind, most of the people doing these jobs are from a bricks-and-mortar world. They don’t have eCommerce experience because no one does. It is too new. We don’t have the resources to hire more people because eCommerce makes up less than 1% of our total revenue.”

    As an industry, brands are collectively making it up as they go. Few admit it, but the industry is growing and evolving so fast, the best that some do is hold on for the ride (while taking a few calculated chances along the way).

    2. “We measure success poorly”

    I have noticed that, with time, many brands are starting to get a better grasp on how to operate online, though there is still a long way to go for many. The best evidence for this improvement is the growth in the number of job posts for eCommerce-focused roles, new vendors popping up in this space, and industry centers of excellence being developed. As more people choose eCommerce roles, the biggest challenge that I see is the lack of effective measurement and training processes.

    Often, the issue is that many brands take a long-standing, loyal executive and assign them as the eCommerce leader. When this person is not forward thinking, analytical or open to trying things a new way, brands fail. The reason startup brands are winning online is because they are entering the eCommerce game with an open and fresh perspective. Forcing old ways into eCommerce will surely lead to failure.

    I have worked with many brands that have developed eCommerce centers of excellence and have shared best practices on how to measure teams and success. The most painful to deal with were the organizations that brought their bricks-and-mortar measurements into the eCommerce world. The data used to measure success was the wrong data. The KPIs were set in a way that people would surely fail.

    In my opinion, the best measurements for success are sales growth (not share growth), digital shelf KPIs (search and content first), and a subjective measure on maturity in the industry. The best first step is to have someone lead the team who understands how to measure success and execute in a cutting-edge and evolving environment.

    3. “We sign up with either too many or the wrong service providers”

    The final observation is one that is costing many teams a lot of money. Many brands start to move into eCommerce based on their old team structures. Each team has a separate eCommerce objective, budget, and set of tools to execute with.

    Then, when the centralized eCommerce team (Center of Excellence) gets established, they will likely find many teams working with many tools. Sometimes, they see many teams signed up, via separate contracts, with the same tool. Worse still – it’s often the wrong type of tool.

    As brands evaluate tools, they need to ask questions such as:

    • Does this vendor provide global coverage, so that we can establish a global way of thinking and executing (with the ability to customize for local consumption when required)?
    • Does this vendor have the backbone (people and technology) to scale with my business?
    • (The best question, in my opinion) Does this vendor have people who are willing to listen and understand my business, or are they simply people who want to sell me a cookie-cutter solution?

    In my experience, I have seen brands spending way too much time, effort, and money on vendors who do not check the boxes listed above.

    Summing up…

    As I look back over my time serving brands in the eCommerce analytics space, I have seen an industry morph and transform time and time again. I have seen companies shift, re-shift, panic and pivot.

    If you’re a brand, my encouragement to your team is to hit the pause button. Ask the right questions. Evaluate your goals, your team structure, and your vendor partners. If the strategy, execution, people, and measurement, are not aligned, come up with a plan to get them back on track. Be willing to learn a new way to do business.

    Pause. Reset. Measure.

  • Flaunt Your Deep-Tech Prowess at Bootstrap Paradox Hackathon Hosted by Blume Ventures

    Flaunt Your Deep-Tech Prowess at Bootstrap Paradox Hackathon Hosted by Blume Ventures

    When DataWeave was founded in 2011, we set out to democratize data by enabling businesses to leverage public Web data to solve mission-critical business problems. Eight years on, we have done just that, and grown to deliver AI-powered competitive intelligence and digital shelf analytics to several global retailers and brands, which include the likes of Adidas, QVC, Overstock, Sauder, Dorel, and more.

    As the company has grown, so has our team, which is now 140+ members strong. We’re still constantly on the lookout for smart, open, and driven folks to join us and contribute to our success.

    And so, we’re excited to partner with Skillenza and Blume Ventures to co-host the Bootstrap Paradox Hackathon, where we are eager to engage with the developer community and contribute in our own way back to the startup ecosystem.

    The event will be conducted as an offline product building competition, with a duration of 24 hours on August 3-4, 2019 at the Microsoft India office in Bengaluru. It will provide a platform for developers and coders to interact with and solve challenges thrown up by DataWeave and other Blume portfolio companies, such as Dunzo, Unacademy, Milkbasket, Mechmocha, and Locus.

     

     

    Taking up DataWeave’s challenge during this Hackathon will give you a sneak peek into what our team works on daily. It’s no surprise that we have “At DataWeave, it’s a Hackathon every day!” plastered on our walls. After all, it’s not just all about intense work, but also a lot of fun and frolic.

    The problems that we deal with are as exciting as they are hard. Some of our key accomplishments in technology include:

    • Matching products across e-commerce websites at massive scale and at high levels of accuracy and coverage
    • Using Computer Vision to detect product attributes in fashion such as a color, sleeve length, collar type, etc. by analyzing catalog images
    • Aggregating data from complex web environments, including mobile apps, and across 25+ international languages

    One of our more recent innovations has been in optimizing e-commerce product discovery engines, which dramatically improves shopper experience and purchase conversion rates. During the Bootstrap Paradox Hackathon, coders will get a chance to build a similar engine, with guidance and assistance from DataWeave’s technology leaders.

    Data sets containing product information like title, description, image URL, price, category etc. will be provided, and coders will need to clean up the data, extract information on relevant product attributes and features, and index them, in the process of building the product discovery engine.

    For more details on the challenge, register here on the Skillenza platform.

    As a sweetener, the event also promises everyone a chance to win over 10 lakhs in prize money.

    Simply put, if you love code, this is the place to be this weekend. See you there!

  • The Importance of Pricing Parity for Brands

    The Importance of Pricing Parity for Brands

    With bricks-and-mortar stores steadily increasing their online presence, the balancing act of pricing online and in-store is now more important and complex than ever. Companies spend years building brands and brand equity. Yet, a misplaced or poorly executed pricing strategy to handle both online and offline pricing can erode that equity with consumers very quickly.

    This problem is not new. It first started when Clubs like Costco and Sam’s started popping up in the 80’s. Suddenly, brands had to figure out a way to balance Club and Grocery pricing while taking advantage of a new, fast-growing channel. The biggest difference between now and then is that consumers now can check prices within seconds on their phone.

    So, how do you avoid losing your brand equity while ensuring price parity across online and offline channels?

    The key areas to consider are:

    1. Product Mix

    Do you have a broad enough mix of product sizes and case configurations for each channel? To maximize your sales and minimize your price disruption, reviewing your supply chain and product mix to ensure you are able to deliver value to both online and offline retailers is critical. Each channel is looking for ways to improve and maximize your brand sales. If you do not give them the right size and case configuration to enable them to increase margins, you will end up relying disproportionately on trade spend (dollars a brand spends with a retailer to promote products) to do so, or find your product on page 212 of every search.

    Examples of this strategy can be seen with companies offering only “bundled” items such as 12 cans or a large case on online marketplaces, while other retailers offer individual cans for purchase. This allows your online partners to make up margin by shipping a full case and not going through the process of breaking down a case and shipping single units. Also, this allows bricks-and-mortar retailers to have a sharper price point to lure consumers into the store. This strategy has played out well for many brands as they dealt with the rise of Club stores and can be played successfully in e-commerce as well, benefiting all parties.

    2. Price Lists

    Do you have harmonized price lists that do not favor one channel over another? If you do not, you are likely subsidizing the higher list cost in a channel with trade spend, which is highly inefficient. A single price list that provides an adequate price slope between the various sizes across your product range will maximize your ability to manage both channel pricing and brand equity.

    The single largest mistake brands tend to make is thinking that offering “net price” price lists to online marketplaces will benefit them while they use trade dollars in bricks-and-mortar stores to cater to EDLP (Everyday low price) customers. This approach is quite inefficient in many ways, and consumes valuable time and resources that can otherwise be better utilized. Having a single price list with the same price offered to all retailers allows for a more manageable and equitable pricing environment. It also enables a more profitable distribution of trade spend across the most effective areas to invest in for each retailer.

    I have worked with two brands in the past – one that managed two separate price lists and one that we implemented as a single-standard. While the one with the single price list saw sales grow and trade spend remain constant, the other saw trade spend double in just two years as it got caught in a scenario of always having to placate one side of the equation or the other.

    3. Trade Spend

    Today’s brands need to focus on a balanced trade spend strategy to address each channel’s unique needs. Using trade spend with online retailers can be tricky, as the channel is usually assumed to be the lowest priced anyway. Still, it can be used to drive traffic and offset supply chain costs, in order to ensure sufficient margins for the retailer, which will keep you off the CRAP (Can’t Realize A Profit) lists. Meanwhile, as JC Penny quickly learned when it made the disastrous shift to EDLP, consumers still want in-store discounts and sales.

    The best approach I have worked with is to set a single dead net price inclusive of all trade. For example, if your product’s standard list cost is $6.80 and you have a dead net price for promotions (or EDLP) of $5.40, then all retailers – online and bricks-and-mortar – are on equal footing. The only variance in the price for consumers will be the margin each operator chooses to take. This approach is not without issues, as you have to apply all elements of trade spend (such as ad fees, etc.) to the promotional unit costs to ensure you are truly capturing the dead net cost of the retailer.

    Still, the advantage of utilizing this approach is that when a retailer complains about the price another is offering to consumers, the conversation turns to margins being taken and not the cost of the product. At the very least, this approach provides a common ground on which to have a constructive conversation with all retailers.

    So why does this all matter so much to a brand?

    The road to selling online is littered with disaster and missed expectations for sales. Most manufacturers that jumped to online sales without considering pricing quickly learned that abandoning one channel for another does not lead to increased sales. Conversely, we have seen a few brands go from online only to in-store as well. These brands seem to have learned from the others’ mistakes and rarely will you find price variances between the online and offline channels. Instead, you tend to see these brands growing, as online consumers start experiencing the brand in-store.

    A Business To Community study by Larisa Bedgood in 2019 showed that “lower price” was second to only “convenience” for why consumers shop online, while 51% of consumers said that the biggest drawback to shopping online was not being able to touch and feel the product. Brands that are able to bridge the gap and provide consumers with the convenience of online while also showing up well in-store at the right price point will be able to break out of the stagnate 1-2% (if they are lucky) growth most CPG companies are experiencing. If online selling is growing 40-50% a year, why are these companies only managing brand declines and flat growth? I believe it is mainly due to the lack of a proper pricing parity strategy for the two channels along with a lack of actionable e-commerce data.

    Brands that do not focus on all three areas listed above often find themselves in a constant churn of conversations with retailers on all sides, which will typically lead to either online marketplaces or bricks-and-mortar stores deprioritizing the brand in promotions or search. Finding and setting a level playing field will allow for a balanced trade spend and growth for brands on both platforms, while also enabling a brand to break out of the net 1% growth that is plaguing a lot of CPG brands today.

    Outside of deploying basic pricing principles for your brand, I would also suggest early and strong investments in data, systems and people to monitor your brand’s health and pricing. Many brands jumped online without any way to monitor the consumer conversation around the brand or the pricing of the brand online. Not having the tools and resources in place to do this can lead to a quick and long-lasting erosion of brand equity and sales. Most, if not all, large manufacturers have subscribed to POS data for years and fully understand how to analyze this data. But the world has shifted. If your organization has not invested in digital shelf analytics, you may be driving blind and unaware that your brand is losing equity, which equals losing consumers and sales.

    Using a combination of pricing principles and e-commerce data mining tools will help you maintain price parity and brand relevance, while keeping you from becoming the last brand of choice for consumers, regardless of where they shop.

  • Compete Profitably in Retail: Leveraging AI-Powered Competitive Intelligence at Massive Scale

    Compete Profitably in Retail: Leveraging AI-Powered Competitive Intelligence at Massive Scale

    AI is everywhere. Any retailer worth his salt knows that in today’s hyper-competitive environment, you can’t win just by fighting hard – you have to do it by fighting smart. The solution? Retailers are turning to AI in droves.

    The problem is that many organizations regard AI as a black box of sorts – where you can throw all your data (the digital era’s blessing that feels like a curse) in at one end and have miraculously meaningful output appearing out the other. The reality of how AI works, however, is a lot more complex. It takes a lot of work to make AI work for you – and then to derive value out of it.

    Image Source: https://xkcd.com/1838

    Following the advent of the digital era, businesses across industries, particularly retail, were left grappling with massive amounts of internal data. To make things worse, this data was unstructured and siloed, making it difficult to process effectively. Yet, businesses learned to leverage simple analytics to extract relevant data and insights to affect smarter decisions.

    But just as that happened, the e-commerce revolution stirred things up again. As businesses of all shapes, sizes, and types moved online, they suddenly became a whole lot more vulnerable to other players’ movements than they were just about a decade ago, when buyers rarely visited more than one store before they made a purchase. In other words, retailers are now operating in entire ecosystems – with consumers evaluating a number of retailers before making a purchase, and a disproportionate number of players vying for the same consumer mindshare and share of wallet.

    Thus, external data from the web – the largest source of data known to man at present – is becoming critical to business’ ability to compete profitably in the market.

    Competing profitably in the digital era: Can AI help?

    As organizations across industries and geographies increasingly realized that their business decisions were affected by what’s happening around them (such as competitors’ pricing and merchandize decisions), they started shifting away from their excessive obsession with internal data, and began to look for ways to gather external data, integrate it with their internal data, and process it all in entirety to derive wholesome, meaningful insights.

    Simply put, harnessing external data consistently and on a large scale is the only way for businesses to gain a sustainable competitive advantage in the retail market. And the only way to practically accomplish that is with the help of AI. Many global giants are already doing this – they’re analyzing loads of external data every minute to take smarter decisions.

    That said, though, what you need to know is that all this data, while publicly available and therefore accessible, is massive, unstructured, noisy, scattered, dynamic, and incomplete. There’s no algorithm in the world that can start working on it overnight to churn out valuable insights. AI can only be effective if enormous amounts of training data is constantly fed back into it, coaxing it to get better and more astute each time. However, given the scarcity of readily available training datasets, limited and unreliable access to domain-specific data, and the inconsistent nature of the data itself, a majority of AI initiatives have ended up in a “garbage in, garbage out” loop that they can’t break out of.

    What you need is the perfect storm

    At DataWeave, we understand the challenge of blindly dealing with data at such a daunting scale. We get that what you need is a practical way to apply AI to the abundant web data out there and generate specific, relevant, and actionable insights that enable you to make the right decisions at the right time. That’s why we’ve developed a system that runs on a human-aided-machine-intelligence driven virtuous loop, ensuring better, sharper outcomes each time.

    Our technology platform includes four modules:

    1. Data aggregation: Here, we capture public web data at scale – whatever format, size, or shape it’s in – by deploying a variety of techniques.

    2. AI-driven analytics: Since the gathered data is extremely raw, it’s cleaned, curated, and normalized to remove the noise and prepare it for the AI layer, which then analyzes the data and generates insights.

    3. Human-supervised feedback: Though AI is getting smarter with time, we see that it’s still far from human cognitive capabilities – so we’ve introduced a human in the loop to validate the AI-generated insights, and use this as training data that gets fed back to the AI layer. Essentially, we use human intelligence to make AI smarter.

    4. Data-driven decision-making: Once the data has been analyzed and the insights generated, they can either be used as it to drive decision-making, or then integrated with internal data for decision-making at a higher level.

    With intelligent, data-backed decision-making capabilities, you can outperform your competitors

    Understandably, pricing is one of the most popular applications of data analytics in retail. For instance, a leading, US-based online furniture retailer approached us with the mission-critical challenge of pricing products just right to maximize sell-through rates as well as gross margin in a cost-effective and sustainable manner. We matched about 2.5 million SKUs across 75 competitor websites using AI and captured pricing, discounts, and stock status data every day. As a result, we were able to affect an up to 30% average increase in the sales of the products tracked, and up to a 3x increase in their gross margin.

    DataWeave’s powerful AI-driven platform is essentially an engine that can help you aggregate and process external data at scale and in near-real time to manage unavoidably high competition and margin pressures by enabling much sharper business decisions than before. The potential applications for the resulting insights are diverse – ranging from pricing, merchandize optimization, determination of customer perception, brand governance, and business performance analysis.

    If you’d like to learn more about our unique approach to AI-driven competitive intelligence in retail, reach out to us for a demo today!

  • 2018 at DataWeave: A Year of Prolific Success and Growth

    2018 at DataWeave: A Year of Prolific Success and Growth

    As we enter 2019, in the backdrop of DataWeave’s unprecedented growth and success, we decided to take a breath and look back at some of the highlights of our progress over the last 12 months.

    DataWeave’s growth through the year has been complemented and influenced by the evolution of the retail sector, reinforcing the relevance of our technology platform.

    Amazon continued to dominate the online retail landscape, now commanding a staggering 49% of US e-commerce. At the same time, several large retailers have taken sure-footed strides toward establishing a stronger e-commerce presence, which places them head to head against the Seattle-based retail behemoth. As a result, competitive intelligence is no longer a “good-to-have” but is fundamental to the survival and growth of both traditional and new-age retailers, enabling them to devise smarter, data-driven competitive strategies.

    Consumer brands are continuing to figure out the dynamics of selling on online marketplaces, which happens to give them valuable access to a vast base of shoppers while simultaneously restricting their ability to influence the brand experience. In their quest to sell more through the e-commerce channel, while trying to safeguard the brand experience and loyalty, consumer brands have turned increasingly toward e-commerce performance platforms to augment their decision-making process.

    These trends have reinforced our confidence in our technology platform, which aggregates and analyzes data from the Web at massive scale to deliver actionable competitive insights, as we’re well poised to address the evolving challenges presented to retailers and brands today.

    In 2019, there are no signs of slowing down for DataWeave.

    We will continue to execute strongly in high-growth regions, and especially in the US, which has, in a span of two years, become the largest revenue generating region for DataWeave. We will also build a stronger footing in Europe, with specific focus on the UK market.

    With time, our historical repository of data increases in volume and granularity, which enables us to better serve the maturing space of Alternative Data. We have already witnessed highly encouraging inbound interest over the last year, and we expect this interest to rise significantly moving forward.

    With great success, comes the need for great people. In 2019, we will aggressively expand our team across functions, organization levels, and regions. As always, DataWeave is on the lookout for people who flourish in a competitive environment and can propel us to the next stage of growth.

    Our technology platform never ceases to impress in its ability to aggregate and analyze billions of data points accurately each day. As our pipeline swells and we onboard bigger and more diverse customers, the platform will consistently be pushed to its limits, driving further innovations and improved efficiency.

    Over the following 12 months, on the strength of all the lessons learnt and successes achieved in 2018, we look forward to another challenging year of empowering retailers and consumer brands to compete profitably in the new world order.

    Watch this space for more on DataWeave through the year!

  • CEO Speak: Serving the US Market, Hiring the Right Talent, And More

    CEO Speak: Serving the US Market, Hiring the Right Talent, And More

    Recently, Karthik Bettadapura, Co-founder & CEO at DataWeave, was interviewed by Vishal Krishna, Business Editor at YourStory, in the Bay Area, California. They discussed DataWeave’s focus on the US market, challenges that retailers face today, DataWeave’s technology platform and hiring practices, and more.

    The following is a transcript of the interview.

    (The transcript has been edited for clarity and brevity)

    Vishal Krishna (VK): You left India to come and conquer America, why is that?

    Karthik Bettadapura (KB) : Just a bit of history — we started in 2011 and product development and research was based in Bangalore, and still is. At the end of the first 5 years, we realized that we built great technology, but we were not able to scale beyond a certain point [in India]. If we had to build a growing business, we had to look at other markets as well.

    VK: Quickly, can you tell me what DataWeave does?

    KB: We provide Competitive Intelligence to retailers and customer brands. We work with some of the largest brands and retailers out there and we provide them with analyses to compete profitably.

    VK: You said you had marque clients in India, yet you didn’t want to stay there because you wouldn’t have scaled beyond a particular point. Why is that?

    KB :The ticket size in India is still on the lower side. If you must build a sustainable business, you need access to a much larger customer base and we found that in the US.

    VK: Let’s start from the basics. What are a few things that a startup should decide to do when coming to America?

    KB: A few things:

    • A good understanding of the market
    • Learn fast about the market
    • Build a team here, or a have a team here already doing some work initially
    • Consider how your team back in India will go about doing things in your absence
    • The last one is about your own personal journey. I was so used to walking into an office and interacting with people. You come here, and you are all alone!

    VK: It’s a lonely journey. Doors don’t open all that easily and you’ve got to hustle. Why?

    KB: For people here, you are an unknown entity. Why should they be trusting someone who does not have enough customers here or has not raised money here? We had two US-based customers when we came in. It’s an uphill task to ensure that customers trust you.

    VK: Who was the first customer you personally met here and why was that meeting so important?

    KB: The first customer I met here was a large, big box retailer, and the meeting was primarily focused around why they should trust us — how can they know that we would survive and serve them, as well as how we are better than some of the other guys out there.

    VK: Can you tell us what DataWeave does for US retailers?

    KB: For retailers, we provide competitive intelligence, primarily around pricing optimization and assortment analytics. In the US, a lot of retailers are shutting shop and filing for bankruptcy.

    VK: Yeah, we saw Sears go through something like that.

    KB: The reasons fall broadly into 3 categories:

    • They failed to compete profitably with a lot of these new age businesses.
    • The new age retailers offer superior customer experience. They have figured out a better assortment/product strategy.
    • The third one is ‘Price’ — price is such an important feature.
      What we do is help these retailers optimize their strategies around pricing, assortment and promotions, eventually enabling them to compete profitably.

    VK: Typically, customers pay you on the outcome, pricing, license or subscription?

    KB: It’s a subscription-based model. There is a one-time setup fee and an ongoing subscription fee.

    VK: So you plug into their data management system?

    KB: Yes, but we can also have our product sit independently. Sitting out of their internal systems is a benefit for us as we don’t have to get into the entire loop of integrations into their internal systems right from Day 1. We prove our product works and then we integrate with their systems.

    VK: How do you integrate? Is the CIO your target?

    KB: No, we don’t sell to the CIO world. We sell to analytics, pricing, and merchandising teams.

    VK: Can pricing alone give retailers a competitive edge?

    KB: Yes, pricing is a big lever that retailers use. For example, last holiday season’s sale, Amazon and Walmart made 120 million price changes in just 2–3 days.

    VK: So they change the prices so dynamically to compete with each other. Is this price war coming to India?

    KB: It is happening in India already.

    VK: How much data can DataWeave’s infrastructure ingest?

    KB: We are a global platform — we have customers across the globe, not just the US or India. So, on a daily basis, we process data on around 120 million products.

    VK: Talk a little bit on R&D quickly. Do you have your marketing team in the US?

    KB: We have marketing teams in the US and India.

    VK: And the engineering team?

    KB: The engineering team is in Bangalore.

    VK: For people who want to work in your company, what kind of talent are you looking for?

    KB: We look at 4 broad talent areas:

    • One is in the world of data acquisition, which addresses issues like how data can be aggregated from thousands of websites and millions of pages on an ongoing basis, and how this data can be stored.
    • The second area is on what kind of insights can be generated using this data. This could be done using text analytics, image analytics, and other technologies. This includes process optimization, in terms of building efficient and scalable systems.
    • The third area is on how well the data can be represented if we have a customer who wants 60–70 million data points to be consumed on a weekly basis.
    • And the last area is on data modeling — what kind of insights can we eventually give to the customer? And, when I say insights, I mean specific actions.

    VK: You want people who can handle massive scale and for that they should be good at linear regression.

    KB: We value people who write good code. We primarily work in Python, and we use a lot of optimization techniques in the middle of the stack to help us scale.

    VK: Would you do something for supermarkets?

    KB: Absolutely. The largest offline supermarket in India is our customer.

    VK: So what can you do for supermarkets?

    KB: Offline retailers across the world are facing something that’s called showrooming. This is when a shopper walks up to a store, looks at and feels a product, then searches online to see it’s available at a better price. So we have retailers who are wary of this phenomenon. We also have retailers who are wary of diminishing customer loyalty. So they have to constantly ensure that they are priced better in the market and are not losing customers because of [online] pricing.

    VK: How powerful are your algorithms?

    KB: There is a dedicated team that works on our algorithms. These fall into several buckets. One is pure data scale algorithms — how do you build systems which ensure that you are able to efficiently query them in real time and get the desired output. The second one is — how do you keep improving your machine learning algorithms. For example, computer vision algorithms, text analytics algorithm, etc. The third — how do you keep experimenting effectively.

    VK: What role can an MBA degree holder play in DataWeave?

    KB: We have people who hold MBA degrees and are working in customer success, delivery management, marketing, and sales.

    VK: Do you spend time in training?

    KB: You do have some lead time if you are a fresher, but if you are a lateral hire, its expected that you keep the ball rolling. They should be able to learn and learn fast — learning is more important than knowing. So, we give a lot of importance to people who can learn and pick up things quickly – about our product, handling customer objections, etc.

    *

    Watch the whole video here or check out DataWeave’s website to know more about how we use data engineering and artificial intelligence to enable retailers and brands to compete profitably in the age of eCommerce.

  • Evaluating the Influence of Learning Models

    Evaluating the Influence of Learning Models

    Natt Fry, a renowned thought leader in the world of retail and analytics, published recently an article expounding the value and potential of learning models influencing business decision-making across industries over the next few years.

    He quotes a Wall Street Journal article (paywall) published by Steven A. Cohen and Matthew W. Granade who claim that, “while software ate the world the past 7 years, learning models will ‘eat the world’ in the next 7 years.”

    The article defines a learning model as a “decision framework in which the logic is derived by algorithm from data. Once created, a model can learn from its successes and failures with speed and sophistication that humans usually cannot match.”

    Narrowing this down to the world of retail, Natt states, “if we believe that learning models are the future, then retailers will need to rapidly transform from human-learning models to automated-learning models.”

    This, of course, comes with several challenges, one of which is the scarcity of easily consumable data for supervised learning algorithms to get trained on. This scarcity often results in a garbage-in-garbage-out situation and limits the ability of AI systems to improve in accuracy over time, or to generate meaningful output on a consistent basis.

    Enabling Retailers Become More Model-Driven
    As a provider of Competitive Intelligence as a Service to retailers and consumer brands, DataWeave uses highly trained AI models to harness and analyze massive volumes of Web data consistently.

    Far too often, we’ve seen traditional retailers rely disproportionately on internal data (such as POS data, inventory data, traffic data, etc.) to inform their decision-making process. This isn’t a surprise, as internal data is readily accessible and likely to be well structured.

    However, if retailers can harness external data at scale (from the Web — the largest and richest source of information, ever), and use it to generate model-driven insights, they can achieve a uniquely holistic perspective to business decision-making. Also, due simply to the sheer vastness of Web data, it serves as a never-ending source of training data for existing models.

    DataWeave’s AI-based model to leverage Web data

     

    Web data is typically massive, noisy, unstructured, and constantly changing. Therefore, at DataWeave, we’ve designed a proprietary data aggregation platform that is capable of capturing millions of data points from complex Web and mobile app environments each day.

    We then apply AI/ML techniques to process the data into a form that can be easily interpreted and acted on. The human-in-the-loop is an additional layer to this stack which ensures a minimum threshold of output accuracy. Simultaneously, this approach feeds information on human-driven decisions back to the algorithm, thereby rendering it more and more accurate with time.

    Businesses derive the greatest value when external model-based competitive and market insights are blended with internal data and systems to generate optimized recommendations. For example, our retail customers combine competitor pricing insights provided by our platform with their internal sales and inventory data to develop algorithmic price optimization systems that maximize revenue and margin for millions of products.

    This way, DataWeave enables retailers and consumer brands to utilize a unique model-based decision framework, something that will soon be fundamental (if not already) to business decision-making across industry verticals and global regions.

    As AI-based technologies become more pervasive in retail, it’s only a matter of time before they’re considered merely table stakes. As summarized by Natt, “going forward, retailers will be valued on the completeness of the data they create and have access to.”

    If you would like to learn more about how we use AI to empower retailers and consumer brands to compete profitably, check out our website!

    Read Natt’s article in full below:

    Steven A. Cohen and Matthew W. Granade published a very interesting article in the Wall Street Journal on August 19, 2018 — https://www.wsj.com/articles/models-will-run-the-world-1534716720

    Their premise is that while software ate the world (Mark Andreessen essay in 2011, “Why Software is Eating the World”) the past 7 years, learning models will “eat the world” in the next 7 years.

    A learning model is a decision framework in which the logic is derived by algorithm from data. Once created, a model can learn from its successes and failures with speed and sophistication that humans usually cannot match.

    The authors believe a new, more powerful, business opportunity has evolved from software. It is where companies structure their business processes to put continuously learning models at their center.

    Amazon, Alibaba, and Tencent are great examples of companies that widely use learning models to outperform their competitors.

    The implications of a model-driven world are significant for retailers.

    Incumbents can have an advantage in a model-driven world as they already have troves of data.

    Going forward retailers will be valued on the completeness of the data they create and have access to.

    Retailers currently rely on the experience and expertise of their people to make good decisions (what to buy, how much to buy, where to put it, etc.).

    If we believe that learning models are the future then retailers will need to rapidly transform from human-learning models to automated-learning models, creating two significant challenges.

    First, retailers have difficulty in finding and retaining top learning-model talent (data scientists).

    Second, migrating from human-based learning models to machine-based learning models will create significant cultural and change management issues.

    Overcoming these issues is possible, just as many retailers have overcome the issues presented by the digital age. The difference is, that while the digital age has developed over a 20 year period, the learning-model age will develop over the next 7 years. The effort and pace of change will need to be much greater.

  • What Retailers Can Learn from the Lowe’s Board Announcement

    What Retailers Can Learn from the Lowe’s Board Announcement

    Last Friday, Reuters published, “Home Improvement chain Lowe’s said it has nominated two independent board members and plans to add a third following “constructive” talks with hedge fund D.E. Shaw Group, which has taken an activist stake.”

    It was reported that D.E. Shaw Group had utilized available external data to identify quantifiable opportunities to grow sales by several billion dollars and to reduce costs significantly.

    A question that comes immediately to mind is, “Why didn’t Lowe’s utilize this same available external data themselves?”

    Is it because Lowe’s and many other retailers spend their time focusing on internally generated data, rather than looking at available external data, or better yet, combining available external data with their internal data?

    There are huge opportunities to drive incremental sales, margins and profits through leveraging external data, like competitive intelligence data produced by firms like DataWeave.

    There are huge opportunities to drive incremental store sales, margins, and profits through leveraging digital data to drive better store specific assortments, prices and promotions by providing relevant local digital data to store executives using solutions by firms like Radius8.

    I expect to see more Lowe’s-like announcements in the near future as investment firms realize there are very substantial, untapped financial opportunities within retail.

  • Dataweave – CherryPy vs Sanic: Which Python API Framework is Faster?

    Dataweave – CherryPy vs Sanic: Which Python API Framework is Faster?

    Rest APIs play a crucial role in the exchange of data between internal systems of an enterprise, or when connecting with external services.

    When an organization relies on APIs to deliver a service to its clients, the APIs’ performance is crucial, and can make or break the success of the service. It is, therefore, essential to consider and choose an appropriate API framework during the design phase of development. Benefits of choosing the right API framework include the ability to deploy applications at scale, ensuring agility of performance, and future-proofing front-end technologies.

    At DataWeave, we provide Competitive Intelligence as a Service to retailers and consumer brands by aggregating Web data at scale and distilling them to produce actionable competitive insights. To this end, our proprietary data aggregation and analysis platform captures and compiles over a hundred million data points from the Web each day. Sure enough, our platform relies on APIs to deliver data and insights to our customers, as well as for communication between internal subsystems.

    Some Python REST API frameworks we use are:

    • Tornado — which supports asynchronous requests
    • CherryPy — which is multi-threaded
    • Flask-Gunicorn — which enables easy worker management

    It is essential to evaluate API frameworks depending on the demands of your tech platforms and your objectives. At DataWeave, we assess them based on their speed and their ability to support high concurrency. So far, we’ve been using CherryPy, a widely used framework, which has served us well.

    CherryPy

    An easy to use API framework, Cherrypy does not require complex customizations, runs out of the box, and supports concurrency. At DataWeave, we rely on CherryPy to access configurations, serve data to and from different datastores, and deliver customized insights to our customers. So far, this framework has displayed very impressive performance.

    However, a couple of months ago, we were in the process of migrating to python 3 (from python 2), opening doors to a new API framework written exclusively for python 3 — Sanic.

    Sanic

    Sanic uses the same framework that libuv uses, and hence is a good contender for being fast.

    (Libuv is an asynchronous event handler, and one of the reasons for its agility is its ability to handle asynchronous events through callbacks. More info on libuv can be found here)

    In fact, Sanic is reported to be one of the fastest API frameworks in the world today, and uses the same event handler framework as nodejs, which is known to serve fast APIs. More information on Sanic can be found here.

    So we asked ourselves, should we move from CherryPy to Sanic?

    Before jumping on the hype bandwagon, we looked to first benchmark Sanic with CherryPy.

    CherryPy vs Sanic

    Objective

    Benchmark CherryPy and Sanic to process 500 concurrent requests, at a rate of 3500 requests per second.

    Test Setup

    Machine configuration: 4 VCPUs/ 8GB RAM.
Network Cloud: GCE
Number of Cherrypy/Sanic APIs: 3 (inserting data into 3 topics of a Kafka cluster)
Testing tool : apache benchmarking (ab)
Payload size: All requests are POST requests with 2.1KB of payload.

    API Details

    Sanic: In Async mode
Cherrypy: 10 concurrent threads in each API — a total of 30 concurrent threads
Concurrency: Tested APIs at various concurrency levels. The concurrency varied between 10 and 500
Number of requests: 1,00,000

    Results

    Requests Completion: A lower mean and a lower spread indicate better performance

     

    Observation

    When the concurrency is as low as 10, there is not much difference between the performance of the two API frameworks. However, as the concurrency increases, Sanic’s performance becomes more predictable, and the API framework functions with lower response times.

    Requests / Second: Higher values indicate faster performance

    Sanic clearly achieves higher requests/second because:

    • Sanic is running in Async mode
    • The mean response time for Sanic is much lower, compared to CherryPy

    Failures: Lower values indicate better reliability

    Number of non-2xx responses increased for CherryPy with increase in concurrency. In contrast, number of failed requests in Sanic were below 10, even at high concurrency values.

    Conclusion

    Sanic clearly outperformed CherryPy, and was much faster, while supporting higher concurrency and requests per second, and displaying significantly lower failure rates.

    Following these results, we transitioned to Sanic for ingesting high volume data into our datastores, and started seeing much faster and reliable performance. We now aggregate much larger volumes of data from the Web, at faster rates.

    Of course, as mentioned earlier in the article, it is important to evaluate your API framework based on the nuances of your setup and its relevant objectives. In our setup, Sanic definitely seems to perform better than CherryPy.

    What do you think? Let me know your thoughts in the comments section below.

    If you’re curious to know more about DataWeave’s technology platform, check out our website, and if you wish to join our team, check out our jobs page!

     

  • [INFOGRAPHIC] 2017 at DataWeave: A Year in Retrospect

    [INFOGRAPHIC] 2017 at DataWeave: A Year in Retrospect

    And that’s a wrap! Another exciting year done and dusted, in which DataWeave continued to execute strongly through accelerated revenue growth, new customer wins, and expansion to heretofore unchartered regions.

    Through the year, we engaged with retailers and consumer brands of all types and sizes, and our belief that actionable competitive insights will increasingly play a defining role in driving profitable growth in retail was reinforced. Competition was stiff, and more times than not, we came out on top due to our ability to process huge data-sets, and the unmatched accuracy of our insights.

    Encouragingly, the emerging vertical of Alternative Data gained greater maturity, as adoption of non-traditional data sources from the Web by Asset Managers picked up steam.

    Our extensive focus on the North American market yielded impressive results, and we’ve only just scratched the surface.

    Other regions and verticals continued to contribute significantly, helping us close out the year with record sale volumes.

    As we wind ourselves up again for another marathon year in 2018, we look back at some of our achievements across the board, including customer impact, technology leadership, and team contribution and growth:

    Moving into 2018, we have a lot to look forward to.

    We’ll roll out a new and improved version of our SaaS-based data visualization platform, built with greater focus on actionability and customizability for our customers. Feedback from early beta tests have already been promising.

    As our team size swells, we’ll be on the lookout for passionate problem solvers, who thrive in a hyper-competitive environment, to join us and contribute to the next stage of our growth journey.

    Across verticals, we are well on our way to digging our heels into the North American market. 2018 will also see us gain a more solid footing in the Alternative Data space.

    With eCommerce adoption showing no sign of slowing down, demand in retail for competitive intelligence solutions is set to soar, and our proprietary data aggregation and analysis platform is up to the challenge of catering to this growing need.

    Stay tuned for more from DataWeave in 2018!

  • Walmart’s Online Pricing Analysis | DataWeave

    Walmart’s Online Pricing Analysis | DataWeave

    In an increasingly competitive retail landscape and facing intense margin pressures, improving the profitability of online commerce is a growing area of focus for all retailers.

    When Amazon acquired Whole Foods in August, several media outlets and analysts speculated whether there would be a slashing of prices across the board. Instead, Amazon lowered prices only on those items that it knew would drive increased traffic to the stores, resulting in a 25% increase in footfall the first 30 days after the acquisition closed.

    (Read also: Amazon’s Whole Foods Pricing Strategy Revealed)

    Disrupting the Status Quo

    Walmart has now announced a shift in its online pricing to draw more shoppers to purchase from its brick-and-mortar stores and save on shipping costs.

    Sarah Nassauer wrote an interesting article for the Wall Street Journal recently, outlining Walmart’s online pricing strategy and its approach to pricing its products differently between its online and offline stores.

    Sarah reports, “Walmart wants to charge customers more to buy some products online than in stores, part of the company’s efforts to boost profits and drive store traffic as it competes with Amazon.”

    What’s interesting is Walmart’s move to display the lower offline store prices on its website for some grocery products, nudging shoppers to drive down to the nearest Walmart store.

    Again, Walmart did not raise prices for all items but only a few, select food and household items, “including boxes of Kraft Macaroni & Cheese, Colgate toothbrushes and bags of Purina dog food, according to people familiar with the matter and comparisons between online and in-store prices.”

    The article goes on to state that, “[T]he move is unusual for Walmart, which has long honed an ‘everyday low price’ message and has worked to keep online prices at least as low as shoppers find in its 4,700 U.S. stores. Walmart e-commerce workers responsible for product sales have been instructed to boost profits along with sales, according to the people familiar with the situation, and are ‘no longer obligated to follow store pricing.’”

    This move indicates a greater focus on online-to-offline (O2O) strategies by the world’s largest retailer in an effort to cut down on the crippling costs of transport operations and logistics. According to a cost analysis by consultants Spend Management Experts, “A $1.28 box of Kraft Macaroni & Cheese could cost a big retailer around $10 to ship from Chicago to Atlanta, depending on how remote the buyer’s address is . . . A smaller retailer would likely pay about double.”

    With this news, the days of providing the same price online and in stores are over, setting a precedent and reflecting important differences in costs and competitor capabilities.

    But how did Walmart know which items to focus on for lowering (or raising) prices?

    Cutting-Edge Competitive Intelligence Solutions

    Did Walmart pick items at random or guess? Not likely. With recent enhancements in competitive intelligence and data analysis solutions, the era of guesswork, gut-fuelled decisions, and manual number crunching is over.

    In today’s digital economy, actionable competitive intelligence has become a critical component in the transformation of retail. Retailers like Amazon and Walmart use competitive insights to identify categories and items that show the greatest potential for increased shopper interest, sales, and profits, to adjust their prices.

    Competitive intelligence providers like DataWeave provide unique, AI-driven, competitive insights and business recommendations by harnessing and analyzing competitive data from the Web.

    When retailers link these competitive insights and data to their internal pricing and inventory systems, they create a powerful engine that marries internal and external forces to produce highly accurate assortment, pricing, and promotion recommendations, all in near real-time.

    As retailers like Walmart experiment with their pricing and merchandizing across channels, they have come to rely on modern retail technology solutions that continue to evolve to help them reduce operational complexities and yield higher ROI.

  • Top 5 Drivers of Successful eCommerce | DataWeave

    Top 5 Drivers of Successful eCommerce | DataWeave

    Retail has undergone a dramatic transformation over the last decade. Once dominant retailers are today being given a run for their money amid a gradual decline in mall traffic and sharply growing consumer preference for shopping online.

    Surfing this online retail wave is Internet behemoth Amazon, which is raking in 43% of all new eCommerce dollars, leaving other retailers floundering in its wake.

    As it unfolded, this transformation has unleashed changes across many areas of retail, a phenomenon that’s been well documented by industry commentators in the media. Some of these shifts include:

    Customer preferences: Customers today are spoilt for choice, both in terms of being able to quickly and easily compare product prices across websites, as well as consistently driving the demand for new and unique products from retailers.

    Hyper-personalization: With shoppers increasingly relying on mobile apps, highly personalized shopping experiences are becoming the new normal.

    Delivery: e-Retailers are competing on faster home deliveries, stretching themselves to guarantee same day delivery, or even (as in the case of hyper-local grocery retailers) within a few hours. Drones, anyone?

    Payment Modes: Even the more tactical aspects of retail, like payment modes, have been forced to evolve. Starting with cash-on-delivery, this trend quickly spread to embrace card payments and digital wallets. These initiatives have posed significant technological and security challenges for retailers.

    As with a forced move in chess, traditional retailers have had to evolve and embrace changes like the ones listed above, in order to survive the incredibly cutthroat world of modern retail. Similar challenges exist for up-and-coming eCommerce companies as well.

    However, many pundits and retailers alike often forget that doing even simple, time-tested things correctly can go a long way in forging an effective competitive position, helping win both market share and customer affections. While digital transformation has altered how these strategies were routinely executed, the fundamentals remain as relevant today as they ever were.

    1. Smarter Pricing

    With 80 percent of first-time shoppers comparing products prior to buying, the need for an eCommerce website to offer competitive pricing has become a mandatory cost-of-entry capability. While dynamic pricing poses a challenge for e-retailers to stay competitive, it also presents them with an opportunity to track their competitors’ pricing and exploit that information to optimize their own pricing.

    However, e-retailers today are frequently forced to perform millions of price-changes every day in the eternal quest to either offer the lowest price or entrench a calculated premium price perception among shoppers.

    For instance, as far back as Christmas season 2014, Amazon is estimated to have made a total of 80 million price changes per day. Similarly, today’s hyper-local grocery retailers offer differentiated and targeted prices for shoppers living in specific zip codes.

    To achieve price controls on this level of scale demands sophisticated automated tracking of competitor pricing to facilitate timely, data-driven dynamic pricing decisions. This has, today, become a table stakes requirement.

    2. Variety and Depth of Product Range

    If customers cannot find what they are looking for on a website, all other aspects of how an eCommerce operator optimizes their retail strategy falls by the wayside.

    A website’s success remains dependent largely on it being able to cater effectively to the needs, wants and desires of its target audience. Simply put, a website offering a mammoth product range may still end up failing compared to a small niche website with a limited but highly targeted assortment that understands closely its customer’s sweet spot.

    However, with millions of products on offer online all day every day, gathering and harvesting deep insights into a competitor’s assortment mix can appear daunting. Include dynamically changing product assortments and different product taxonomies into the standard research mix, and many who lack access to automated competitive intelligence systems find themselves struggling to find the expertise required to gather and summarize this information in an actionable form.

    3. Customer Centricity

    Today, customers demand to be heard. As competitive pricing becomes an expected cost of doing business, retailers will need to place greater support resources and more effective processes to resolve customer problems and complaints in a timely fashion at the heart of their customer service model.

    Following the online social revolution, 9 out of 10 retail customers now expect a consistent response across all social media channels.

    Successful companies like Zappos, Best Buy and Amazon have been quick to understand this significant shift in customer preferences. These retailers have demonstrated their willingness to go the extra mile by establishing a robust, scalable omni-channel support structure.

    The level of this commitment can be seen in Amazon’s recent vision statement announcement, “Amazon today boasts of one of the most responsive omni-channel customer support and Zappos takes pride in sending a personalized response to customer queries. We seek to become Earth’s most customer centric company.” This aggressive customer centric sentiment drives a stake in the ground for all competing eCommerce companies’ to match via their customer service strategy.

    4. Superior Customer Experience

    While bricks and mortar retail stores continue to attract customers by enabling shoppers to touch, feel and test items before they purchase, online and omni-channel retailers have channelized their efforts into increasingly refining their web user experience.

    Several studies reveal it takes only a couple of seconds for a website visitor to decide whether to stay on or leave a website. Aspects such as visual design, ease of use, content attractiveness, website loading time and pervasive calls to action (CTA) are a few of the key user experience parameters that influence visitors to stay on a website.

    eCommerce sites such as Zara, Graze, Asos, and Amazon offer attractively organized and clutter-free designs, which are visually engaging and easy to navigate. While these design elements help them keep their customers engaged, it’s their disciplined focus on content that stimulates visitor conversions.

    Detailed product descriptions and high-quality images are helping these eCommerce sites educate their customers about their products while simultaneously boosting their website’s SEO ranking, helping it attract and engage still more online visitors.

    Complementing the online retailing revolution are substantial efforts by omni-channel retailers to optimize O2O (online to offline) strategies designed to bring together the best of both worlds — the discoverability of online, with the touch-and-feel of an offline environment.

    5. Optimized Promotional Strategies

    With so many options for a shopper to choose from in an increasingly cluttered and competitive online retailing environment, attracting new customers and entrenching customer loyalty is an ongoing challenge. Strategic online promotions are emerging as an effective technique in solving the customer recruitment and retention dilemma. Online promotions if executed effectively are doing wonders for generating inbound website traffic.

    However, for online promotions to be effective, it is critical for e-retailers to understand their competitor’s strategy if they are going to be able to sustain their competitiveness. Key questions to answer in this context are, what brands are they promoting more than others? For how long? At what frequency?

    Keeping a keen eye on and reacting to competitors’ promotions is a key aspect to designing effective online promotions. Being able to exploit this competitive intelligence not only boosts their own sales volumes but erodes that of their competitors as well.

    Competitive Intelligence As A Service

    Having understood the far-reaching impact of these evergreen drivers of eCommerce success, we at DataWeave work with omni-channel and online retailers to provide Competitive Intelligence as a Service and help them evaluate and optimize their strategic approach across the eCommerce landscape.

    If you’re interested in DataWeave’s solutions and would like to learn more about how we help retailers and brands optimize their retail strategies, visit our website!

  • The Role of Competitive Intelligence in Modern Retail

    The Role of Competitive Intelligence in Modern Retail

    When retailers today look to compete in the cutthroat world of online commerce, they face several challenges unique to the nature of modern retail. It is now significantly harder for retailers to benchmark their pricing, assortment, and promotions against their competition, as the online world is highly dynamic and significantly more complex than before.

    Trends like the growing adoption of mobile shopping apps, the rising influence of customer reviews in buying behavior, hyperlocal e-commerce websites differentiating themselves by fulfilling deliveries in a matter of hours — the list goes on — have only added to this complexity.

    However, this complexity also presents an opportunity for retailers to incorporate layers of external competitive information into their merchandising strategies to deliver more value to customers and personalize their experience.

    Vipul Mathur, Chief Branding and Merchandising Officer at Aditya Birla Online Fashion, recently published an article highlighting some of the areas in which Competitive Intelligence providers like DataWeave can strategically influence modern merchandising.

    “The consumer is often driven by the aesthetics of a product, more so in the fashion and lifestyle industries than others. Hence, the choices of buyers are hard to interpret. However, innovative modern technologies are helping us understand these decisions,” says Vipul.

    He provides an example of how using AI-based tools (like DataWeave’s) to unearth the sentiments behind thousands of online reviews can help retailers better channel and message their online promotions.

    “Deciphering the consumers’ comments and converting them into tangible insights is incredible proof of the refinement possible with data analysis tools. It’s like knowing that consumers are delighted by the quality of the soles of a pair of Adidas running shoes. Using this, marketing communication can be modified to highlight this specific product feature,” explains Vipul.

    And it’s not just merchandising. This data can percolate across multiple functions in retail, enabling greater efficiency in operations. “If we have data on the best-selling styles across websites, including other attributes like pricing, region/locality (through pin-code mapping), and possibly even rate of sales, it’s up to our supply-chain systems to ensure that the supply is in accordance with demand.”

    DataWeave’s Retail Intelligence offers global retailers and e-commerce websites with these benefits and more. Our AI-powered technology platform aggregates and analyzes vast volumes of online competitive data and presents them in an easily consumable and actionable form, aiding quick, data-driven merchandising decisions.

    “DataWeave, our partner, has helped us refine our merchandising decisions, saving cost and creating value,” sums up Vipul.

    Read the entire article here, and if you’re intrigued by what DataWeave can do for retail businesses and wish to learn more, visit our website!

     

  • Advantage Flipkart: The Motives Behind Acquiring eBay India

    Advantage Flipkart: The Motives Behind Acquiring eBay India

    Flipkart recently acquired eBay’s India business in an announcement that made a huge splash across the country. With Flipkart already having acquired Myntra and Jabong, and talks of a Snapdeal acquisition picking up steam, this level of consolidation comes clearly as a direct response to internet behemoth Amazon’s aggressive expansion strategies in India.

    With this acquisition play, Flipkart stands to gain primarily on two fronts.

    eBay’s Seller Network

    Firstly, eBay has built a strong network of authorized and highly-rated global sellers, something that Flipkart can leverage to drive increased sales and market share.

    Per Flipkart’s announcement to the press — “Flipkart and eBay have signed an exclusive cross-border trade agreement, as a result of which customers of Flipkart will gain access to the wide array of global inventory on eBay, while eBay’s customers will have access to unique Indian inventory provided by Flipkart sellers. Thus, sellers on Flipkart will now have an opportunity to expand their sales globally.”

    At DataWeave, we ran our proprietary data aggregation and analysis algorithms over eBay’s websites and unearthed some interesting numbers about their seller network.

    eBay.com has a global network of 17,361 sellers, 41% of whom ship to India. Therefore, this acquisition opens the door for Flipkart to gain access to over 7000 global eBay.com sellers who ship to India — a huge boost to the range of products Flipkart can host on its platform.

    Additionally, a sizable chunk — 14% — of eBay.in sellers ship to international destinations. This provides Flipkart with opportunities to expand its reach globally.

    The other, rather lesser known advantage that Flipkart stands to gain from this acquisition is in the refurbished and pre-owned goods space.

    The Emergence of Refurbished and Pre-Owned Goods

    The market for refurbished and pre-owned products is estimated to be between $15 billion and $20 billion globally, with exponential growth forecast for the near future.

    Part of the reason for growth in this segment is it yields higher returns on investment for retailers. While a retailer typically earns 3–5% margin by selling a new smartphone, refurbished smartphones fetch 7–8% margin, and pre-owned smartphones 9–10%.

    The Hidden Advantage

    eBay has established itself over the years as a reliable source of refurbished and pre-owned products, with impressive levels of authentication and warranties. We did a quick analysis of eBay.in, Flipkart, and Amazon to identify their relative strengths in this space.

    Unsurprisingly, Flipkart has close to zero refurbished or pre-owned products hosted on their website. With Amazon, 12% of mobile phones and 9% of Books on their website are refurbished or pre-owned, the largest selling categories in this space.

    eBay.in, though, has a significant share of these products across categories — 95% of books & magazines, 36% of mobile phones, and 28% of televisions — a substantial portion of eBay’s business.

    With this acquisition, Flipkart can now take a gigantic step into the relatively more profitable and exponentially growing refurbished and pre-owned products space. It will also be a strong competitive differentiator for the company as they go head to head with Amazon in India.

    While the refurbished and pre-owned goods space poses a series of advantages for retailers, it sits well with consumer preferences as well, drawing more shoppers, and retaining existing ones.

    Influence of Shopping Behavior on Product Assortment

    Refurbished and pre-owned products provide consumers with attractive alternatives, both in terms of price and variety. Shoppers today explore and research new, pre-owned and refurbished products, all at the same time, and compare prices across e-commerce websites before deciding on a purchase.

    As a result, comprehensive product assortments across price ranges and attributes drive higher engagements, traffic and improve customer conversion and retention rates, as they cater to a more diverse set of consumers.

    For modern retailers, this reinforces the importance of investing in tools that enable to them to identify high-value gaps in their assortment and plug them. To achieve this, they need up-to-date, accurate data, at scale, on the assortments of their competitors.

    DataWeave’s Assortment Intelligence solution is designed to give retailers near-real-time insights on competing retailers’ product mix and suggests product additions to retailer catalogs.

    Click here to know more about how Assortment Intelligence can help your retail business manage assortment efficiently and profitably.