The DataWeave Blog

Tag: analytics for retail industry

Normalizing Size and Color in Fashion Using AI to Power Competitive Price Intelligence
Fashion is as dynamic a market as any—and more competitive than most others. Consumer trends and customer needs are always evolving, making it challenging for fashion and apparel brands to keep up.

Despite the inherent difficulties fashion and apparel sellers face, this industry is one of the largest grossing markets in the world, estimated at $1.79 trillion in 2024. Global revenue for apparel is expected to grow at an annual rate of about 3.3% over the next four years. That means companies in this space stand to make significant revenue if they can competitively price their products, keep up with the competition, and win customer loyalty with consistent product availability.

There are three main categories in fashion and apparel. These include:
- Apparel and clothing (i.e., shirts, pants, dresses, and other apparel)
- Footwear (i.e., sneakers, sandals, heels, and other products)
- Accessories (i.e., bags, belts, watches, and so on)
If you look at all of these product types across all sorts of retailers, there is a massive amount of overlapping data based on product attributes like style and size that are difficult to normalize.

Fashion Attributes

Style, color, and size are the main attribute categories in fashion and apparel. Style attributes include things like design, look, and overall aesthetics of the product. They’re very dependent on the actual product category of fashion as well. A shirt might have a slim fit attribute associated with it, whereas a belt might have a length. All these different attributes are usually labeled within a product listing and affect the consumer’s decision-making process:
- Color (red, blue, sea green, etc.)
- Pattern (solid, striped, checked, floral, etc.)
- Material (cotton, polyester, leather, denim, silk, etc.)
- Fit (regular, slim, relaxed, oversized, tailored, etc.)
- Type (casual, formal, sporty, vintage, streetwear)
Color Complexity in Fashion

Color is perhaps the most visually distinctive attribute in fashion, yet it presents unique challenges for retailers. This is because color naming can vary across retailers and marketplaces. There are several major differences in color convention:
- A single color can be labeled differently across brands (e.g., “navy,” “midnight blue,” “deep blue”)
- Seasonal color names (e.g., “summer sage” vs. “forest green”)
- Marketing-driven names (e.g., “sunset coral” vs. “pale orange”)
Size: The Other Critical Dimension

Size in fashion refers to the dimensions or measurements that determine how fashion products fit. Depending on whether the product is a clothing item, shoes, or a hat, there will be different sizing options. Types of sizes include:
- Standard sizes (XS, S, M, L, XL, XXL, XXL)
- Custom sizes (based on brand, retailer, country, etc.)
A single type of product may have different sizing labels. For instance, one pants listing may use traditional S, M, L, XL sizing, while another pants listing may use 24, 25, or 26, to refer to the waist measurement.

Size is a dynamic attribute that changes based on current trends. For example, there has recently been a significant shift towards inclusive sizing. Size inclusivity refers to the practice of selling apparel in a wide range of sizes to accommodate people of all body types. Consumers are more aware of this trend and are demanding a broader range of sizing offerings from the brands they shop from.

In the US market, in particular, some 67% of American women wear a size 14 or above and may be interested in purchasing plus-size clothing. There is a growing demand in the plus-size market for more options and a wider selection. Many brands are considering expanding their sizes to accommodate more shoppers and tap into this growing revenue channel.

Pricing Based on Size and Color

Many fashion products are priced differently based on size and color. Let’s take a look at an example of what this can look like.

A popular beauty brand (see image) is known for its viral lip tint. While most of the color variants are priced at $9.90 on Amazon, a specific colorway option, featuring less pigmented options, is priced at $9.57. This price differential is driven by both material costs and market demand.

Different colorways (any of a range of combinations of colors in which a style or design is available) of the same product often command different prices also. This is based on:
- Dye costs (some colors require more expensive processes)
- Seasonal demand (traditional colors vs. trend colors)
- Exclusivity (limited edition colors)
An example of price variations by size is a women’s shirt that is being sold on Amazon as shown below. For this product, there are no style attributes to choose from. The only parameter the shopper has to select is the size they’d like to purchase. They can choose from S to XL. On the top, we can see that the product in size S is ₹389. Below, the size XL version of this same shirt is ₹399. This price increase is correlated to the change in size.

So why are these same products priced differently? In an analysis of One Six, a plus-size clothing brand, several reasons for this difference in plus-size clothing were determined.
- Extra material is needed, hence an increase in production costs
- Extra stitching costs, hence an increase in production costs
- Production of plus-size clothing often means acquiring specialized machinery
- Smaller scale production runs for plus-size clothing means these initiatives often don’t benefit from cost savings
Some sizes are sold more than others, meaning that in-demand sizes for certain apparel can affect pricing as well. Brands want to be able to charge as much as possible for their listing without risking losing a sale to a competitor.

The Competitive Pricing Challenge: Normalizing Product Attributes Across Competitors in Apparel and Fashion

There are hundreds of possible attribute permutations for every single apparel product. Some retailers may only sell core sizes and basic colors; some may sell a mix of sizes for multiple style types. Most retailers also sell multiple color variants for all styles they have on catalog. Other retailers may only sell a single, in-demand size of the product. Also, when other retailers are selling the product, it’s unlikely that their naming conventions, color options, style options, and sizing match yours one-for-one.

In one analysis, it was found that there were 800+ unique values for heel sizes and 1000+ unique values for shirts and tops at a single retailer! If you’re looking to compare prices, the effort involved in setting up and managing lookup tables to identify discrepancies when one retailer uses European sizes and another uses USA sizes, for example, is simply too onerous to contemplate doing. Colors only add to the complexity – as similar colors may have new names in different regions and locations as well!

Even if you managed to find all the discrepancies between product attributes, you would still need to update them any time a competitor changed a convention.

Still, monitoring your competitors and strategically pricing your listings is essential to maintain and grow market share. So what do you do? You can’t simply eyeball your competitor’s website to check their pricing and naming conventions. Instead, you need advanced algorithms to scan the entire marketplace, identify individual products being sold, and normalize their data and attributes for analysis.

Getting Color and Size Level Pricing Intelligence

With DataWeave, size and color are just two of several dimensions of a product instead of an impossible big data problem for teams. Our product matching engine can easily handle color and sizing complexity via our AI-driven approach combined with human verification.

This works by using AI built on more than 10 years of product catalog data across thousands of retail websites. It matches common identifiers, like UPC, SKU code, and other attributes for harmonization before employing a large language model (LLM) prompts to normalize color variations and sizing to a single standard.

For example, if a competitor has the smallest size listed as Sm but has your smallest listing identified as S, DataWeave can match those two attributes using AI. Similar classification can be performed on color as well.

Complex LLM prompts are pre-established so that this process is fast and efficient, taking minutes rather than weeks of manual effort.

Harmonizing products along with their color and sizing data across different retailers for further analysis has several benefits. Most importantly, product matching helps teams conduct better competitive analysis, allowing them to stay informed about market trends, competitors’ offerings, and how those competitors are pricing various permutations of the same product. It helps ensure that you’re offering the most competitive assortment of sizing in several colors to win more market share as well. Overall, it’s easier for teams to gain insights and exploit their findings when all the data is clean and available at their fingertips.

Product Matching Size and Color in Apparel and Fashion

Color and size are crucial attributes for retailers and brands in the apparel and fashion industry. It adds a level of complexity that can’t be overstated. While it’s a necessity to win consumers (more colors and sizes will mean a wider potential reach), the more permutations you add to your listing, the more complicated it will be to track it against your competition. However, This challenge is worth undertaking as long as you have the right solutions at your disposal.

With a strategy backed by advanced technology to discover identical and similar products across the competitive landscape and normalize their color and sizing attributes, you can ensure that you are competitively pricing your products and offering the best assortment possible. Employing DataWeave’s AI technology to find competitor listings, match products across variants, and track pricing regularly is the way to go.

Interested in learning more about DataWeave? Click here to get in touch!
November 6, 2024
DataWeave’s AI Evolution: Delivering Greater Value Faster in the Age of AI and LLMs
In retail, competition is fierce, and in its ever-evolving landscape, consumer expectations are higher than ever.

For years, our AI-driven solutions have been the foundation that empowers businesses to sharpen their competitive pricing and optimize digital shelf performance. But in today’s world, evolution is constant—so is innovation. We now find ourselves at the frontier of a new era in AI. With the dawn of Generative AI and the rise of Large Language Models (LLMs), the possibilities for eCommerce companies are expanding at an unprecedented pace.

These technologies aren’t just a step forward; they’re a leap—propelling our capabilities to new heights. The insights are deeper, the recommendations more precise, and the competitive and market intelligence we provide is sharper than ever. This synergy between our legacy of AI expertise and the advancements of today positions DataWeave to deliver even greater value, thus helping businesses thrive in a fast-paced, data-driven world.

This article marks the beginning of a series where we will take you through these transformative AI capabilities, each designed to give retailers and brands a competitive edge.

In this first piece, we’ll offer a snapshot of how DataWeave aggregates and analyzes billions of publicly available data points to help businesses stay agile, informed, and ahead of the curve. These fall into four broad categories:
- Product Matching
- Attribute Tagging
- Content Analysis
- Promo Banner Analysis
- Other Specialized Use Cases
Product Matching

Dynamic pricing is an indispensable tool for eCommerce stores to remain competitive. A blessing—and a curse—of online shopping is that users can compare prices of similar products in a few clicks, with most shoppers gravitating toward the lowest price. Consequently, retailers can lose sales over minor discrepancies of $1–2 or even less.

All major eCommerce platforms compare product prices—especially their top selling products—across competing players and adjust prices to match or undercut competitors. A typical product undergoes 20.4 price changes annually, or roughly once every 18 days. Amazon takes it to the extreme, changing prices approximately every 10 minutes. It helps them maintain a healthy price perception among their consumers.

However, accurate product matching at scale is a prerequisite for the above, and that poses significant challenges. There is no standardized approach to product cataloging, so even identical products bear different product titles, descriptions, and attributes. Information is often incomplete, noisy, or ambiguous. Image data contains even more variability—the same product can be styled using different backgrounds, lighting, orientations, and quality; images can have multiple overlapping objects of interest or extraneous objects, and at times the images and the text on a single page might belong to completely different products!

DataWeave leverages advanced technologies, including computer vision, natural language processing (NLP), and deep learning, to achieve highly accurate product matching. Our pricing intelligence solution accurately matches products across hundreds of websites and automatically tracks competitor pricing data.

Here’s how it works:

Text Preprocessing

It identifies relevant text features essential for accurate comparison.
- Metadata Parsing: Extracts product titles, descriptions, attributes (e.g., color, size), and other structured data elements from Product Description Pages (PDP) that can help in accurately identifying and classifying products.
- Attribute-Value Normalization: Normalize attributes names (e.g. RAM vs Memory) and their values (e.g., 16 giga bytes vs 16 gigs vs 16 GB); brand names (e.g., Benetton vs UCB vs United Colors of Benetton); mapping category hierarchies a standard taxonomy.
- Noise Removal: Removes stop words and other elements with no descriptive value; this focuses keyword extraction on meaningful terms that contribute to product identification.
Image Preprocessing

Image processing algorithms use feature extraction to define visual attributes. For example, when comparing images of a red T-shirt, the algorithm might extract features such as “crew neck,” “red,” or “striped.”

Image hashing techniques create a unique representation (or “hash”) of an image, allowing for efficient comparison and matching of product images. This process transforms an image into a concise string or sequence of numbers that captures its essential features even if the image has been resized, rotated, or edited.

Before we perform these activities there is a need to preprocess images to prepare them for downstream operations. These include object detection to identify objects of interest, background removal, face/skin detection and removal, pose estimation and correction, and so forth.

Embeddings

We have built a hybrid or a multimodal product-matching engine that uses image features, text features, and domain heuristics. For every product we process we create and store multiple text and image embeddings in a vector database. These include a combination of basic feature vectors (e.g. tf-idf based, colour histograms, share vectors) to more advanced deep learning algorithms-based embeddings (e.g., BERT, CLIP) to the latest LLM-based embeddings.

Classification

Classification algorithms enhance product attribute tagging by designating match types. For example, the product might be identified as an “exact match”, “variant”, “similar”, or “substitute.” The algorithm can also identify identical product combinations or “baskets” of items typically purchased together.

What is the Business Impact of Product Matching?
- Pricing Intelligence: Businesses can strategically adjust pricing to remain competitive while maintaining profitability. High-accuracy price comparisons help businesses analyze their competitive price position, identify opportunities to improve pricing, and reclaim market share from competitors.
- Similarity-Based Matching: Products are matched based on a range of similarity features, such as product type, color, price range, specific features, etc., leading to more accurate matches.
- Counterfeit Detection: Businesses can identify counterfeit or unauthorized versions of branded products by comparing them against authentic product listings. This helps safeguard brand identity and enables brands to take legal action against counterfeiters.
Attribute Tagging

Attribute tagging involves assigning standardized tags for product attributes, such as brand, model, size, color, or material. These naming conventions form the basis for accurate product matching. Tagging detailed attributes, such as specifications, features, and dimensions, helps match products that meet similar criteria. For example, tags like “collar” or “pockets” for apparel ensure high-fidelity product matches for hard-to-distinguish items with minor stylistic variations.

Including tags for synonyms, variants, and long-tail keywords (e.g., “denim” and “jeans”) improves the matching process by recognizing different terms used for similar products. Metadata tags categorize similar items according to SKU numbers, manufacturer details, and other identifiers.

Altogether, these capabilities provide high-quality product matches and valuable metadata for retailers to classify their products and compare their product assortment to competitors.

User-Generated Content (UGC) Analysis

Customer reviews and ratings are rich sources of information, enabling brands to gauge consumer sentiment and identify shortcomings regarding product quality or service delivery. However, while informative, reviews constitute unstructured “noisy” data that is actionable only if parsed correctly.

Here’s where DataWeave’s UGC analysis capability steps in.
- Feature Extractor: Automatically pulls specific product attributes mentioned in the review (e.g., “battery life,” “design” and “comfort”)
- Feature Opinion Pair: Pairs each product attribute with a corresponding sentiment from the review (e.g., “battery life” is “excellent,” “design” is “modern,” and “comfort” is “poor”)
- Calculate Sentiment: Calculates an overall sentiment score for each product attribute
The final output combines the information extracted from each of these features, which looks something like this:
- Battery life is excellent
- Design is modern
- Not satisfied with the comfort
The algorithm also recognizes spammy reviews and distinguishes subjective reviews (i.e., those fueled by emotion) from objective ones.

Promo Banner Analysis

Our image processing tool can interpret promotional banners and extract information regarding product highlights, discounts, and special offers. This provides insights into pricing strategies and promotional tactics used by other online stores.

For example, if a competitor offers a 20% discount on a popular product, you can match or exceed this discount to attract more customers.

The banner reader identifies successful promotional trends and patterns from competitors, such as the timing of discounts, frequently promoted product categories or brands, and the duration of sales events. Ecommerce stores can use this information to optimize their promotion strategies, ensuring they launch compelling and timely offers.

Other Specialized Use Cases

While these generalized AI tools are highly useful in various industries, we’ve created other category—and attribute-specific capabilities for specialty goods (e.g., those requiring certifications or approval by federal agencies) and food items. These use cases help our customers adhere to compliance requirements.

Certification Mark Detector

This detector lets retailers match items based on official certification marks. These marks represent compliance with industry standards, safety regulations, and quality benchmarks.

Example:
- USDA Organic: Certification for organic food production and handling
- ISO 9001: Quality Management System Certification
By detecting these certification marks, the system can accurately match products with their certified counterparts. By identifying which competitor products are certified, retailers can identify products that may benefit from certification.

Nutrition Fact Table Reader

Product attributes alone are insufficient for comparing food items. Differences in nutrition content can influence product category (e.g., “health food” versus regular food items), price point, and consumer choice. DataWeave’s nutrition fact table reader scans nutrition information on packaging, capturing details such as calorie count, macronutrient distribution (proteins, fats, carbohydrates), vitamins, and minerals.

The solution ensures items with similar nutritional profiles are correctly identified and grouped based on specific dietary requirements or preferences. This helps with price comparisons and enables eCommerce stores to maintain a reliable database of product information and build trust among health-conscious consumers.

Building Next-Generation Competitive and Market Intelligence

Moving forward, breakthroughs in generative AI and LLMs have fueled substantial innovation, which has enabled us to introduce powerful new capabilities for our customers.

These include:
- Building Enhanced Products, Solutions, and Capabilities: Generative AI and LLMs can significantly elevate the performance of existing solutions by improving the accuracy, relevance, and depth of insights. By leveraging these advanced AI technologies, DataWeave can enhance its product offerings, such as pricing intelligence, product matching, and sentiment analysis. These tools will become more intuitive, allowing for real-time updates and deeper contextual understanding. Additionally, AI can help create entirely new solutions tailored to specific use cases, such as automating competitive analysis or identifying emerging market trends. This positions DataWeave to remain at the forefront of innovation, offering cutting-edge solutions that meet the evolving needs of retailers and brands.
- Reducing Turnaround Time (TAT) to Go-to-Market Faster: Generative AI and LLMs streamline data processing and analysis workflows, enabling faster decision-making. By automating tasks like data aggregation, sentiment analysis, and report generation, AI dramatically reduces the time required to derive actionable insights. This efficiency means that businesses can respond to market changes more swiftly, adjusting pricing or promotional strategies in near real-time. Faster insights translate into reduced turnaround times for product development, testing, and launch cycles, allowing DataWeave to bring new solutions to market quickly and give clients a competitive advantage.
- Improving Data Quality to Achieve Higher Performance Metrics: AI-driven technologies are exceptionally skilled at cleaning, organizing, and structuring large datasets. Generative AI and LLMs can refine the data input process, reducing errors and ensuring more accurate, high-quality data across all touchpoints. Improved data quality enhances the precision of insights drawn from it, leading to higher performance metrics like better product matching, more accurate price comparisons, and more effective consumer sentiment analysis. With higher-quality data, businesses can make smarter, more informed decisions, resulting in improved revenue, market share, and customer satisfaction.
- Augmenting Human Bandwidth with AI to Enhance Productivity: Generative AI and LLMs serve as powerful tools that augment human capabilities by automating routine, time-consuming tasks such as data entry, classification, and preliminary analysis. This allows human teams to focus on more strategic, high-value activities like interpreting insights, building relationships with clients, and developing new business strategies. By offloading these repetitive tasks to AI, human productivity is significantly enhanced. Employees can achieve more in less time, increasing overall efficiency and enabling teams to scale their operations without needing a proportional increase in human resources.
In our ongoing series, we will dive deep into each of these capabilities, exploring how DataWeave leverages cutting-edge AI technologies like Generative AI and LLMs to solve complex challenges for retailers and brands.

In the meantime, talk to us to learn more!
September 18, 2024
Using Siamese Networks to Power Accurate Product Matching in eCommerce
Retailers often compete on price to gain market share in high performance product categories. Brands too must ensure that their in-demand assortment is competitively priced across retailers. Commerce and digital shelf analytics solutions offer competitive pricing insights at both granular and SKU levels. Central to this intelligence gathering is a vital process: product matching.

Product matching or product mapping involves associating identical or similar products across diverse online platforms or marketplaces. The matching process leverages the capabilities of Artificial Intelligence (AI) to automatically create connections between various representations of identical or similar products. AI models create groups or clusters of products that are exactly the same or “similar” (based on some objectively defined similarity criteria) to solve different use cases for retailers and consumer brands.

Accurate product matching offers several key benefits for brands and retailers:
- Competitive Pricing: By identifying identical products across platforms, businesses can compare prices and adjust their strategies to remain competitive.
- Market Intelligence: Product matching enables brands to track their products’ performance across various retailers, providing valuable insights into market trends and consumer preferences.
- Assortment Planning: Retailers can analyze their product range against competitors, identifying gaps or opportunities in their offerings.
Why Product Matching is Incredibly Hard

But product matching stands out as one of the most demanding technical processes for commerce intelligence tools. Here’s why:

Data Complexity

Product information comes in various (multimodal) formats – text, images, and sometimes video. Each format presents its own set of challenges, from inconsistent naming conventions to varying image quality.

Data Variance

The considerable fluctuations in both data quality and quantity across diverse product categories, geographical regions, and websites introduce an additional layer of complexity to the product matching process.

Industry Specific Nuances

Industry specific nuances introduce unique challenges to product matching. Exact matching may make sense in certain verticals, such as matching part numbers in industrial equipment or identifying substitute products in pharmaceuticals. But for other industries, exactly matched products may not offer accurate comparisons.
- In the Fashion and Apparel industry, style-to-style matching, accommodating variants and distinguishing between core sizes and non-core sizes and age groups become essential for accurate results.
- In Home Improvement, the presence of unbranded products, private labels, and the preference for matching sets rather than individual items complicates the process.
- On the other hand, for grocery, product matching becomes intricate due to the distinction between item pricing and unit pricing. Managing the diverse landscape of different pack sizes, quantities, and packaging adds further layers of complexity.
Diverse Downstream Use Cases

The diverse downstream business applications give rise to various flavors of product matching tailored to meet specific needs and objectives.

In essence, while product matching is a critical component in eCommerce, its intricacies demand sophisticated solutions that address the above challenges.

To solve these challenges, at DataWeave, we’ve developed an advanced product matching system using Siamese Networks, a type of machine learning model particularly suited for comparison tasks.

Siamese Networks for Product Matching

Our methodology involves the use of ensemble deep learning architectures. In such cases, multiple AI models are trained and used simultaneously to ensure highly accurate matches. These models tackle NLP (natural language processing) and Computer Vision challenges specific to eCommerce. This technology helps us efficiently narrow down millions of product candidates to just 5-15 highly relevant matches.

The Tech Powering Siamese Networks

The key to our approach is creating what we call “embeddings” – think of these as unique digital fingerprints for each product. These embeddings are designed to capture the essence of a product in a way that makes similar products easy to identify, even when they look slightly different or have different names.

Our system learns to create these embeddings by looking at millions of product pairs. It learns to make the embeddings for similar products very close to each other while keeping the embeddings for different products far apart. This process, known as metric learning, allows our system to recognize product similarities without needing to put every product into a rigid category.

This approach is particularly powerful for eCommerce, where we often need to match products across different websites that might use different names or images for the same item. By focusing on the key features that make each product unique, our system can accurately match products even in challenging situations.

How Siamese Networks Work?

Imagine having a pair of identical twins who are experts at spotting similarities and differences. That’s essentially what a Siamese network is – a pair of identical AI systems working together to compare things.

How it works:
- Twin AI systems: Two identical AI systems look at two different products.
- Creating ‘fingerprints’ or ‘embedding’: Each system creates a unique ‘fingerprint’ of the product it’s looking at.
- Comparison: These ‘fingerprints’ are then compared to see how similar the products are.
Architecture

The architecture of a Siamese network typically consists of three main components: the shared network, the similarity metric, and the contrastive loss function.
- Shared Network: This is the ‘brain’ that creates the product ‘fingerprints’ or ‘embeddings.’ It is responsible for extracting meaningful feature representations from the input samples. This network is composed of layers of neural units that work together. Weight sharing between the twin networks ensures that the model learns to extract comparable features for similar inputs, providing a basis for comparison.
- Similarity Metric: After the shared network processes the inputs, a similarity metric is employed. This decides how alike two ‘fingerprints’ or ‘embeddings’ are. The selection of a similarity metric depends on the specific task and characteristics of the input data. Frequently used similarity metrics include the Euclidean distance, cosine similarity, or correlation coefficient, each chosen based on its suitability for the given context and desired outcomes.
- Loss Function: For training the Siamese network, a specialized loss function is used. This helps the system improve its comparison skills over time. It guides and trains the network to generate akin embeddings for similar inputs and disparate embeddings for dissimilar inputs.
  
  This is achieved by imposing penalties on the model when the distance or dissimilarity between similar pairs surpasses a designated threshold, or when the distance between dissimilar pairs falls below another predefined threshold. This training strategy ensures that the network becomes adept at discerning and encoding the desired level of similarity or dissimilarity in its learned embeddings.
How DataWeave Uses Siamese Networks for Product Matching

At DataWeave, we use Siamese Networks to match products across different retailer websites. Here’s how it works:

Pre-processing (Image Preparation)
- We collect product images from various websites.
- We clean these images up to make them easier for our AI to understand.
- We use techniques like cropping, flipping, and adjusting colors to help our AI recognize products even if the images are slightly different.
Training The AI
- We show our AI system millions of product images, teaching it to recognize similarities and differences.
- We use a special learning method called “Triplet Loss” to help our AI understand which products are the same and which are different.
- We’ve tested different AI structures to find the one that works best for product matching, including ResNet, EfficientNet, NFNet, and ViT.
Image Retrieval
- Once trained, our AI creates a unique “fingerprint” for each product image.
- We store these fingerprints in a smart database.
- When we need to find a match for a product, we:
  - Create a fingerprint for the new product.
  - Quickly search our database for the most similar fingerprints.
  - Return the top matching products.
Matches are then assigned a high or a low similarity score and segregated into “Exact Matches” or “Similar Matches.” For example, check out the image of this white shoe on the left. It has a low similarity score with the pink shoe (below) and so these SKUs are categorized as a “Similar Match.” Meanwhile, the shoe on the right is categorized as an “Exact Match.”

Similarly, in the following image of the dress for a young girl, the matched SKU has a high similarity score and so this pair is categorized as an “Exact Match.”

Siamese Networks play a pivotal role in DataWeave’s Product Matching Engine. Amid the millions of images and product descriptions online, our Siamese Networks act as an equalizing force, efficiently narrowing down millions of candidates to a curated selection of 10-15 potential matches.

In addition, these networks also find application in several other contexts at DataWeave. They are used to train our system to understand text-only data from product titles and joint multimodal content from product descriptions.

Leverage Our AI-Driven Product Matching To Get Insightful Data

In summary, accurate and efficient product matching is no longer a luxury – it’s a necessity. DataWeave’s advanced product matching solution provides brands and retailers with the tools they need to navigate this complex landscape, turning the challenge of product matching into a competitive advantage.

By leveraging cutting-edge technology and simplifying it for practical use, we empower businesses to make informed decisions, optimize their operations, and stay ahead in the ever-evolving eCommerce market. To learn more, reach out to us today!
June 26, 2024
How DataWeave Enhances Transparency in Competitive Pricing Intelligence for Retailers
Retailers heavily depend on pricing intelligence solutions to consistently achieve and uphold their desired competitive pricing positions in the market. The effectiveness of these solutions, however, hinges on the quality of the underlying data, along with the coverage of product matches across websites.

As a retailer, gaining complete confidence in your pricing intelligence system requires a focus on the trinity of data quality:
- Accuracy: Accurate product matching ensures that the right set of competitor product(s) are correctly grouped together along with yours. It ensures that decisions taken by pricing managers to drive competitive pricing and the desired price image are based on reliable apples-to-apples product comparisons.
- Freshness: Timely data is paramount in navigating the dynamic market landscape. Up-to-date SKU data from competitors enables retailers to promptly adjust pricing strategies in response to market shifts, competitor promotions, or changes in customer demand.
- Product matching coverage: Comprehensive product matching coverage ensures that products are thoroughly matched with similar or identical competitor products. This involves accurately matching variations in size, weight, color, and other attributes. A higher coverage ensures that retailers seize all available opportunities for price improvement at any given time, directly impacting revenues and margins.
However, the reality is that untimely data and incomplete product matches have been persistent challenges for pricing teams, compromising their pricing actions. Inaccurate or incomplete data can lead to suboptimal decisions, missed opportunities, and reduced competitiveness in the market.

What’s worse than poor-quality data? Poor-quality data masquerading as accurate data.

In many instances, retailers face a significant challenge in obtaining comprehensive visibility into crucial data quality parameters. If they suspect the data quality of their provider is not up to the mark, they are often compelled to manually request reports from their provider to investigate further. This lack of transparency not only hampers their pricing operations but also impedes the troubleshooting process and decision-making, slowing down crucial aspects of their business.

We’ve heard about this problem from dozens of our retail customers for a while. Now, we’ve solved it.

DataWeave’s Data Statistics and SKU Management Capability Enhances Data Transparency

DataWeave’s Data Statistics Dashboard, offered as part of our Pricing Intelligence solution, enables pricing teams to gain unparalleled visibility into their product matches, SKU data freshness, and accuracy.

It enables retailers to autonomously assess and manage SKU data quality and product matches independently—a crucial aspect of ensuring the best outcomes in the dynamic landscape of eCommerce.

Beyond providing transparency and visibility into data quality and product matches, the dashboard facilitates proactive data quality management. Users can flag incorrect matches and address various data quality issues, ensuring a proactive approach to maintaining the highest standards.

Retailers can benefit in several ways with this dashboard, as listed below.

View Product Match Rates Across Websites

The dashboard helps retailers track match rates to gauge their health. High product match rates signify that pricing teams can move forward in their pricing actions with confidence. Low match rates would be a cause for further investigation, to better understand the underlying challenges, perhaps within a specific category or competitor website.

Our dashboard presents both summary statistics on matches and data crawls as well as detailed snapshots and trend charts, providing users with a holistic and detailed perspective of their product matches.

Additionally, the dashboard provides category-wise snapshots of reference products and their matching counterparts across various retailers, allowing users to focus on areas with lower match rates, investigate underlying reasons, and develop strategies for speedy resolution.

Track Data Freshness Easily

The dashboard enables pricing teams to monitor the timeliness of pricing data and assess its recency. In the dynamic realm of eCommerce, having up-to-date data is essential for making impactful pricing decisions. The dashboard’s presentation of freshness rates ensures that pricing teams are armed with the latest product details and pricing information across competitors.

Within the dashboard, users can readily observe the count of products updated with the most recent pricing data. This feature provides insights into any temporary data capture failures that may have led to a decrease in data freshness. Armed with this information, users can adapt their pricing decisions accordingly, taking into consideration these temporary gaps in fresh data. This proactive approach ensures that pricing strategies remain agile and responsive to fluctuations in data quality.

Proactively Manage Product Matches

The dashboard provides users with proactive control over managing product matches within their current bundles via the ‘Data Management’ panel. This functionality empowers users to verify, add, flag, or delete product matches, offering a hands-on approach to refining the matching process. Despite the deployment of robust matching algorithms that achieve industry-leading match rates, occasional instances may arise where specific matches are overlooked or misclassified. In such cases, users play a pivotal role in fine-tuning the matching process to ensure accuracy.

The interface’s flexibility extends to accommodating product variants and enables users to manage product matches based on store location. Additionally, the platform facilitates bulk match uploads, streamlining the process for users to efficiently handle large volumes of matching data. This versatility ensures that users have the tools they need to navigate and customize the matching process according to the nuances of their specific product landscape.

Gain Unparalleled Visibility into your Data Quality

With DataWeave’s Pricing Intelligence, users gain the capability to delve deep into their product data, scrutinize match rates, assess data freshness, and independently manage their product matches. This approach is instrumental in fostering informed and effective decisions, optimizing inventory management, and securing a competitive edge in the dynamic world of online retail.

To learn more, reach out to us today!
January 22, 2024
Why Unit of Measure Normalization is Critical For Accurate and Actionable Competitive Pricing Intelligence

Competitive pricing intelligence is pivotal for retailers seeking to analyze their product pricing in relation to competitors. This practice is essential for ensuring that their product range maintains a competitive edge, meeting both customer expectations and market demands consistently.

Product matching serves as a foundational element within any competitive pricing intelligence solution. Products are frequently presented in varying formats across different websites, featuring distinct titles, images, and descriptions. Undertaking this process at a significant scale is highly intricate due to numerous factors. One such complication arises from the fact that products are often displayed with differing units of measurement on various websites.

The Challenge of Varying Units

In certain product categories, retailers often offer the same item in varying volumes, quantities, or weights. For instance, a clothing item might be available as a single piece or in packs of 2 or 3, while grocery brands commonly sell eggs in counts of 6, 12, or 24.

Consider this example: a quick glance might suggest that an 850g pack of Kellogg’s Corn Flakes priced at $5 is a better deal than a 980g pack of Nestle Cornflakes priced at $5.2. However, this assumption can be deceptive. In reality, the latter offers better value for your money, a fact that only becomes evident through price comparisons after standardizing the units.

This issue is particularly relevant due to the prevalence of “shrinkflation,” where brands adjust packaging sizes or quantities to offset inflation while keeping prices seemingly low. When quantities, pack sizes, weight, etc. reduce instead of prices increasing, it’s important that this change is considered while analyzing competitive pricing.

Normalizing Units of Measure

In order to effectively compare prices among different competitors, retailers must standardize the diverse units of measurement they encounter. This standardization (or normalization) is crucial because price comparisons should extend beyond individual product SKUs to accommodate variations in package sizes and quantities. It’s essential to normalize units, ranging from “each” (ea) for individual items to “dozen” (dz) for sets, and from “pounds” (lb), “kilograms” (kg), “liters” (ltr), to “gallons” (gal) for various product types.

For example, a predetermined base unit of measure, such as 100 grams for a specific product like cornflakes, serves as the reference point. The unit-normalized price for any cornflake product would then be the price per 100 grams. In the example provided, this reveals that Kellogg’s is priced at $0.59 per 100 grams, while Nestle is priced at $0.53 per 100 grams.

Various Categories of Unit Normalization

1. Weight Normalization

Retailers frequently feature products with weight measurements expressed in grams (g), kilograms (kg), pounds (lbs), or ounces (oz).

2. Quantity or Pack Size Normalization

Products are also often featured with varying pick sizes or quantities in each SKU.

3. Volume or Capacity Normalization

Products can also vary in volumes or capacities with units like liters (L) or fluid ounces (fl oz).

DataWeave’s Unit Normalized Pricing Intelligence Solution

DataWeave’s highly sophisticated product matching engine can match the same or similar products and normalize their units of measurement, leading to highly accurate and actionable competitive pricing insights. It standardizes different units of measurement, like weight, quantity, and volume, ensuring fair comparisons across similar and exact matched products.

Retailers have the flexibility to view pricing insights either with retailer units or normalized units. This capability empowers retailers and analysts to perform accurate, in-depth analyses of pricing information at a product level.

In some scenarios, analyzing unit normalized pricing reflects pricing trends and competitiveness more accurately than retail price alone. This is particularly true for categories like CPG, where products are sold in diverse units of measure. For instance, in the example shown here, we can view a comparison of price position trends for the category of Fruits and Vegetables based on both retail price and unit price.

The difference is striking: the original retail price based analysis shows a stagnation in price position, whereas unit normalized pricing analysis reflects a more dynamic pricing scenario.

With DataWeave, retailers can specify which units to compare, ensuring that comparisons are made accurately. For example, a retailer can specify that unit price comparisons apply only to 8, 12, or 16-ounce packs, as well as 1 or 3-pound packs, but not to 10 and 25-pound bags. This precision ensures that products are matched correctly, and prices are represented for appropriately normalized units, leading to more accurate pricing insights.

To learn more about this capability, write to us at contact@dataweave.com or visit our website today!

November 6, 2023
Top 10 Retail Analytics that You Must Know

Customers expect personalization. Unless they have a seamless experience on your online channels, they’ll leave for a different retailer. Retail analytics can solve these problems for merchants looking to increase customer satisfaction and sales. It provides insights into inventory, sales, customers, and other essential aspects crucial for decision-making. Retail analytics also encompasses several granular fields to create a broad picture of a retail business’s health and sales, along with improvement areas.

Big data analytics in the retail market

Big data analytics in the retail market is expected to reach USD 13.26 billion by the end of 2026, registering a CAGR of 21.20% during the forecast period (2021-2026). The growth of analytics in retail depicts how it can help companies run businesses more efficiently, make data-backed choices, and deliver improved customer service.

In this blog, we’ll discuss the top 10 analytics that retailers are using to gain a competitive advantage in accurately evaluating business & market performance.

Top 10 of Retail Analytics You Must Know

1. Assortment

Assortment planning allows retailers to choose the right breadth (product categories) and depth (product variation within each category) for their retail or online stores. Assortment management has grown beyond simple performance metrics like total sales or rotation numbers. Instead, retail analytics offers a comprehensive analysis of product merchandise and an estimated number of units at the push of a button. Retailers that effectively apply assortment analytics can enjoy increased gross margins and prevent significant losses from overstocks sold at discounted prices or out-of-stock inventory leading their customers to buy from competitors.

It also helps retailers gain insights into the trendy and discoverable brands and products on all e-commerce websites across the globe. They can boost sales by making sure they have an in-demand product assortment. They can also track pricing information and attributes common across popular products to drive their pricing and promotion strategies.

2. Inventory Management

An inadequately maintained inventory is every retailer’s worst nightmare. It represents a poor indicator of inadequate demand for a product and leads to a loss in sales. Data can help companies answer issues like what to store and what to discard. It’s beneficial to discard or increase offers on products that are not generating sales and keep replenished stocks of popular items.

In 2020, the estimated value for out-of-stock items ($1.14 trillion) was double that of overstock items ($626 billion). A similar trend was especially prominent in grocery stores, where out-of-stock items were worth five times more than overstock items.

Unavailability of high-selling products can lead to reduced sales, ultimately generating incorrect data for future forecasting and producing skewed demand and supply insights. Retailers can now use analytics to identify which products are in demand, which are moving slowly, and which ones contribute to dead stock. They can know in real-time if a high-demand product is unavailable at a specific location and take action to increase the stock. Retailers can use this historical data to predict what to stock, at what place, time, and cost to maintain and optimize revenue. It helps satisfy consumer needs, prevents loss of sales, reduces inventory cost, and streamlines the complete supply chain.

3. Competitive Intelligence

Market intelligence & Competitive Insights

The ability to accurately predict trends after the global pandemic and with an unknown economic future is becoming the cornerstone for successful retailers. Smart retailers know how important it is to Pandemic-Proof their retail strategy with Market Intelligence & Competitive Insights

With 90% of Fortune 500 companies using competitive intelligence, it’s an essential tool to gain an advantage over industry competitors. Competitive Intelligence allows you to gather and analyze information about your competitors and understand the market–providing valuable insights that you can apply to your own business. A more strategic competitor analysis will explain brand affinities and provide insights on what to keep in stock and when to start promotions. Customer movement data will also give you access to where your customers are shopping.

4. Fraud Detection

Fraud Detection

Retailers have been in a constant struggle with fraud detection and prevention since time immemorial. Fraudulent products lead to substantial financial losses and damage the reputation of both brands and retailers. Every $1 of fraud now costs U.S. retail and eCommerce merchants $3.60, a 15% growth since the pre-Covid study in 2019, which was $3.13. Retail Analytics acts as a guardian against fraudsters by constantly monitoring, identifying, and flagging fraud products and sellers.

5. Campaign Management

Some of the challenges of the retail industry are that it’s seasonal, promotion-based, highly competitive, and fast-moving. In today’s competitive marketplace, consumers compare prices and expect personalized shopping experiences. Campaign management allows marketing teams to plan, track, and analyze marketing strategies for promoting products and attracting audiences. Retail analytics can help businesses predict consumer behavior, improve decision-making across the company, and determine the ROI of their marketing efforts.

According to Invesp, 64% of marketing executives “strongly agree” that data-driven marketing is crucial in the economy. Retail analytics can help businesses analyze their data to learn about their customers with target precision. With predictive analysis, retailers can design campaigns that encourage consumers to interact with the brand, move down the sales funnel, and ultimately convert.

6. Behavioral Analytics

Retail firms often look to improve customer conversion rates, personalize marketing campaigns to increase revenue, predict and avoid customer churn, and lower customer acquisition costs. Data-driven insights on customer shopping behaviors can help companies tackle these challenges. However, several interaction points like social media, mobile, e-commerce sites, stores, and more, cause a substantial increase in the complexity and diversity of data to accumulate and analyze.

Insider Intelligence forecasts that m-eCommerce volume will rise at 25.5% (CAGR) until 2024, hitting $488 billion in sales, or 44% of all e-commerce transactions.

Data can provide valuable insights, for example, recognizing your high-value customers, their motives behind the purchase, their buying patterns, behaviors, and which are the best channels to market to them and when. Having these detailed insights increases the probability of customer acquisition and perhaps drives their loyalty towards you.

7. Pricing

Competitive pricing in retail

Market trends fluctuate at an unprecedented pace, and pricing has become as competitive as it’s ever been. The only way to keep up with competitive pricing in retail is to use retail analytics that enables retailers to drive more revenue & margin by pricing products competitively.

A report from Inside Big Data found companies experience anywhere from 0.5% up to 17.1% in margin loss purely because of pricing errors. Pricing analytics provides companies with the tools and methods to perceive better, interpret and predict pricing that matches consumer behavior. Appropriate pricing power comes from understanding what your consumers want, which offers they respond to, how and where they shop, and how much they will pay for your products.

In 2021, the price optimization segment is anticipated to own the largest share of the overall retail analytics market. Retailers can identify gaps and set alerts to track changes across crucial SKUs or products with pricing analytics. Knowing your customer’s price perception will increase sales and also allow you to design promotions that’ll attract customers. Pricing analytics also accounts for factors like demographics, weather forecasting, inventory levels, real-time sales data, product movement, purchase history, and much more to arrive at an excellent price.

8. Sales and Demand Forecasting

Sales and demand forecasting allow retailers to plan for levels of granularity—monthly, weekly, daily, or even hourly—and use the insights in their marketing campaigns and business decisions. The benefits of a granular forecast are apparent since retailers don’t have to bank on historical data of previous clients and customers to predict revenues. Retailers can plan their strategies and promotions that suit their customer’s demands.

With sales and demand forecasting, retailers can also consider the most recent, historical, and real-time data to predict potential future revenue. Sales and demand analytics can predict buying patterns and market trends based on socioeconomic and demographic conditions.

9. Customer Service and Experience

With the development of eCommerce, more and more customers prefer to browse and interact with the product before purchasing online. They look for better deals and discounts across stores and platforms. 3 out of 5 consumers say retail’s investment in technology is improving their online and in-store shopping experiences. To enhance merchandising and marketing strategies, retailers can gather data on customer buying journeys to understand their in-store and online experiences.

Retailers can run test campaigns to know the impact on sales and use historical data to predict consumers’ needs based on their demographics, buying patterns, and interests. Retail analytics help retailers to bring more efficiency in promotions and drive impulsive purchases and cross-selling.

10. Promotion

Analyze Competitors’ Promotions

Promotions are potent sales drivers and need to be cleverly targeted towards specific customers with precise deals to generate outstanding sales. Retail analytics allows companies to study their customers and competitors to a vastly elevated level.

To be an industry leader, retail companies not only have to understand their customers, but they must also analyze competitors’ promotions to improve their marketing strategies. Analyzing your competitor’s promotional banners, ads, and marketing campaigns are no more associated with imitation.

With data analytics and AI, retailers can watch their competitors’ commercialization strategies. It can uncover vital information about their target audience, sales volume fluctuations, popular seasonal product types, product attributes of popular items, and significant industry trends. Knowing exactly which products and brands are popular among your competitor’s campaigns can help retailers improve their promotional strategies.

Conclusion

The benefits of retail analytics are spread across various verticals, from merchandising, assortment, inventory management, and marketing to reducing losses. The need for analytics has become even more apparent considering the growing eCommerce platforms, changing customer buying journeys, and the complexity of the industry. Understanding which products sell best among which customers will help retailers to deliver an optimized shopping experience.

Want to drive profitable growth by making smarter pricing, promotions, and product merchandising decisions using real-time retail insights? DataWeave’s AI-powered Competitive Intelligence can help! Reach out to our Retail Analytics experts to know more.

November 19, 2021