The DataWeave Blog

Tag: AI

Maximizing Competitive Match Rates: The Foundation of Effective Price Intelligence
Merchants make countless pricing decisions every day. Whether you’re a brand selling online, a traditional brick-and-mortar retailer, or another seller attempting to navigate the vast world of commerce, figuring out the most effective price intelligence strategy is essential. Having your plan in place will help you price your products in the sweet spot that enhances your price image and maximizes profits.

For the best chance of success, your overall pricing strategy must include competitive intelligence.

Many retailers focus their efforts on just collecting the data. But that’s only a portion of the puzzle. The real value lies in match accuracy and knowing exactly which competitor products to compare against. In this article, we will dive deeper into cutting-edge approaches that combine the traditional matching techniques you already leverage with AI to improve your match rates dramatically.

If you’re a pricing director, category manager, commercial leader, or anyone else who deals with pricing intelligence, this article will help you understand why competitive match rates matter and how you can improve yours.

Change your mindset from tactical to strategic and see the benefits in your bottom line.

The Match Rate Challenge

To the layman, tracking and comparing prices against the competition seems easy. Just match up two products and see which ones are the same! In reality, it’s much more challenging. There are thousands of products to discover, analyze, compare, and derive subjective comparisons from. Not only that, product catalogs across the market are constantly evolving and growing, so keeping up becomes a race of attrition with your competitors.

Let’s put it into focus. Imagine you’re trying to price a 12-pack of Coca-Cola. This is a well-known product that, hypothetically, should be easy to identify across the web. However, every retailer uses their own description in their listing. Some examples include:
- Retailer A lists it as “Coca-Cola 12 Fl. Oz 12 Pack”
- Retailer B shows “Coca Cola Classic Soda Pop Fridge Pack, 12 Fl. Oz Cans, 12-Pack”
- Retailer C has “Coca-Cola Soda – 12pk/12 fl oz Cans”
While a human can easily deduce that these are the same product, the automated system you probably have in place right now is most likely struggling. It cannot tell the difference between the retailers’ unique naming conventions, including brand name, description, bundle, unit count, special characters, or sizing.

This has real-world business impacts if your tools cannot accurately compare the price of a Coca-Cola 12-pack across the market.

Why Match Rates Matter

If your competitive match rates are poor, you aren’t seeing the whole picture and are either overcharging, undercharging, or reacting to market shifts too slowly.

Overcharging can result in lost sales, while undercharging may result in out-of-stock due to spikes in demand you haven’t accounted for. Both are recipes to lose out on potential revenue, disappoint customers, and drive business to your competitors.

What you need is a sophisticated matching capability that can handle the tracking of millions of competitive prices each week. It needs to be able to compare using hundreds of possible permutations, something that is impossible for pricing teams to do manually, especially at scale. With technology to make this connection, you aren’t missing out on essential competitive intelligence.

The Business Impact

Besides the bottom-line savings, accurately matching competitor products for pricing intelligence has other business impacts that can help your business. Adding technology to your workflow to improve match rates can help identify blind spots, improve decision quality, and improve operational efficiency.
- Pricing Blind Spots
  - Missing competitor prices on key products
  - Inability to detect competitive threats
  - Delayed response to market changes
- Decision Quality
  - Incomplete competitive coverage leads to suboptimal pricing
  - Risk of pricing decisions based on wrong product comparisons
- Operational Efficiency
  - Manual verification costs
  - Time spent reconciling mismatched products
  - Resources needed to maintain price position
Current Industry Challenges

As mentioned, the #1 reason businesses like yours probably aren’t already finding the most accurate matches is that not all sites carry comparable product codes. If every listing had a consistent product code, it would be very easy to match that code to your code base. In fact, most retailers currently only achieve 60-70% match rates using their traditional methods.

Different product naming conventions, constantly changing product catalogs, and regional product variations contribute to the industry challenges, not to mention the difficulty of finding brand equivalencies and private label comparisons across the competition. So, if you’re struggling, just know everyone else is as well. However, there is a significant opportunity to get ahead of your competition if you can improve your match rates with technology.

The Matching Hierarchy
- Direct Code Matching: There are a number of ways to start finding matches across the market. The base tier of the hierarchy of most accurate approaches is Direct Code matching. Most likely, your team already has a process in place that can compare UPC to UPC, for example. When no standard codes are listed, your team is left with a blind spot. This poses limitations in modern retail but is an essential first step to identifying the “low-hanging fruit” to start getting matches.
- Non-Code-Based Matching: The next level of the hierarchy is implementing non-code-based matching strategies. This is when there are no UPCs, DPCIs, ASINs, or other known codes that make it easy to do one-to-one comparisons. These tools can analyze complex metrics like direct size comparisons, unique product descriptions, and features to find more accurate matches. They can look deep into the listing to extract data points beyond a code, even going as far as analyzing images and video content to help find matches. Advanced technologies for competitive matching can help pricing teams by adding different comparison metrics to their arsenal beyond code-based.
- Private Label Conversions: Up until this level of the hierarchy, comparisons relied on direct comparisons. Finding identical codes and features and naming similarities is excellent for figuring out one-to-one comparisons, but when there is no similar product to compare with for pricing intelligence, things get more complicated. This is the third tier of the matching hierarchy. It’s the ability to find similar product matches for ‘like’ products. This can be used for private label conversions and to create meaningful comparisons without direct matches.
- Similar Size Mappings: This final rung on the matching hierarchy adds another layer of advanced calculations to the comparison capability. Often, retailers and merchants list a product with different sizing values. One may choose to bundle products, break apart packs to sell as single items or offer a special-sized product manufactured just for them.
While at the end of the day, the actual product is the same, when there are unusual size permutations, it can be hard to identify the similarities. Technology can help with value size relationships, package variation handling, size equalization, and unit normalization.

The AI Advantage

AI is the natural solution for efficiently executing competitive product matching at scale. DataWeave offers solutions for pricing teams to help them reach over 95% product match accuracy. The tools leverage the most modern Natural Language Processing models for ingesting and analyzing product descriptions. Image recognition capabilities apply methods such as object detection, background removal, and image quality enhancement to focus on an individual product’s key features to improve match accuracy.

Deep learning models have been trained on years of data to perform pattern recognition in product attributes and to learn from historical matches. All of these capabilities, and others, automate the attribute matching process, from code to image to feature description, to help pricing teams build the most accurate profile of products across the market for highly accurate pricing intelligence.

Implementation Strategy

We understand that moving away from manual product comparison methods can be challenging. Every organization is different, but some fundamental steps can be followed for success when leveling up your pricing teams’ workflow.
1. First, conduct a baseline assessment. Figure out where you are on the Matching hierarchy. Are you still only doing direct code-based comparisons? Has your team branched out to compare other non-code-based identifiers?
2. Next, establish clear match rate targets for yourself. If your current match rate is aligned with industry norms, strive to significantly improve it, aiming for a high alignment that supports maximizing the match rate. Break this down into achievable milestones across different stages of the implementation process.
3. Work with your vendor on quality control processes. It may be worth running your current process in tandem to be able to calculate the improvements in real time. With a veteran technology provider like DataWeave, you can rely on the most cutting-edge technology combined with human-in-the-loop checks and balances and a team of knowledgeable support personnel. Additionally, for teams wanting direct control, DataWeave’s Approve/Disapprove Module lets your team review and validate match recommendations before they go live, maintaining full oversight of the matching process.
4. The more data about your products it has, the better your match rates. DataWeave’s competitive intelligence tools also come with a built-in continuous improvement framework. Part of this is the human element that continually ensures high-quality matches, but another is the AI’s ‘learning’ capabilities. Every time the AI is exposed to a new scenario, it learns for the next time.
5. The final step, ensure cross-functional alignment is achieved. Every one from the C-Suite down should be able to access the synthesized information useful for their role without complex data to sift through. Customized dashboards and reports can help with this process.
Future-Proofing Match Rates

The world of retail is constantly evolving. If you don’t keep up, you’re going to be left behind. There are emerging retail channels, like the TikTok shop, and new product identification methods to leverage, like image comparisons. As more products enter the market along with new retailers, figuring out how to scale needs to be taken into consideration. It’s impossible to keep up with manual processes. Instead, think about maximizing your match rates every week and not letting them degrade over time. A combination of scale, timely action, and highly accurate match rates will help you price your products the most competitively.

Key Takeaways

Match rates are the foundation of pricing intelligence. You can evaluate how advanced your match rate strategy is based on the matching hierarchy. If you’re still early in your journey, you’re likely still relying on code-to-code matches. However, using a mix of AI and traditional methods, you can achieve a 95% accuracy rate on product matching, leading to overall higher competitive match rates. As a result, with continuous improvement, you will stay ahead of the competition even as the goalposts change and new variables are introduced to the competitive landscape.

Starting this process to add AI to your pricing strategy can be overwhelming. At DataWeave, we work with you to make the change easy. Talk to us today to know more.
February 5, 2025
How AI Can Drive Superior Data Quality and Coverage in Competitive Insights for Retailers and Brands
Managing the endlessly growing competitive data from across your eCommerce landscape can feel like pushing a boulder uphill. The sheer volume can be overwhelming, and ensuring that data meets standards of high accuracy and quality, and the insights are actionable is a constant challenge.

This article explores the challenges eCommerce companies face in having sustained access to high-quality competitive data and how AI-driven solutions like DataWeave empower brands and retailers with reliable, comprehensive, and timely market intelligence.

The Data Quality Challenge for Retailers and Brands

Brands and retailers make innumerable daily business decisions relying on accurate competitive and market data. Pricing changes, catalog expansion, development of new products, and where to go to market are just a few. However, these decisions are only as good as the insights derived from the data. If the data is made up of inaccurate or low-quality inputs, the outputs will also be low-quality.

Managing eCommerce data at scale gets more complex every year. There are more market entrants, retailers, and copy-cats trying to sell similar or knock-off products. There are millions of SKUs from thousands of retailers in multiple markets. Not only that, the data is constantly changing. Amazon may add a new subcategory definition in an existing space, or Staples might decide to branch out into a new industry like “snack foods for the office”, an established brand might introduce new sizing options in their apparel, or shrinkflation might decrease the size of a product.

Given this, it is imperative that conventional data collection and validation methods need to be revised. Teams that rely on spreadsheets and manual auditing processes can’t keep up with the scale and speed of change. An algorithm that once could match products easily needs to be updated when trends, categories, or terminology change.

With SKU proliferation, visually matching product images against the competition becomes impossible. Knowing where to look for comprehensive data becomes impossible with so many new sellers in the market. Luckily, technology has advanced to a place where manual intervention isn’t the main course of action.

Advanced AI capabilities, like DataWeave’s, tackle these challenges to help gather, categorize, and extract insights that drive impactful business decisions. It performs the millions of actions that your team can’t accomplish with greater accuracy and in near real-time.

Improving the Accuracy of Product Matching

DataWeave’s product matching capabilities rely on an ensemble of text and image-based models with built-in loss functions to determine confidence levels in all insights. These loss functions measure precision and recall. They help in determining how accurate – both in terms of correctness and completeness – the results are so the system can learn and improve over time. The solution’s built-in scoring function provides a confidence metric that brands and retailers can rely on.

The product matching engine is configurable based on the type of products that we are matching. It uses a “pipelined mode” that first focuses on recall or coverage by maximizing the search space for viable candidates, followed by mechanisms to improve the precision.

How ‘Embeddings’ Enhance Scoring

Embeddings are like digital fingerprints. They are dense vector representations that capture the essence of a product in a way that makes it easy to identify similar products. With embeddings, we can codify a more nuanced understanding of the varied relationships between different products. Techniques used to create good embeddings are generic and flexible and work well across product categories. This makes it easier to find similarities across products even with complex terminology, attributes, and semantics.

These along with advanced scoring mechanisms used across DataWeave’s eCommerce offerings provide the foundation for:
- Semantic Analysis: Embeddings identify subtle patterns and meanings in text and image data to better align with business contexts.
- Multimodal Integration: A comprehensive representation of each SKU is created by incorporating embeddings from both text (product descriptions) and images or videos (product visuals)
- Anomaly Detection: AI models leverage embeddings to identify outliers and inconsistencies to improve the overall score accuracy.
Vector Databases for Enhanced Accuracy

Vector databases play a central role in DataWeave’s AI ecosystem. These databases help with better storage, retrieval, and scoring of embeddings and serve to power real-time applications such as Verification. This process helps pinpoint the closest matches for products, attributes, or categories with the help of similarity algorithms. It can even operate when there is incomplete or noisy data. After identification, the system prioritizes data that exhibits high semantic alignment so that all recommendations are high-quality and relevant.

Evolution of Embeddings and Scoring: A Multimodal Perspective

Product listings undergo daily visual and text changes. DataWeave takes a multimodal approach in its AI to ensure that any content shown on a listing is accounted for, including visuals, videos, contextual signals, and text. DataWeave is continually evolving its embedding and scoring models to align with industry advancements and always works within an up-to-date context.

DataWeave’s AI framework can:
- Handle Diverse Data Types: The framework captures a holistic view of the digital shelf by integrating insights from multiple sources.
- Improve Matching Precision: Sophisticated scoring methods refine the accuracy of matches so that brands and retailers can trust the competitive intelligence.
- Scale Across Markets: Additional, expansive datasets are easy for DataWeave’s capabilities, meaning brands and retailers can scale across markets without pausing.
Quantified Improvements: Model Accuracy and Stats
- Since we deployed LLMs and CLIP Embeddings, Product Matching accuracy improved by > 15% from the previous baseline numbers in categories such as Home Improvement, Fashion, and CPG.
- High precision in certain categories such as Electronics and Fashion. Upwards of 85%.
- Close to 90% of matches are auto-processed (auto-verified or auto-rejected).
- Attribute tagging accuracy > 75% and significant improvement for the top 5 categories.
Business Use Case: Multimodal Matching for Price Leadership

For example, if you’re a retailer selling consumer electronics, you probably want to maintain your price leadership across your key markets during peak times like Black Friday Cyber Monday. Doing so is a challenge, as all your competitors are changing prices several times a day to steal your sales. To get ahead of them, this retailer could use DataWeave’s multimodal embedding-based scoring framework to:
- Detect Discrepancies: Isolate SKUs with price mismatches with your competition and take action before revenue is lost.
- Optimize Coverage: Establish a process to capture complete data across the competition so you can avoid knowledge gaps.
- Enable Timely Decisions: Address the ‘low-hanging fruit’ by prioritizing products that need pricing adjustments based on confidence scores on high-impact products. Leverage confidence metrics to prioritize pricing adjustments for high-impact products.
This approach helps retailers stay competitive even as eCommerce evolves around us. By acting fast on complete and reliable data, they can earn and sustain their competitive advantage.

DataWeave’s AI-Driven Data Quality Framework

Let’s look at how our AI can gather the most comprehensive data and output the highest-quality insights. Our framework evaluates three critical dimensions:
- Accuracy: “Is my data correct?” – Ensuring reliable product matches and attribute tracking
- Coverage: “Do I have the complete picture?” – Maintaining comprehensive market visibility
- Freshness: “Is my data recent?” – Guaranteeing timely and current market insights
Scoring Data Quality

To maintain the highest levels of data quality, we rely on a robust scoring mechanism across our solutions. Every dataset that is evaluated is done so based on several key parameters. These can include things like accuracy, consistency, timeliness, and completeness of data. Scores are dynamically updated as new data flows in so that insights can be acted upon.
- Accuracy: Compare gathered data with multiple trusted sources to reduce discrepancies.
- Consistency: Detect and rectify variations or contradictions across the data with regular audits.
- Timeliness: Scoring emphasizes data recency, especially for fast-changing markets like eCommerce.
- Completeness: Ensure all essential data points are included and gaps in coverage are highlighted by analysis.
Apart from this, we also leverage an evolved quality check framework:

Statistical Process Control

DataWeave implements a sophisticated system of statistical process control that includes:
- Anomaly Detection: Using advanced statistical techniques to identify and flag outlier data, particularly for price and stock variations
- Intelligent Alerting: Automated system for notifying stakeholders of significant deviations
- Continuous Monitoring: Real-time tracking of data patterns and trends
- Error Correction: Systematic approach to addressing and rectifying data discrepancies
Transparent Quality Assurance

The platform provides complete visibility into data quality through:
- Comprehensive Data Transparency & Statistics Dashboard: Offering detailed insights into match performance and data freshness
- Match Distribution Analysis: Tracking both exact and similar matches across retailers and locations as required
- Product Tracking Metrics: Visibility into the number of products being monitored
- Autonomous Audit Mechanisms: Giving customers access to cached product pages for transparent, on-demand verification
Human-in-the-Loop Validation (Véracité)

DataWeave’s Véracité system combines AI capabilities with human expertise to ensure unmatched accuracy:
- Expert Validation: Product category specialists who understand industry-specific similarity criteria
- Continuous Learning: AI models that evolve through ongoing expert feedback
- Adaptive Matching: Recognition that similarity criteria can vary by category and change over time
- Detailed Documentation: Comprehensive reasoning for product match decisions
Together, these elements create a robust framework that delivers accurate, complete, and relevant product data for competitive intelligence. The system’s combination of automated monitoring, statistical validation, and human expertise ensures businesses can make decisions based on reliable, high-quality data.

In Conclusion

DataWeave’s AI-driven approach to data quality and coverage empowers retailers and brands to navigate the complexities of eCommerce with confidence. By leveraging advanced techniques such as multimodal embeddings, vector databases, and advanced scoring functions, businesses can ensure accurate, comprehensive, and timely competitive intelligence. These capabilities enable them to optimize pricing, improve product visibility, and stay ahead in an ever-evolving market. As AI continues to refine product matching and data validation processes, brands can rely on DataWeave’s technology to eliminate inefficiencies and drive smarter, more profitable decisions.

The evolution of AI in competitive intelligence is not just about automation—it’s about precision, scalability, and adaptability. DataWeave’s commitment to high data quality standards, supported by statistical process controls, transparent validation mechanisms, and human-in-the-loop expertise, ensures that insights remain actionable and trustworthy. In a digital landscape where data accuracy directly impacts profitability, investing in AI-powered solutions like DataWeave’s is not just an advantage—it’s a necessity for sustained eCommerce success.

To learn more, reach out to us today or email us at contact@dataweave.com.
January 22, 2025
Redefining Product Attribute Tagging With AI-Powered Retail Domain Language Models
In online retail, success hinges on more than just offering quality products at competitive prices. As eCommerce catalogs expand and consumer expectations soar, businesses face an increasingly complex challenge: How do you effectively organize, categorize, and present your vast product assortments in a way that enhances discoverability and drives sales?

Having complete and correct product catalog data is key. Effective product attribute tagging—a crucial yet frequently undervalued capability—helps in achieving this accuracy and completeness in product catalog data. While traditional methods of tagging product attributes have long struggled with issues of scalability, consistency, accuracy, and speed, a new thinking and fundamentally new ways of addressing these challenges are getting established. These follow from the revolution brought in Large Language Models but they fashion themselves as Small Language Models (SLM) or more precisely as Domain Specific Language Models. These can be potentially considered foundational models as they solve a wide variety of downstream tasks albeit within specific domains. They are a lot more efficient and do a much better job in those tasks compared to an LLM. .

Retail Domain Language Models (RLMs) have the potential to transform the eCommerce customer journey. As always, it’s never a binary choice. In fact, LLMs can be a great starting point since they provide an enhanced semantic understanding of the world at large: they can be used to mine structured information (e.g., product attributes and values) out of unstructured data (e.g., product descriptions), create baseline domain knowledge (e.g, manufacturer-brand mappings), augment information (e.g., image to prompt), and create first cut training datasets.

Powered by cutting-edge Generative AI and RLMs, next-generation attribute tagging solutions are transforming how online retailers manage their product catalog data, optimize their assortment, and deliver superior shopping experiences. As a new paradigm in search emerges – based more on intent and outcome, powered by natural language queries and GenAI based Search Agents – the capability to create complete catalog information and rich semantics becomes increasingly critical.

In this post, we’ll explore the crucial role of attribute tagging in eCommerce, delve into the limitations of conventional tagging methods, and unveil how DataWeave’s innovative AI-driven approach is helping businesses stay ahead in the competitive digital marketplace.

Why Product Attribute Tagging is Important in eCommerce

As the eCommerce landscape continues to evolve, the importance of attribute tagging will only grow, making it a pertinent focus for forward-thinking online retailers. By investing in robust attribute tagging systems, businesses can gain a competitive edge through improved product comparisons, more accurate matching, understanding intent, and enhanced customer search experiences.

Taxonomy Comparison and Assortment Gap Analysis

Products are categorized and organized differently on different retail websites. Comparing taxonomies helps in understanding focus categories and potential gaps in assortment breadth in relation to one’s competitors: missing product categories, sizes, variants or brands. It also gives insights into the navigation patterns and information architecture of one’s competitors. This can help in making search and navigation experience more efficient by fine tuning product descriptions to include more attributes and/or adding additional relevant filters to category listing pages.

For instance, check out the different Backpack categories on Amazon and Staples in the images below.

Or look at the nomenclature of categories for “Pens” on Amazon (left side of the image) and Staples (right side of the image) in the image below.

Assortment Depth Analysis

Another big challenge in eCommerce is the lack of standardization in retailer taxonomy. This inconsistency makes it difficult to compare the depth of product assortments across different platforms effectively. For instance, to categorize smartphones,
- Retailer A might organize it under “Electronics > Mobile Phones > Smartphones”
- Retailer B could use “Technology > Phones & Accessories > Cell Phones”
- Retailer C might opt for “Consumer Electronics > Smartphones & Tablets”
Inconsistent nomenclature and grouping create a significant hurdle for businesses trying to gain a competitive edge through assortment analysis. The challenge is exacerbated if you want to do an in-depth assortment depth analysis for one or more product attributes. For instance, look at the image below to get an idea of the several attribute variations for “Desks” on Amazon and Staples.

Custom categorization through attribute tagging is essential for conducting granular assortment comparisons, allowing companies to accurately assess their product offerings against those of competitors.

Enhancing Product Matching Capabilities

Accurate product matching across different websites is fundamental for competitive pricing intelligence, especially when matching similar and substitute products. Attribute tagging and extraction play a crucial role in this process by narrowing down potential matches more effectively, enabling matching for both exact and similar products, and tagging attributes such as brand, model, color, size, and technical specifications.

For instance, when choosing to match similar products in the Sofa category for 2-3 seater sofas from Wayfair and Overstock, tagging attributes like brand, color, size, and more is a must for accurate comparisons.

Taking a granular approach not only improves pricing strategies but also helps identify gaps in product offerings and opportunities for expansion.

Fix Content Gaps and improve Product Detail Page (PDP) Content

Attribute tagging plays a vital role in enhancing PDP content by ensuring adherence to brand integrity standards and content compliance guidelines across retail platforms. Tagging attributes allows for benchmarking against competitor content, identifying catalog gaps, and enriching listings with precise details.

This strategic tagging process can highlight missing or incomplete information, enabling targeted optimizations or even complete rewrites of PDP content to improve discoverability and drive conversions. With accurate attribute tagging, businesses can ensure each product page is fully optimized to capture consumer attention and meet retail standards.

Elevating the Search Experience

In today’s online retail marketplace, a superior search experience can be the difference between a sale and a lost customer. Through in-depth attribute tagging, vendors can enable more accurate filtering to improve search result relevance and facilitate easier product discovery for consumers.

By integrating rich product attributes extracted by AI into an in-house search platform, retailers can empower customers with refined and user-friendly search functionality. Enhanced search capabilities not only boost customer satisfaction but also increase the likelihood of conversions by helping shoppers find exactly what they’re looking for more quickly and with minimal effort.

Pitfalls of Conventional Product Tagging Methods

Traditional methods of attribute tagging, such as manual and rule-based systems, have been significantly enhanced by the advent of machine learning. While these approaches may have sufficed in the past, they are increasingly proving inadequate in the face of today’s dynamic and expansive online marketplaces.

Scalability

As eCommerce catalogs expand to include thousands or even millions of products, the limitations of machine learning and rule-based tagging become glaringly apparent. As new product categories emerge, these systems struggle to keep pace, often requiring extensive revisions to existing tagging structures.

Inconsistencies and Errors

Not only is reliance on an entirely human-driven tagging process expensive, but it also introduces a significant margin for error. While machine learning can automate the tagging process, it’s not without its limitations. Errors can occur, particularly when dealing with large and diverse product catalogs.

As inventories grow more complex to handle diverse product ranges, the likelihood of conflicting or erroneous rules increases. These inconsistencies can result in poor search functionality, inaccurate product matching, and ultimately, a frustrating experience for customers, drawing away the benefits of tagging in the first place.

Speed

When product information changes or new attributes need to be added, manually updating tags across a large catalog is a time-consuming process. Slow tagging processes make it difficult for businesses to quickly adapt to emerging market trends causing significant delays in listing new products, potentially missing crucial market opportunities.

How DataWeave’s Advanced AI Capabilities Revolutionize Product Tagging

Advanced solutions leveraging RLMs and Generative AI offer promising alternatives capable of overcoming these challenges and unlocking new levels of efficiency and accuracy in product tagging.

DataWeave automates product tagging to address many of the pitfalls of other conventional methods. We offer a powerful suite of capabilities that empower businesses to take their product tagging to new heights of accuracy and scalability with our unparalleled expertise.

Our sophisticated AI system brings an advanced level of intelligence to the tagging process.

RLMs for Enhanced Semantic Understanding

Semantic Understanding of Product Descriptions

RLMs analyze the meaning and context of product descriptions rather than relying on keyword matching.
Example: “Smartphone with a 6.5-inch display” and “Phone with a 6.5-inch screen” are semantically similar, though phrased differently.

Attribute Extraction

RLMs can identify important product attributes (e.g., brand, size, color, model) even from noisy or unstructured data.
Example: Extracting “Apple” as a brand, “128GB” as storage, and “Pink” as the color from a mixed description.

Identifying Implicit Relationships

RLMs find implicit relationships between products that traditional rule-based systems miss.
Example: Recognizing that “iPhone 12 Pro” and “Apple iPhone 12” are part of the same product family.

Synonym Recognition in Product Descriptions

Synonym Matching with Context

RLMs identify when different words or phrases describe the same product.
Examples: “Sneakers” = “Running Shoes”, “Memory” = “RAM” (in electronics)
Even subtle differences in wording, like “rose gold” vs “pink” are interpreted correctly.

Overcoming Brand-Specific Terminology

Some brands use their own terminologies (e.g., “Retina Display” for Apple).
RLMs can map proprietary terms to more generic ones (e.g., Retina Display = High-Resolution Display).

Dealing with Ambiguities

RLMs analyze surrounding text to resolve ambiguities in product descriptions.
Example: Resolving “charger” to mean a “phone charger” when matched with mobile phones.

Contextual Understanding for Improved Accuracy and Precision

By leveraging advanced natural language processing (NLP), DataWeave’s AI can process and understand the context of lengthy product descriptions and customer reviews, minimizing errors that often arise at human touch points. The solution processes and interprets information to extract key information to dramatically improve the overall accuracy of product tags.

It excels at grasping the subtle differences between similar products, sizes, colors and identifying and tagging minute differences between items, ensuring that each product is uniquely and accurately represented in a retailer’s catalog.

This has a major impact on product and similarity-based matching that can even help optimize similar and substitute product matching to enhance consumer search. At the same time, our AI can understand that the same term might have different meanings in various product categories, adapting its tagging approach based on the specific context of each item.

This deep comprehension ensures that even nuanced product attributes are accurately captured and tagged for easy discoverability by consumers.

Case Study: Niche Jewelry Attributes

DataWeave’s advanced AI can assist in labeling the subtle attributes of jewelry by analyzing product images and generating prompts to describe the image. In this example, our AI identifies the unique shapes and materials of each item in the prompts.

The RLM can then extract key attributes from the prompt to generate tags. This assists in accurate product matching for searches as well as enhanced product recommendations based on similarities.

This multi-model approach provides the flexibility to adapt as product catalogs expand while remaining consistent with tagging to yield more robust results for consumers.

Unparalleled Scalability

DataWeave can rapidly scale tagging for new categories. The solution is built to handle the demands of even the largest eCommerce catalogs enabling:
- Effortless management of extensive product catalogs: We can process and tag millions of products without compromising on speed or accuracy, allowing businesses to scale without limitations.
- Automated bulk tagging: New product lines or entire categories can be tagged automatically, significantly reducing the time and resources required for catalog expansion.
Normalizing Size and Color in Fashion

Style, color, and size are the core attributes in the fashion and apparel categories. Style attributes, which include design, appearance, and overall aesthetics, can be highly specific to individual product categories.

Our product matching engine can easily handle color and sizing complexity via our AI-driven approach combined with human verification. By leveraging advanced technology to identify and normalize identical and similar products from competitors, you can optimize your pricing strategy and product assortment to remain competitive. Using Generative AI in normalizing color and size in fashion is key to powering competitive pricing intelligence at DataWeave.

Continuous Adaptation and Learning

Our solution evolves with your business, improving continuously through feedback and customization for retailers’ specific product categories. The system can be fine-tuned to understand and apply specialized tagging for niche or industry-specific product categories. This ensures that tags remain relevant and accurate across diverse catalogs and as trends emerge.

The AI in our platform also continuously learns from user interactions and feedback, refining its tagging algorithms to improve accuracy over time.

Stay Ahead of the Competition With Accurate Attribute Tagging

In the current landscape, the ability to accurately and consistently tag product attributes is no longer a luxury—it’s essential for staying competitive. With advancements in Generative AI, companies like DataWeave are revolutionizing the way product tagging is handled, ensuring that every item in a retailer’s catalog is presented with precision and depth. As shoppers demand a more intuitive, seamless experience, next-generation tagging solutions are empowering businesses to meet these expectations head-on.

DataWeave’s innovative approach to attribute tagging is more than just a technical improvement; it’s a strategic advantage in an increasingly competitive market. By leveraging AI to scale and automate tagging processes, online retailers can keep pace with expansive product assortments, manage content more effectively, and adapt swiftly to changes in consumer behavior. In doing so, they can maintain a competitive edge.

To learn more, talk to us today!
November 14, 2024
Normalizing Size and Color in Fashion Using AI to Power Competitive Price Intelligence
Fashion is as dynamic a market as any—and more competitive than most others. Consumer trends and customer needs are always evolving, making it challenging for fashion and apparel brands to keep up.

Despite the inherent difficulties fashion and apparel sellers face, this industry is one of the largest grossing markets in the world, estimated at $1.79 trillion in 2024. Global revenue for apparel is expected to grow at an annual rate of about 3.3% over the next four years. That means companies in this space stand to make significant revenue if they can competitively price their products, keep up with the competition, and win customer loyalty with consistent product availability.

There are three main categories in fashion and apparel. These include:
- Apparel and clothing (i.e., shirts, pants, dresses, and other apparel)
- Footwear (i.e., sneakers, sandals, heels, and other products)
- Accessories (i.e., bags, belts, watches, and so on)
If you look at all of these product types across all sorts of retailers, there is a massive amount of overlapping data based on product attributes like style and size that are difficult to normalize.

Fashion Attributes

Style, color, and size are the main attribute categories in fashion and apparel. Style attributes include things like design, look, and overall aesthetics of the product. They’re very dependent on the actual product category of fashion as well. A shirt might have a slim fit attribute associated with it, whereas a belt might have a length. All these different attributes are usually labeled within a product listing and affect the consumer’s decision-making process:
- Color (red, blue, sea green, etc.)
- Pattern (solid, striped, checked, floral, etc.)
- Material (cotton, polyester, leather, denim, silk, etc.)
- Fit (regular, slim, relaxed, oversized, tailored, etc.)
- Type (casual, formal, sporty, vintage, streetwear)
Color Complexity in Fashion

Color is perhaps the most visually distinctive attribute in fashion, yet it presents unique challenges for retailers. This is because color naming can vary across retailers and marketplaces. There are several major differences in color convention:
- A single color can be labeled differently across brands (e.g., “navy,” “midnight blue,” “deep blue”)
- Seasonal color names (e.g., “summer sage” vs. “forest green”)
- Marketing-driven names (e.g., “sunset coral” vs. “pale orange”)
Size: The Other Critical Dimension

Size in fashion refers to the dimensions or measurements that determine how fashion products fit. Depending on whether the product is a clothing item, shoes, or a hat, there will be different sizing options. Types of sizes include:
- Standard sizes (XS, S, M, L, XL, XXL, XXL)
- Custom sizes (based on brand, retailer, country, etc.)
A single type of product may have different sizing labels. For instance, one pants listing may use traditional S, M, L, XL sizing, while another pants listing may use 24, 25, or 26, to refer to the waist measurement.

Size is a dynamic attribute that changes based on current trends. For example, there has recently been a significant shift towards inclusive sizing. Size inclusivity refers to the practice of selling apparel in a wide range of sizes to accommodate people of all body types. Consumers are more aware of this trend and are demanding a broader range of sizing offerings from the brands they shop from.

In the US market, in particular, some 67% of American women wear a size 14 or above and may be interested in purchasing plus-size clothing. There is a growing demand in the plus-size market for more options and a wider selection. Many brands are considering expanding their sizes to accommodate more shoppers and tap into this growing revenue channel.

Pricing Based on Size and Color

Many fashion products are priced differently based on size and color. Let’s take a look at an example of what this can look like.

A popular beauty brand (see image) is known for its viral lip tint. While most of the color variants are priced at $9.90 on Amazon, a specific colorway option, featuring less pigmented options, is priced at $9.57. This price differential is driven by both material costs and market demand.

Different colorways (any of a range of combinations of colors in which a style or design is available) of the same product often command different prices also. This is based on:
- Dye costs (some colors require more expensive processes)
- Seasonal demand (traditional colors vs. trend colors)
- Exclusivity (limited edition colors)
An example of price variations by size is a women’s shirt that is being sold on Amazon as shown below. For this product, there are no style attributes to choose from. The only parameter the shopper has to select is the size they’d like to purchase. They can choose from S to XL. On the top, we can see that the product in size S is ₹389. Below, the size XL version of this same shirt is ₹399. This price increase is correlated to the change in size.

So why are these same products priced differently? In an analysis of One Six, a plus-size clothing brand, several reasons for this difference in plus-size clothing were determined.
- Extra material is needed, hence an increase in production costs
- Extra stitching costs, hence an increase in production costs
- Production of plus-size clothing often means acquiring specialized machinery
- Smaller scale production runs for plus-size clothing means these initiatives often don’t benefit from cost savings
Some sizes are sold more than others, meaning that in-demand sizes for certain apparel can affect pricing as well. Brands want to be able to charge as much as possible for their listing without risking losing a sale to a competitor.

The Competitive Pricing Challenge: Normalizing Product Attributes Across Competitors in Apparel and Fashion

There are hundreds of possible attribute permutations for every single apparel product. Some retailers may only sell core sizes and basic colors; some may sell a mix of sizes for multiple style types. Most retailers also sell multiple color variants for all styles they have on catalog. Other retailers may only sell a single, in-demand size of the product. Also, when other retailers are selling the product, it’s unlikely that their naming conventions, color options, style options, and sizing match yours one-for-one.

In one analysis, it was found that there were 800+ unique values for heel sizes and 1000+ unique values for shirts and tops at a single retailer! If you’re looking to compare prices, the effort involved in setting up and managing lookup tables to identify discrepancies when one retailer uses European sizes and another uses USA sizes, for example, is simply too onerous to contemplate doing. Colors only add to the complexity – as similar colors may have new names in different regions and locations as well!

Even if you managed to find all the discrepancies between product attributes, you would still need to update them any time a competitor changed a convention.

Still, monitoring your competitors and strategically pricing your listings is essential to maintain and grow market share. So what do you do? You can’t simply eyeball your competitor’s website to check their pricing and naming conventions. Instead, you need advanced algorithms to scan the entire marketplace, identify individual products being sold, and normalize their data and attributes for analysis.

Getting Color and Size Level Pricing Intelligence

With DataWeave, size and color are just two of several dimensions of a product instead of an impossible big data problem for teams. Our product matching engine can easily handle color and sizing complexity via our AI-driven approach combined with human verification.

This works by using AI built on more than 10 years of product catalog data across thousands of retail websites. It matches common identifiers, like UPC, SKU code, and other attributes for harmonization before employing a large language model (LLM) prompts to normalize color variations and sizing to a single standard.

For example, if a competitor has the smallest size listed as Sm but has your smallest listing identified as S, DataWeave can match those two attributes using AI. Similar classification can be performed on color as well.

Complex LLM prompts are pre-established so that this process is fast and efficient, taking minutes rather than weeks of manual effort.

Harmonizing products along with their color and sizing data across different retailers for further analysis has several benefits. Most importantly, product matching helps teams conduct better competitive analysis, allowing them to stay informed about market trends, competitors’ offerings, and how those competitors are pricing various permutations of the same product. It helps ensure that you’re offering the most competitive assortment of sizing in several colors to win more market share as well. Overall, it’s easier for teams to gain insights and exploit their findings when all the data is clean and available at their fingertips.

Product Matching Size and Color in Apparel and Fashion

Color and size are crucial attributes for retailers and brands in the apparel and fashion industry. It adds a level of complexity that can’t be overstated. While it’s a necessity to win consumers (more colors and sizes will mean a wider potential reach), the more permutations you add to your listing, the more complicated it will be to track it against your competition. However, This challenge is worth undertaking as long as you have the right solutions at your disposal.

With a strategy backed by advanced technology to discover identical and similar products across the competitive landscape and normalize their color and sizing attributes, you can ensure that you are competitively pricing your products and offering the best assortment possible. Employing DataWeave’s AI technology to find competitor listings, match products across variants, and track pricing regularly is the way to go.

Interested in learning more about DataWeave? Click here to get in touch!
November 6, 2024
DataWeave’s AI Evolution: Delivering Greater Value Faster in the Age of AI and LLMs
In retail, competition is fierce, and in its ever-evolving landscape, consumer expectations are higher than ever.

For years, our AI-driven solutions have been the foundation that empowers businesses to sharpen their competitive pricing and optimize digital shelf performance. But in today’s world, evolution is constant—so is innovation. We now find ourselves at the frontier of a new era in AI. With the dawn of Generative AI and the rise of Large Language Models (LLMs), the possibilities for eCommerce companies are expanding at an unprecedented pace.

These technologies aren’t just a step forward; they’re a leap—propelling our capabilities to new heights. The insights are deeper, the recommendations more precise, and the competitive and market intelligence we provide is sharper than ever. This synergy between our legacy of AI expertise and the advancements of today positions DataWeave to deliver even greater value, thus helping businesses thrive in a fast-paced, data-driven world.

This article marks the beginning of a series where we will take you through these transformative AI capabilities, each designed to give retailers and brands a competitive edge.

In this first piece, we’ll offer a snapshot of how DataWeave aggregates and analyzes billions of publicly available data points to help businesses stay agile, informed, and ahead of the curve. These fall into four broad categories:
- Product Matching
- Attribute Tagging
- Content Analysis
- Promo Banner Analysis
- Other Specialized Use Cases
Product Matching

Dynamic pricing is an indispensable tool for eCommerce stores to remain competitive. A blessing—and a curse—of online shopping is that users can compare prices of similar products in a few clicks, with most shoppers gravitating toward the lowest price. Consequently, retailers can lose sales over minor discrepancies of $1–2 or even less.

All major eCommerce platforms compare product prices—especially their top selling products—across competing players and adjust prices to match or undercut competitors. A typical product undergoes 20.4 price changes annually, or roughly once every 18 days. Amazon takes it to the extreme, changing prices approximately every 10 minutes. It helps them maintain a healthy price perception among their consumers.

However, accurate product matching at scale is a prerequisite for the above, and that poses significant challenges. There is no standardized approach to product cataloging, so even identical products bear different product titles, descriptions, and attributes. Information is often incomplete, noisy, or ambiguous. Image data contains even more variability—the same product can be styled using different backgrounds, lighting, orientations, and quality; images can have multiple overlapping objects of interest or extraneous objects, and at times the images and the text on a single page might belong to completely different products!

DataWeave leverages advanced technologies, including computer vision, natural language processing (NLP), and deep learning, to achieve highly accurate product matching. Our pricing intelligence solution accurately matches products across hundreds of websites and automatically tracks competitor pricing data.

Here’s how it works:

Text Preprocessing

It identifies relevant text features essential for accurate comparison.
- Metadata Parsing: Extracts product titles, descriptions, attributes (e.g., color, size), and other structured data elements from Product Description Pages (PDP) that can help in accurately identifying and classifying products.
- Attribute-Value Normalization: Normalize attributes names (e.g. RAM vs Memory) and their values (e.g., 16 giga bytes vs 16 gigs vs 16 GB); brand names (e.g., Benetton vs UCB vs United Colors of Benetton); mapping category hierarchies a standard taxonomy.
- Noise Removal: Removes stop words and other elements with no descriptive value; this focuses keyword extraction on meaningful terms that contribute to product identification.
Image Preprocessing

Image processing algorithms use feature extraction to define visual attributes. For example, when comparing images of a red T-shirt, the algorithm might extract features such as “crew neck,” “red,” or “striped.”

Image hashing techniques create a unique representation (or “hash”) of an image, allowing for efficient comparison and matching of product images. This process transforms an image into a concise string or sequence of numbers that captures its essential features even if the image has been resized, rotated, or edited.

Before we perform these activities there is a need to preprocess images to prepare them for downstream operations. These include object detection to identify objects of interest, background removal, face/skin detection and removal, pose estimation and correction, and so forth.

Embeddings

We have built a hybrid or a multimodal product-matching engine that uses image features, text features, and domain heuristics. For every product we process we create and store multiple text and image embeddings in a vector database. These include a combination of basic feature vectors (e.g. tf-idf based, colour histograms, share vectors) to more advanced deep learning algorithms-based embeddings (e.g., BERT, CLIP) to the latest LLM-based embeddings.

Classification

Classification algorithms enhance product attribute tagging by designating match types. For example, the product might be identified as an “exact match”, “variant”, “similar”, or “substitute.” The algorithm can also identify identical product combinations or “baskets” of items typically purchased together.

What is the Business Impact of Product Matching?
- Pricing Intelligence: Businesses can strategically adjust pricing to remain competitive while maintaining profitability. High-accuracy price comparisons help businesses analyze their competitive price position, identify opportunities to improve pricing, and reclaim market share from competitors.
- Similarity-Based Matching: Products are matched based on a range of similarity features, such as product type, color, price range, specific features, etc., leading to more accurate matches.
- Counterfeit Detection: Businesses can identify counterfeit or unauthorized versions of branded products by comparing them against authentic product listings. This helps safeguard brand identity and enables brands to take legal action against counterfeiters.
Attribute Tagging

Attribute tagging involves assigning standardized tags for product attributes, such as brand, model, size, color, or material. These naming conventions form the basis for accurate product matching. Tagging detailed attributes, such as specifications, features, and dimensions, helps match products that meet similar criteria. For example, tags like “collar” or “pockets” for apparel ensure high-fidelity product matches for hard-to-distinguish items with minor stylistic variations.

Including tags for synonyms, variants, and long-tail keywords (e.g., “denim” and “jeans”) improves the matching process by recognizing different terms used for similar products. Metadata tags categorize similar items according to SKU numbers, manufacturer details, and other identifiers.

Altogether, these capabilities provide high-quality product matches and valuable metadata for retailers to classify their products and compare their product assortment to competitors.

User-Generated Content (UGC) Analysis

Customer reviews and ratings are rich sources of information, enabling brands to gauge consumer sentiment and identify shortcomings regarding product quality or service delivery. However, while informative, reviews constitute unstructured “noisy” data that is actionable only if parsed correctly.

Here’s where DataWeave’s UGC analysis capability steps in.
- Feature Extractor: Automatically pulls specific product attributes mentioned in the review (e.g., “battery life,” “design” and “comfort”)
- Feature Opinion Pair: Pairs each product attribute with a corresponding sentiment from the review (e.g., “battery life” is “excellent,” “design” is “modern,” and “comfort” is “poor”)
- Calculate Sentiment: Calculates an overall sentiment score for each product attribute
The final output combines the information extracted from each of these features, which looks something like this:
- Battery life is excellent
- Design is modern
- Not satisfied with the comfort
The algorithm also recognizes spammy reviews and distinguishes subjective reviews (i.e., those fueled by emotion) from objective ones.

Promo Banner Analysis

Our image processing tool can interpret promotional banners and extract information regarding product highlights, discounts, and special offers. This provides insights into pricing strategies and promotional tactics used by other online stores.

For example, if a competitor offers a 20% discount on a popular product, you can match or exceed this discount to attract more customers.

The banner reader identifies successful promotional trends and patterns from competitors, such as the timing of discounts, frequently promoted product categories or brands, and the duration of sales events. Ecommerce stores can use this information to optimize their promotion strategies, ensuring they launch compelling and timely offers.

Other Specialized Use Cases

While these generalized AI tools are highly useful in various industries, we’ve created other category—and attribute-specific capabilities for specialty goods (e.g., those requiring certifications or approval by federal agencies) and food items. These use cases help our customers adhere to compliance requirements.

Certification Mark Detector

This detector lets retailers match items based on official certification marks. These marks represent compliance with industry standards, safety regulations, and quality benchmarks.

Example:
- USDA Organic: Certification for organic food production and handling
- ISO 9001: Quality Management System Certification
By detecting these certification marks, the system can accurately match products with their certified counterparts. By identifying which competitor products are certified, retailers can identify products that may benefit from certification.

Nutrition Fact Table Reader

Product attributes alone are insufficient for comparing food items. Differences in nutrition content can influence product category (e.g., “health food” versus regular food items), price point, and consumer choice. DataWeave’s nutrition fact table reader scans nutrition information on packaging, capturing details such as calorie count, macronutrient distribution (proteins, fats, carbohydrates), vitamins, and minerals.

The solution ensures items with similar nutritional profiles are correctly identified and grouped based on specific dietary requirements or preferences. This helps with price comparisons and enables eCommerce stores to maintain a reliable database of product information and build trust among health-conscious consumers.

Building Next-Generation Competitive and Market Intelligence

Moving forward, breakthroughs in generative AI and LLMs have fueled substantial innovation, which has enabled us to introduce powerful new capabilities for our customers.

These include:
- Building Enhanced Products, Solutions, and Capabilities: Generative AI and LLMs can significantly elevate the performance of existing solutions by improving the accuracy, relevance, and depth of insights. By leveraging these advanced AI technologies, DataWeave can enhance its product offerings, such as pricing intelligence, product matching, and sentiment analysis. These tools will become more intuitive, allowing for real-time updates and deeper contextual understanding. Additionally, AI can help create entirely new solutions tailored to specific use cases, such as automating competitive analysis or identifying emerging market trends. This positions DataWeave to remain at the forefront of innovation, offering cutting-edge solutions that meet the evolving needs of retailers and brands.
- Reducing Turnaround Time (TAT) to Go-to-Market Faster: Generative AI and LLMs streamline data processing and analysis workflows, enabling faster decision-making. By automating tasks like data aggregation, sentiment analysis, and report generation, AI dramatically reduces the time required to derive actionable insights. This efficiency means that businesses can respond to market changes more swiftly, adjusting pricing or promotional strategies in near real-time. Faster insights translate into reduced turnaround times for product development, testing, and launch cycles, allowing DataWeave to bring new solutions to market quickly and give clients a competitive advantage.
- Improving Data Quality to Achieve Higher Performance Metrics: AI-driven technologies are exceptionally skilled at cleaning, organizing, and structuring large datasets. Generative AI and LLMs can refine the data input process, reducing errors and ensuring more accurate, high-quality data across all touchpoints. Improved data quality enhances the precision of insights drawn from it, leading to higher performance metrics like better product matching, more accurate price comparisons, and more effective consumer sentiment analysis. With higher-quality data, businesses can make smarter, more informed decisions, resulting in improved revenue, market share, and customer satisfaction.
- Augmenting Human Bandwidth with AI to Enhance Productivity: Generative AI and LLMs serve as powerful tools that augment human capabilities by automating routine, time-consuming tasks such as data entry, classification, and preliminary analysis. This allows human teams to focus on more strategic, high-value activities like interpreting insights, building relationships with clients, and developing new business strategies. By offloading these repetitive tasks to AI, human productivity is significantly enhanced. Employees can achieve more in less time, increasing overall efficiency and enabling teams to scale their operations without needing a proportional increase in human resources.
In our ongoing series, we will dive deep into each of these capabilities, exploring how DataWeave leverages cutting-edge AI technologies like Generative AI and LLMs to solve complex challenges for retailers and brands.

In the meantime, talk to us to learn more!
September 18, 2024
Using Siamese Networks to Power Accurate Product Matching in eCommerce
Retailers often compete on price to gain market share in high performance product categories. Brands too must ensure that their in-demand assortment is competitively priced across retailers. Commerce and digital shelf analytics solutions offer competitive pricing insights at both granular and SKU levels. Central to this intelligence gathering is a vital process: product matching.

Product matching or product mapping involves associating identical or similar products across diverse online platforms or marketplaces. The matching process leverages the capabilities of Artificial Intelligence (AI) to automatically create connections between various representations of identical or similar products. AI models create groups or clusters of products that are exactly the same or “similar” (based on some objectively defined similarity criteria) to solve different use cases for retailers and consumer brands.

Accurate product matching offers several key benefits for brands and retailers:
- Competitive Pricing: By identifying identical products across platforms, businesses can compare prices and adjust their strategies to remain competitive.
- Market Intelligence: Product matching enables brands to track their products’ performance across various retailers, providing valuable insights into market trends and consumer preferences.
- Assortment Planning: Retailers can analyze their product range against competitors, identifying gaps or opportunities in their offerings.
Why Product Matching is Incredibly Hard

But product matching stands out as one of the most demanding technical processes for commerce intelligence tools. Here’s why:

Data Complexity

Product information comes in various (multimodal) formats – text, images, and sometimes video. Each format presents its own set of challenges, from inconsistent naming conventions to varying image quality.

Data Variance

The considerable fluctuations in both data quality and quantity across diverse product categories, geographical regions, and websites introduce an additional layer of complexity to the product matching process.

Industry Specific Nuances

Industry specific nuances introduce unique challenges to product matching. Exact matching may make sense in certain verticals, such as matching part numbers in industrial equipment or identifying substitute products in pharmaceuticals. But for other industries, exactly matched products may not offer accurate comparisons.
- In the Fashion and Apparel industry, style-to-style matching, accommodating variants and distinguishing between core sizes and non-core sizes and age groups become essential for accurate results.
- In Home Improvement, the presence of unbranded products, private labels, and the preference for matching sets rather than individual items complicates the process.
- On the other hand, for grocery, product matching becomes intricate due to the distinction between item pricing and unit pricing. Managing the diverse landscape of different pack sizes, quantities, and packaging adds further layers of complexity.
Diverse Downstream Use Cases

The diverse downstream business applications give rise to various flavors of product matching tailored to meet specific needs and objectives.

In essence, while product matching is a critical component in eCommerce, its intricacies demand sophisticated solutions that address the above challenges.

To solve these challenges, at DataWeave, we’ve developed an advanced product matching system using Siamese Networks, a type of machine learning model particularly suited for comparison tasks.

Siamese Networks for Product Matching

Our methodology involves the use of ensemble deep learning architectures. In such cases, multiple AI models are trained and used simultaneously to ensure highly accurate matches. These models tackle NLP (natural language processing) and Computer Vision challenges specific to eCommerce. This technology helps us efficiently narrow down millions of product candidates to just 5-15 highly relevant matches.

The Tech Powering Siamese Networks

The key to our approach is creating what we call “embeddings” – think of these as unique digital fingerprints for each product. These embeddings are designed to capture the essence of a product in a way that makes similar products easy to identify, even when they look slightly different or have different names.

Our system learns to create these embeddings by looking at millions of product pairs. It learns to make the embeddings for similar products very close to each other while keeping the embeddings for different products far apart. This process, known as metric learning, allows our system to recognize product similarities without needing to put every product into a rigid category.

This approach is particularly powerful for eCommerce, where we often need to match products across different websites that might use different names or images for the same item. By focusing on the key features that make each product unique, our system can accurately match products even in challenging situations.

How Siamese Networks Work?

Imagine having a pair of identical twins who are experts at spotting similarities and differences. That’s essentially what a Siamese network is – a pair of identical AI systems working together to compare things.

How it works:
- Twin AI systems: Two identical AI systems look at two different products.
- Creating ‘fingerprints’ or ‘embedding’: Each system creates a unique ‘fingerprint’ of the product it’s looking at.
- Comparison: These ‘fingerprints’ are then compared to see how similar the products are.
Architecture

The architecture of a Siamese network typically consists of three main components: the shared network, the similarity metric, and the contrastive loss function.
- Shared Network: This is the ‘brain’ that creates the product ‘fingerprints’ or ‘embeddings.’ It is responsible for extracting meaningful feature representations from the input samples. This network is composed of layers of neural units that work together. Weight sharing between the twin networks ensures that the model learns to extract comparable features for similar inputs, providing a basis for comparison.
- Similarity Metric: After the shared network processes the inputs, a similarity metric is employed. This decides how alike two ‘fingerprints’ or ‘embeddings’ are. The selection of a similarity metric depends on the specific task and characteristics of the input data. Frequently used similarity metrics include the Euclidean distance, cosine similarity, or correlation coefficient, each chosen based on its suitability for the given context and desired outcomes.
- Loss Function: For training the Siamese network, a specialized loss function is used. This helps the system improve its comparison skills over time. It guides and trains the network to generate akin embeddings for similar inputs and disparate embeddings for dissimilar inputs.
  
  This is achieved by imposing penalties on the model when the distance or dissimilarity between similar pairs surpasses a designated threshold, or when the distance between dissimilar pairs falls below another predefined threshold. This training strategy ensures that the network becomes adept at discerning and encoding the desired level of similarity or dissimilarity in its learned embeddings.
How DataWeave Uses Siamese Networks for Product Matching

At DataWeave, we use Siamese Networks to match products across different retailer websites. Here’s how it works:

Pre-processing (Image Preparation)
- We collect product images from various websites.
- We clean these images up to make them easier for our AI to understand.
- We use techniques like cropping, flipping, and adjusting colors to help our AI recognize products even if the images are slightly different.
Training The AI
- We show our AI system millions of product images, teaching it to recognize similarities and differences.
- We use a special learning method called “Triplet Loss” to help our AI understand which products are the same and which are different.
- We’ve tested different AI structures to find the one that works best for product matching, including ResNet, EfficientNet, NFNet, and ViT.
Image Retrieval
- Once trained, our AI creates a unique “fingerprint” for each product image.
- We store these fingerprints in a smart database.
- When we need to find a match for a product, we:
  - Create a fingerprint for the new product.
  - Quickly search our database for the most similar fingerprints.
  - Return the top matching products.
Matches are then assigned a high or a low similarity score and segregated into “Exact Matches” or “Similar Matches.” For example, check out the image of this white shoe on the left. It has a low similarity score with the pink shoe (below) and so these SKUs are categorized as a “Similar Match.” Meanwhile, the shoe on the right is categorized as an “Exact Match.”

Similarly, in the following image of the dress for a young girl, the matched SKU has a high similarity score and so this pair is categorized as an “Exact Match.”

Siamese Networks play a pivotal role in DataWeave’s Product Matching Engine. Amid the millions of images and product descriptions online, our Siamese Networks act as an equalizing force, efficiently narrowing down millions of candidates to a curated selection of 10-15 potential matches.

In addition, these networks also find application in several other contexts at DataWeave. They are used to train our system to understand text-only data from product titles and joint multimodal content from product descriptions.

Leverage Our AI-Driven Product Matching To Get Insightful Data

In summary, accurate and efficient product matching is no longer a luxury – it’s a necessity. DataWeave’s advanced product matching solution provides brands and retailers with the tools they need to navigate this complex landscape, turning the challenge of product matching into a competitive advantage.

By leveraging cutting-edge technology and simplifying it for practical use, we empower businesses to make informed decisions, optimize their operations, and stay ahead in the ever-evolving eCommerce market. To learn more, reach out to us today!
June 26, 2024
How AI-Powered Visual Highlighting Helps Brands Achieve Product Consistency Across eCommerce
As eCommerce increasingly becomes a prolific channel of sales for consumer brands, they find that maintaining a consistent and trustworthy brand image is a constant struggle. In an ecosystem filled with dozens of marketplaces and hundreds of third-party merchants, ensuring that customers see what aligns with a brand’s intended image is quite tricky. With many fakes and counterfeit products doing the rounds, brands may further struggle to get the right representation.

One way brands can track and identify inconsistencies in their brand representation across marketplaces is to use Digital Shelf Analytics solutions like DataWeave’s – specifically the Content Audit module.

This solution uses advanced AI models to identify image similarities and dissimilarities compared with the original brand image. Brands could then use their PIM platform or work with the retailer to replace inaccurate images.

But here’s the catch – AI can’t always accurately predict all the differences. Relying solely on scores given by these models poses a challenge in tracking the subtle differences between images. Often, image pairs with seemingly high match scores fail to catch important distinctions. Fake or counterfeit products and variations that slip past the AI’s scrutiny can lead to significant inaccuracies. Ultimately, it puts the reliability of the insights that brands depend on for crucial decisions at risk, impacting both top and bottom lines.

Dealing with this challenge means finding a balance between the number-based assessments of AI models and the human touch needed for accurate decision-making. However, giving auditors the ability to pinpoint variations precisely goes beyond simply sharing numerical values of the match scores with them. Visualizing model-generated scores is important as it provides human auditors with a tangible and intuitive understanding of the differences between two images. While numerical scores are comparable in the relative sense, they lack specificity. Visual interpretation empowers auditors to identify precisely where variations occur, aiding in efficient decision-making.

How AI-Powered Image Scoring Works

At DataWeave, our approach involves employing sophisticated computer vision models to conduct extensive image comparisons. Convolutional Neural Network (CNN) models such as Resnet-50 or YOLO, in conjunction with feature extraction models, analyze images quantitatively. This AI-powered image scoring process yields scores that indicate the level of similarity between images.

However, interpreting these scores and understanding the specific areas of difference can be challenging for human auditors. While computer vision models excel at processing vast amounts of data quickly, translating their output into actionable insights can be a stumbling block. A numerical score may not immediately convey the nature or extent of the differences between images

In the assessment of these images, all fall within the 70 to 80 range of scores (out of a maximum of 100). However, discerning the nature of differences—whether they are apparent or subtle—poses a challenge for the AI models and human auditors. For example, there are differences in the placement or type of images in the packaging, as well as packing text that are often in an extremely small font size. It is, of course, possible for human auditors to identify the differences in these images, but it’s a slow, error-prone, and tiring process, especially when auditors often have to check hundreds of image pairs each day.

So how do we ensure that we identify differences in images accurately? The answer lies in the process of visual highlighting.

How Visual Highlighting Works

Visual highlighting is a method that enhances our ability to comprehend differences in images by combining sophisticated algorithms with human understanding. Instead of relying solely on numerical scores, this approach introduces a visual layer, resembling a heatmap, guiding human auditors to specific areas where discrepancies are present.

Consider the scenario depicted in the images above: a computer vision model assigns a score of 70-85 for these images. While this score suggests relatively high similarity, it fails to uncover major differences between the images. Visual highlighting comes into play to overcome this limitation, precisely indicating regions where even subtle differences are seen.

Visual highlighting entails overlaying compared images and emphasizing areas of difference, achieved through techniques like color coding, outlining, or shading specific regions. The significance of the difference in a particular area determines the intensity of the visual highlight.

For instance, if there’s a change in the product’s color or a discrepancy in the packaging, these variations will be visually emphasized. This not only streamlines the auditing process but also enables human evaluators to make well-informed decisions quickly.

Benefits of Visual Highlighting
- Intuitive Understanding: Visual highlighting offers an intuitive method for interpreting and acting upon the outcomes of computer vision models. Instead of delving into numerical scores, auditors can concentrate on the highlighted areas, enhancing the efficiency and accuracy of the decision-making process.
- Accelerated Auditing: By bringing attention to specific regions of concern, visual highlighting speeds up the auditing process. Human evaluators can swiftly identify and address discrepancies without the need for exhaustive image analysis.
- Seamless Communication: Visual highlighting promotes clearer communication between automated systems and human auditors. Serving as a visual guide, it enhances collaboration, ensuring that the subtleties captured by computer vision models are effectively conveyed.
The Way Forward

As technology continues to evolve, the integration of visual highlighting methodologies is likely to become more sophisticated. Artificial intelligence and machine learning algorithms may play an even more prominent role in not only detecting differences but also in refining the visual highlighting process.

The collaboration between human auditors and AI ensures a comprehensive approach to maintaining brand integrity in the ever-expanding digital marketplace. By visually highlighting differences in images, brands can safeguard their visual identity, foster consumer trust, and deliver a consistent and reliable online shopping experience. In the intricate dance between technology and human intuition, visual highlighting emerges as a powerful tool, paving the way for brands to uphold their image with precision and efficiency.

To learn more, reach out to us today!

(This article was co-authored by Apurva Naik)
March 4, 2024
How Enterprise AI is Transforming Business Outcomes for Customers Across Industries

Is AI just a flashy new trend that helps you create some amusing or stunning art and whip up amazing content in seconds (not this one) or is it a force for much greater good? Let’s dive into one of the most disruptive tech moments in our recent history in analyzing the use and transformative powers of AI, for not just end consumers but also businesses as customers.

“Enterprise AI can bring joy and meaning to its stakeholders. No more meaningless work, we are delivering value at the speed of need”, golden words from tech leader Vala Afshar, one of the staunchest endorsers of AI tech today as AI continues to drive top-line revenue growth for the Enterprise (B2B).

How AI has evolved from basic tasks to complex processes

AI or Artificial Intelligence which essentially means the use of a machine – typically computer – to perform tasks usually performed by humans such as processing data, making connections, and coming up with solutions has come a long way since its chess playing days.

Since its application in various industries, AI has been changing the way we live and work. From enabling personalized experiences to automating marketing and CRM to processing vast amounts of complex and scattered data, AI has been helping businesses perform increasingly complex tasks.

With recent advances in Machine Learning and large Language Learning Models (LLM) AI is becoming increasingly sophisticated and is being commissioned in a growing range of applications, from healthcare and education to transportation and finance across the B2B/Enterprise industry.

Enterprise AI vs Consumer AI

Enterprise AI is an ecosystem of tools, processes and companies that leverages AI to provide solutions to businesses, as opposed to the end consumer.

To draw a simple example, a generative prompt-based AI tool like ChatGPT or MidJourney is a consumer AI which helps consumers perform individual tasks. On the other hand, Salesforce provides various AI-led tools that help other businesses in their marketing and CRM processes. A 2020 McKinsey study showed that nearly 70% of all businesses are taking AI seriously and increasibly looking to adopt it into their processes.

The potential of AI is virtually limitless across industries and everyday AI is being leveraged to drive efficiency, innovation and growth across sectors. It is being used to automate routine tasks, provide predictive analytics, analyze customer data, and improve supply chain operations. In this article, we’ll dive into how Enterprise AI is transforming business outcomes for customers across industries through four real world examples.

Enterprise AI at play in these five industries

1. Enterprise AI in Ecommerce

For big ecommerce giants that play in a complex and multi-player ecosystem of today, being on top of their competition is the name of the game. However, it’s nearly impossible to keep a tab on the market manually and that’s where Enterprise AI technology in the domain of Price Intelligence and comparisons come in. Once a domain of manual and web scraping, Enterprise AI now equips both brands and ecommerce marketplaces with large, real-time and dynamic data to help them keep on top of competition and adjust their prices and other variables accordingly.

One of the biggest use cases for Enterprise AI in ecommerce is DataWeave, which provides SaaS-based Pricing Intelligence and Digital Shelf Analytics to its retail and ecommerce clients. Their AI-led matching and image recognition, can scrape millions of ecommerce data like price, inventory, image and others across the ecommerce landscape and enable the companies to make strategic pricing and competitive decisions. According to the company, these real-time pricing insights have helped their 50+ retail customers save millions in revenue by offering the right product at the most strategic price points.

2. Enterprise AI in logistics

AI has a profound impact on logistics, transport and global shipping commerce. By predicting demand, identifying the most efficient routes, and improving warehouse management, AI logistics enterprises are helping their enterprise customers improve delivery rates, timelines and helping reduce carbon emissions.

3. Enterprise AI in healthcare

A case study in how AI has helped logistics comes from Far Eye, an AI-led delivery logistics platform, that helped one of its key clients BlueDart improve first attempt success rate by 22% and delivery success rate by 75%.

AI can be used to improve patient outcomes by evaluating patient data, including processing complex medical report imagery and developing personalized treatment plans. One of the biggest players disrupting the field of AI in healthcare is IBM with its “Watson for Oncology”, a SaaS that delivers an advanced ability to analyze the meaning and context of structured and unstructured data in clinical notes and reports, easily assimilating key patient information written in plain English. By combining attributes from the patient’s file with clinical expertise from Memorial Sloan Kettering, external research, and data, Watson for Oncology identifies and ranks potential treatment plans and options. It has been implemented by several hospitals around the world, including established ones like Apollo in India.

4. Enterprise AI in marketing and CRM

It would be a safe assumption to make in saying that marketing is one of the biggest beneficiaries from the evolution of AI. Marketing is at its core all about understanding the consumer and their problems, crafting a messaging and value to appeal to that consumer, and presenting the solutions at the right time and at the right place. Various AI tools, systems and processes have been continuously enriching marketing. From tools to analyse large amounts of customer data, to generative AI content generators like ChatGPT and Grok, to AI-assisted CRM tools that have made personalization at scale a possibility, AI in marketing has been a boon.

One of the case studies in how a large company has been Uber Eats that used Salesforce’s Einstein’s AI that powers its service operations. With a shared view of merchants, end consumer data, and uber-fast response times in issue resolution, the company can keep 25 million restaurateurs and their customers happy.

Sitting at the confluence of technology, innovation, strategy and foresight, the world is limitless in how AI can be leveraged to improve systems, provide solutions, and drive business growth. Enterprises, by harnessing the ever-evolving powers of AI, can improve business outcomes not only for their top line growth and revenue, but also in offering superior customer experiences and service.

To learn more, reach out to us today or email us at contact@dataweave.com.

January 31, 2024