AGdev
AGdev

Product Intelligence & Competitive Matching

Confidential industrial B2B engagement

From hours of catalogue research to a structured product-matching workflow.to a structured product-matching workflow.

I designed and built an internal product-intelligence system for a confidential industrial B2B engagement.

The system collects product information from different manufacturer catalogues, turns it into a shared data model, finds likely competing alternatives, and gives the team one place to inspect the strongest matches instead of rebuilding the comparison manually each time.

Confidential product-intelligence interface for finding and comparing likely competing industrial products
Role
Full-Stack & AI Product Engineer
Ownership
Data acquisition → Product modelling → Matching → Internal application
Status
Internal product · Confidential engagement
Core flow
Collect → Structure → Match → Compare
Project snapshotThe challenge01 Shared product language02 Reusable product data03 Semantic matching04 Human judgement05 Comparison evidenceProduction realityHow it connectsThe outcomeWhat comes nextWhat I ownedBuilt with

Client identity, production data, competitor names, and proprietary implementation details are intentionally omitted.

[ Project snapshot ]

Problem

Competitive product research required moving through separate manufacturer catalogues, finding possible equivalents, and comparing technical information product by product. For a few products, that was manageable. Repeated across larger product ranges, the same work could consume hours and stretch across days.

[ Let’s work together ]

Turning fragmented operational data into a product people can trust?into a product people can trust?

I can help shape the data, system, and interface into one dependable workflow.

AGdev
ImprintPrivacy policy
ImprintPrivacy policy

© 2026 Aleksandar Gjurchevski. All Rights Reserved.

© 2026 Aleksandar Gjurchevski. All Rights Reserved.

What I built

A product-intelligence system that collects and structures catalogue data, maps products into a shared taxonomy, creates semantic product representations, finds likely competitor alternatives, and presents the strongest candidates inside an authenticated comparison application.

My responsibility

Crawler architecture, data acquisition, product modelling, database design, price and technical-data handling, embeddings, vector search, matching logic, authentication, internal UX/UI, and the product-comparison experience.

[ The challenge ]

Competitive product research was taking too much manual work.

Finding a competitor equivalent for an industrial product was not a simple search.

The work meant opening different manufacturer catalogues, navigating different category structures, finding products that looked comparable, checking technical specifications and prices, accounting for variants, and then deciding whether the comparison actually made sense.

The information was already online.

The problem was that every manufacturer published it differently.

Product names did not line up. Categories were organised differently. Technical specifications used different labels and formats. Some pages loaded dynamically. Some products had variants. Some sources exposed information that others did not.

So the project was not simply about scraping competitor websites.

I needed to turn those disconnected catalogues into a shared product model, make technically related products discoverable, and give the team one place to inspect the strongest candidates instead of rebuilding the comparison manually every time.

The goal was to move product research from catalogue-by-catalogue work into a structured matching workflow.
01

[ Different catalogues needed one product language ]

Before products could be matched, their data had to become comparable.

A single generic scraper would have been convenient.

It also would have hidden the actual problem.

Different sources used different page structures, category hierarchies, pricing formats, product-detail layouts, and ways of presenting technical specifications.

I kept the extraction logic source-specific, but made the output consistent.

Each crawler could understand the catalogue it was reading, while the data it produced had to fit the same internal structure: company, category, subcategory, product, variant relationships, prices, technical data, and source information.

Where competitor catalogues organised products differently, their categories were mapped back into the shared internal taxonomy before matching.

That gave the rest of the system one predictable representation to work with instead of a collection of unrelated scraped pages.

The crawler needed to understand the website. The product model needed to understand the business.
02

[ Collecting the data once was not enough ]

The system also had to preserve the details that make a comparison useful.

A product is more than its name and description.

Technical B2B comparison depends on details such as dimensions, materials, load capacities, availability, pricing, and product variants.

I modelled those concerns separately instead of flattening everything into one record.

Products keep their relationship to the company and catalogue hierarchy. Variants can remain connected to their parent product. Prices and technical specifications are stored independently so they can be inspected and updated without losing the product context around them.

The ingestion pipeline also normalises incoming values, avoids unnecessary duplicate records, and keeps the product data ready for the matching layer that follows.

The useful dataset was not the page I scraped. It was the product structure I could reuse afterwards.
03

[ Matching started with meaning, not identical words ]

Similar products do not always use similar names.

Exact text matching would have missed too much.

Two manufacturers can describe products for the same application using different names, terminology, or catalogue structures. A relevant characteristic may appear in the title on one site and inside the technical specifications on another.

So each product received a second representation alongside its structured database record.

The embedding pipeline combines the product's category context, name, designation, description, and technical specifications into a semantic product document.

That representation can then be searched through vector similarity to find products that are related by meaning rather than only by matching words.

Structured data tells the system what a product contains. Semantic search helps it find what the product resembles.
04

[ A similarity score was the start of the match, not the decision ]

Finding candidates and proving equivalence are different jobs.

This was an important boundary in the product.

A high semantic similarity score can tell us which products are worth looking at.

It cannot guarantee that two technical products are interchangeable.

The internal application therefore uses vector search to narrow the search space.

A user starts with a product from the source catalogue. The system retrieves the strongest semantic candidates, removes products from the same company, keeps the competitor results, attaches their similarity scores, and ranks the candidates from strongest to weakest.

The AI does not make the final commercial or technical decision.

It helps the user get to the right candidates faster.

The system suggests where to look. The person still decides whether the match makes sense.
05

[ A score was not enough to build trust ]

The user still needed to see why two products looked related.

A list of percentages would have made the system fast but difficult to trust.

So I built the comparison step around the underlying product information.

A user can open the selected product and a competitor candidate side by side and inspect the details that matter: product information, descriptions, technical specifications, pricing, availability, and source links.

That turns matching into a decision-support workflow instead of a black box.

The user is not asked to accept:

"These products match."

The system gives them:

"These are the strongest candidates. Here is the information you need to judge them."

The match reduces the search. The comparison provides the evidence.

[ Production reality ]

Product data gets messy long before the interface sees it.

The clean comparison screen sits on top of a much less clean data problem.

Pages can load dynamically. Product listings can paginate differently. Variants may need their own requests. Technical specifications can repeat. Fields can be missing. Price structures can vary. Requests can fail temporarily. The same concept can appear under different category names across sources.

I accounted for those cases in the ingestion layer rather than pushing the complexity into the interface.

The crawler setup includes throttling, retries, duplicate handling, dynamic-page rendering where needed, and logging for URLs that still fail after repeated attempts.

That separation matters.

The matcher should work with structured product data.

It should not need to know how difficult a particular website was to crawl.

The matching screen is the visible part. Most of the reliability work happens before a product ever reaches it.

[ How the system connects ]

Each layer solves a different part of the problem.

  1. Source catalogues
    Provide the raw product information.
  2. Crawler layer
    Navigates each source, extracts product information, handles dynamic pages, variants, pricing, and technical data.
  3. Data pipeline

[ The outcome ]

Product research could start from one product instead of several catalogues.

A user can start with an internal product, retrieve likely competitor alternatives, and inspect the relevant product information in one comparison flow.

The system moves much of the repetitive work — finding catalogue pages, collecting the information, structuring it, and identifying likely candidates — into software.

That leaves the user with the part where human judgment is more valuable: deciding whether a candidate is technically and commercially meaningful.

The project also created something broader than one matching screen.

Once products from different manufacturers live inside a shared structure, the same data foundation can support future competitive analysis, price research, catalogue-gap analysis, and other internal product-intelligence workflows.

Less time searching for what might match. More time deciding what the match means.

[ What I would improve next ]

The next step is not more AI. It is better evidence.

The current system uses semantic similarity to discover strong candidates.

I would evolve that into a hybrid matcher.

First, technical values would be normalised into canonical attributes and units. Different representations such as kilograms and tonnes should become the same machine-readable fact before comparison.

From there, semantic similarity could be combined with hard compatibility rules and weighted technical attributes. A product should not rank highly simply because its description sounds similar if an essential specification makes it unsuitable.

I would also introduce a manually verified evaluation set of known product pairs and measure the matcher with metrics such as Precision@K, Recall@K, false-positive rate, and confidence calibration.

On the ingestion side, I would add stronger provenance and incremental updates: which source produced a value, when it was retrieved, what changed since the previous crawl, and where catalogue structure has drifted.

That would make the system easier to monitor, test, and improve as the underlying catalogues change.

A useful matcher finds candidates. A dependable matcher can explain how confident it should be in them.

[ What I owned ]

I worked across the complete system.

I worked from the data-acquisition problem through the internal product experience.

That included source-specific crawler architecture, dynamic-page handling, catalogue mapping, data normalisation, relational product modelling, product variants, prices and technical specifications, embeddings, vector indexing, semantic matching, Supabase integration, authentication, product search, match ranking, and the side-by-side comparison experience.

  • Data acquisition01
  • Python crawling02
  • Data modelling03
  • PostgreSQL / Supabase04
  • Embeddings05
  • Vector search06
  • Matching logic07
  • Authentication08
  • Full-stack application09
  • Product UX/UI10
  • Reliability11
  • Data acquisition01
  • Python crawling02
  • Data modelling03
  • PostgreSQL / Supabase04
  • Embeddings05
  • Vector search06
  • Matching logic07
  • Authentication08
  • Full-stack application09
  • Product UX/UI10
  • Reliability11

[ Built with ]

Data acquisition
  • Scrapy
Normalises the incoming records and maps them into the shared product structure.
  • Postgres / Supabase
    Stores companies, catalogue hierarchy, products, prices, technical data, variant relationships, and semantic embeddings.
  • Vector search
    Finds likely competitor candidates based on the semantic representation of the selected product.
  • Internal application
    Lets authenticated users search the source catalogue, run matching, rank candidates, and inspect products side by side.

    • The application sits on top of those layers and turns the data pipeline into a repeatable research workflow.

    Collect once. Structure once. Reuse the intelligence every time the next comparison starts.
  • Playwright
    • Data & storage
    AI & matching
    • Claude
    Application
    Authentication
  • Motion