From hours of catalogue research to a structured product-matching workflow.to a structured product-matching workflow.

Finding a competitor equivalent for an industrial product was not a simple search.

The work meant opening different manufacturer catalogues, navigating different category structures, finding products that looked comparable, checking technical specifications and prices, accounting for variants, and then deciding whether the comparison actually made sense.

The information was already online.

The problem was that every manufacturer published it differently.

Product names did not line up. Categories were organised differently. Technical specifications used different labels and formats. Some pages loaded dynamically. Some products had variants. Some sources exposed information that others did not.

So the project was not simply about scraping competitor websites.

I needed to turn those disconnected catalogues into a shared product model, make technically related products discoverable, and give the team one place to inspect the strongest candidates instead of rebuilding the comparison manually every time.

The clean comparison screen sits on top of a much less clean data problem.

Pages can load dynamically. Product listings can paginate differently. Variants may need their own requests. Technical specifications can repeat. Fields can be missing. Price structures can vary. Requests can fail temporarily. The same concept can appear under different category names across sources.

I accounted for those cases in the ingestion layer rather than pushing the complexity into the interface.

The crawler setup includes throttling, retries, duplicate handling, dynamic-page rendering where needed, and logging for URLs that still fail after repeated attempts.

That separation matters.

The matcher should work with structured product data.

It should not need to know how difficult a particular website was to crawl.

1. Source catalogues

   Provide the raw product information.
2. Crawler layer

   Navigates each source, extracts product information, handles dynamic pages, variants, pricing, and technical data.
3. Data pipeline

   Normalises the incoming records and maps them into the shared product structure.
4. Postgres / Supabase

   Stores companies, catalogue hierarchy, products, prices, technical data, variant relationships, and semantic embeddings.
5. Vector search

   Finds likely competitor candidates based on the semantic representation of the selected product.
6. Internal application

   Lets authenticated users search the source catalogue, run matching, rank candidates, and inspect products side by side.

The application sits on top of those layers and turns the data pipeline into a repeatable research workflow.

A user can start with an internal product, retrieve likely competitor alternatives, and inspect the relevant product information in one comparison flow.

The system moves much of the repetitive work — finding catalogue pages, collecting the information, structuring it, and identifying likely candidates — into software.

That leaves the user with the part where human judgment is more valuable: deciding whether a candidate is technically and commercially meaningful.

The project also created something broader than one matching screen.

Once products from different manufacturers live inside a shared structure, the same data foundation can support future competitive analysis, price research, catalogue-gap analysis, and other internal product-intelligence workflows.

The current system uses semantic similarity to discover strong candidates.

I would evolve that into a hybrid matcher.

First, technical values would be normalised into canonical attributes and units. Different representations such as kilograms and tonnes should become the same machine-readable fact before comparison.

From there, semantic similarity could be combined with hard compatibility rules and weighted technical attributes. A product should not rank highly simply because its description sounds similar if an essential specification makes it unsuitable.

I would also introduce a manually verified evaluation set of known product pairs and measure the matcher with metrics such as Precision@K, Recall@K, false-positive rate, and confidence calibration.

On the ingestion side, I would add stronger provenance and incremental updates: which source produced a value, when it was retrieved, what changed since the previous crawl, and where catalogue structure has drifted.

That would make the system easier to monitor, test, and improve as the underlying catalogues change.

I worked from the data-acquisition problem through the internal product experience.

That included source-specific crawler architecture, dynamic-page handling, catalogue mapping, data normalisation, relational product modelling, product variants, prices and technical specifications, embeddings, vector indexing, semantic matching, Supabase integration, authentication, product search, match ranking, and the side-by-side comparison experience.

Data acquisition

Data & storage

AI & matching

Application

Authentication