Supplier Truth: Content Mapping, Deduplication and the Data Behind Personalization

The same sunrise kayak trip appeared in one operator’s catalog four times. The reservation system called it “Sunrise Paddle.” A wholesaler feed listed “Sunrise Kayak Tour (2.5h).” A spreadsheet the sales team kept had “Lake paddle — AM.” And a page scraped from the supplier’s website two seasons earlier offered a “Kayak sunrise experience.” Four records, three durations, two prices, and a cancellation policy on one of them that contradicted the other three.

The recommender treated them as four products. A traveler who asked for “something on the water, early” got three of them in her top five, and the two genuinely different options — a guided birding walk along the shore and a stand-up paddle lesson — never made the list.

No amount of model tuning fixes that. It is a catalog problem, and it sits underneath every personalization feature a travel business wants to build.

Why personalization starts in the catalog

Ranking, routing and preference learning all assume that each item in the catalog is a distinct thing in the world. When that assumption breaks, the damage spreads quietly:

• Diversity collapses. Duplicates occupy slots that should go to alternatives. A top-five list with three copies of one tour is really a top-three list.
• Popularity signals split. Bookings spread across four records make a bestseller look like four mediocre products, so it ranks lower than it should.
• Routing inherits conflicts. If one record says two hours and another says two and a half, the scheduler either picks one arbitrarily or pads the day for both.
• Trust erodes. A traveler who sees two prices for the same trip assumes one of them is a trick.

We use the phrase supplier truth for the fix: one canonical record per real-world thing, every field traceable to an authoritative source, every fact carrying the date it was last verified.

Anatomy of a messy catalog

Here is a composite of the kind of cleanup we see often. A destination management company in the Mountain West pulls from five sources:

Source	Records	Typical strengths	Typical weaknesses
Reservation system	412	Live availability, internal IDs	Terse names, no descriptions
Wholesaler feed A	690	Rich descriptions, photos	Stale policies, its own naming
Wholesaler feed B	318	Net rates	Duplicates within itself
Supplier spreadsheets	260	Guide knowledge, effort levels	Free text, no coordinates
Scraped web content	160	The supplier’s own words	Often two seasons old

That is 1,840 records. After mapping, they resolved to 1,126 canonical entities. The 714 surplus records were not all the same kind of mess, and treating them alike is the first mistake:

• Exact duplicates — the same feed item ingested twice. Collapse them.
• Near duplicates — “Sunrise Paddle” and “Sunrise Kayak Tour (2.5h).” Merge into one entity and keep both names as aliases, because travelers and partners will use both.
• Variants — a private and a shared departure of the same trip. One product with two options; not two products, and not one blurred record.
• Parent and child — a lodge and its cabins. A hierarchy, never a merge.
• Look-alikes — two different outfitters running sunrise paddles from the same beach. Keep them separate. This is the dangerous category.
• Ghosts — products the supplier retired that still sit in a feed. Retire them with a tombstone so they can’t quietly return on the next import.

Match scoring: deciding two records are one thing

Block first, then compare

Comparing every record with every other record means 1,691,880 pairs for this catalog. Most are absurd; a balloon flight will never match a cooking class. Blocking restricts comparison to plausible candidates — here, records within 2 km of each other and in the same broad category — which cut the work to roughly 23,000 pairs.

Score on several signals

Within each block, score every pair on independent signals. A reasonable starting set, with weights you will calibrate later:

Signal	Weight	Notes
Name similarity	0.30	After stripping words like “tour” and “experience,” durations and punctuation
Location	0.25	Distance between meeting points, not business addresses
Supplier identity	0.25	Same legal entity, phone or email domain
Schedule	0.10	Overlapping start times
Duration and category	0.10	Compatible length and type

Price is deliberately missing. Net and retail rates differ by design, so price proximity misleads more often than it helps.

Two worked pairs

Sunrise Paddle vs. Sunrise Kayak Tour (2.5h). Name 0.82, location 0.95 (launch points 180 m apart), supplier 1.0 (same phone number), schedule 1.0 (both 6:30), duration 0.6 (two hours vs. two and a half). Weighted score: 0.246 + 0.238 + 0.25 + 0.10 + 0.06 = 0.89.

Sunrise Paddle vs. Dawn Kayak Tour, run by a different outfitter from the same beach. Name 0.55, location 0.98, supplier 0.0, schedule 1.0, duration 1.0. Score: 0.165 + 0.245 + 0 + 0.10 + 0.10 = 0.61.

The second pair is the instructive one. A naive matcher keyed on place and time would merge them, and a family would be sent to the wrong operator’s van. So on top of the weights, add hard rules. The most important: a different legal supplier is a veto, whatever the score.

Set thresholds asymmetrically

A missed merge produces a duplicate in a list. A false merge can leave a family at the wrong meeting point at 6:15 in the morning. The errors are not symmetric, so the thresholds shouldn’t be either:

• 0.92 and above: merge automatically.
• 0.70 to 0.92: send to a reviewer, with the two records side by side and the disagreeing fields highlighted.
• Below 0.70: keep separate.

In the example, about 640 pairs merged automatically and 410 went to review. With a decent side-by-side screen, a reviewer clears around 60 pairs an hour, so the initial backlog was a little under seven hours of human work. After launch, the weekly queue settled at 20 to 40 pairs.

Then measure both error types. Each month, audit a random sample of 100 automatic merges (target: fewer than one wrong) and 100 pairs scored just below the review band (how many should have merged?). Before you start, calibrate the weights by hand-labeling 200 pairs. It takes an afternoon and saves weeks.

Field-level authority: who gets to say what’s true

Once records are merged, their fields still disagree. The tempting rule — most recent write wins — is wrong. A wholesaler feed updated hourly is not more authoritative about a cancellation policy than the signed contract. It is merely more frequent.

Assign authority per field, and give each field a freshness clock:

Field	Authoritative source	Freshness clock	When the clock runs out
Operating status	Supplier confirmation	90 days	Demote from recommendations; request re-confirmation
Availability and price	Reservation system or live feed	Minutes	Don’t quote; re-check before showing
Cancellation policy	Supplier contract	On contract change	Show “confirmed at booking”
Meeting point	Supplier, verified on a map	180 days	Flag for verification
Duration and effort level	Your guides and operations team	Each season	Keep, but don’t use for tight scheduling
Accessibility details	Supplier questionnaire plus site check	180 days or after renovation	Withhold the claim
Seasonal calendar	Supplier, confirmed before season	Each season	Treat as closed until confirmed
Description and photos	Content team	12 months	Low risk; refresh on schedule

Two rules make the table work. First, a stale field changes behavior, not just a dashboard color: the itinerary builder stops asserting it. Second, store provenance with every value — where it came from, when, and who confirmed it — so a reviewer can resolve conflicts without detective work.

The dividend: attributes personalization can use

Merging pays a dividend that is easy to miss. Attributes scattered across sources come together on one entity. In the example, effort level appeared on only 31% of source records, almost all from the guides’ spreadsheets. After merging, 78% of canonical entities had it, because the spreadsheet row and the feed row finally pointed at the same thing.

That matters because the things travelers actually say — “nothing too strenuous,” “shade in the afternoon,” “kids aged six and nine,” “step-free” — only become useful when they can be matched against attributes on the canonical record. A preference engine can only be as precise as the vocabulary of its catalog. Whether you are generating proposals in Itinerary Studio or scheduling days with a constraint-based routing engine, those attributes are the levers, and they only work when each one is attached to the right thing.

Keeping it true after the cleanup

A one-time cleanup decays within a season. Keep the pipeline running:

• Match on arrival. Every new or changed source record runs through blocking and scoring before it touches the live catalog.
• Diff authoritative fields. When a contract-owned field changes in a feed, raise an event for a person instead of overwriting.
• Tombstone, never delete. Retired entities keep their ID and aliases, so a stale feed can’t resurrect them.
• Make re-confirmation cheap. A short pre-season message to each supplier, listing the five fields you rely on, gets far better response than a long form.
• Feed booking reality back. When supplier confirmations disagree with the catalog — a different meeting point, a new start time — treat the confirmation as evidence and queue the field for review.

Track four numbers on one page: duplicate rate in top-ten results, share of fields past their freshness clock, review-queue age, and audited false-merge rate.

A two-week starting plan

1. List every source with its record count and refresh frequency.
2. Pick one category and one region; guided day activities near your busiest base make a good first slice.
3. Normalize names, coordinates and supplier identities.
4. Block, score, and hand-label 200 pairs to calibrate thresholds.
5. Write the field-authority table with operations, contracting and content in the same room.
6. Set freshness clocks and publish a weekly stale-field report.
7. Switch recommendations and routing to canonical IDs only after the first audit passes.

None of this is glamorous, and almost all of it is decided by people who know the suppliers. When Nova begins a travel data engineering engagement, the field-authority table is usually the first artifact we produce, before any model is touched — because every system feeding the catalog, from reservation platforms to channel managers and wholesaler feeds, believes it is the source of truth. Per field, only one of them can be.