How We Evaluate an Itinerary Generator Before Travelers See It

The draft read beautifully. A sunrise hot-air balloon flight, a slow breakfast, a morning in a small history museum, lunch on a shaded patio. It was warm, specific and on-brand. It was also impossible. The 6:40 AM balloon launch was an hour’s drive from the 7:15 breakfast, the museum was closed that day of the week, and the patio restaurant had closed for the season.

Fluency is the easiest property of a generated itinerary to see and the least important to measure. Everything that matters to a traveler (does this place exist, can I get there, will I be exhausted, is it what I asked for) hides underneath good prose.

The stakes are not hypothetical. In an August 2026 study by Mindtrip with Sabre, Sojern and MMGY Travel Intelligence, 40% of U.S. travelers said they had discovered a destination through AI, and 29% said they had changed plans based on an AI recommendation. Recommendations move people. A wrong one moves them to a locked door.

This is the protocol we run before an itinerary generator is allowed near a traveler. It’s the same whether we built the generator or are assessing someone else’s.

Grade in order of severity

We grade five properties, in this order. A failure at any level makes the levels after it irrelevant.

1. Truthful. Every venue, route and opening time exists and is current.
2. Feasible. The day can actually be done: travel times, opening windows, transfers, pacing.
3. Safe and within policy. No unsafe suggestions, no promises of availability that wasn’t checked, every stated need respected.
4. Fit. It matches the brief: interests, budget band, pace, party.
5. Voice. It’s clear, on-brand and honest about uncertainty.

A gorgeous paragraph about a closed restaurant is a truthfulness failure, not a voice success. Ordering the grades this way stops teams from celebrating improvements that don’t matter.

Build the test set before you tune the prompt

The most common mistake we see is tuning a generator against a handful of briefs the team wrote themselves, then being surprised by real travelers. Build the test set first, from real inquiries, anonymized and used with consent, and stratify it so the hard cases are deliberately overrepresented.

A starting composition for 300 briefs:

Slice	Share	What it tests
Plain briefs: clear interests, no special constraints	30%	The baseline
Constraint-dense: access needs, dietary needs, young kids, fixed reservations	25%	Where feasibility breaks
Seasonal edges: monsoon afternoons, holiday closures, shoulder-season hours	15%	Whether the generator knows the calendar
Ambiguous or contradictory (“relaxing, but see everything”)	15%	Whether it asks before it guesses
Adversarial: impossible or unsafe requests, instructions hidden in supplier text, oversharing of sensitive data	15%	Whether it holds the line

Each brief carries its expected checks: which constraints apply, what a correct response must include, what it must never do. Writing those expectations down is tedious, and it’s most of the value.

What adversarial means in travel

Generic red-teaming misses the failures that matter here. Our adversarial slice always includes a request that can’t be done (“all five national parks in two days”), a request that shouldn’t be done (a ridge hike on a monsoon afternoon), a supplier description containing hidden instructions aimed at the model, and a traveler who volunteers health details the generator has no reason to repeat back. The right answer to the first two is a polite, specific “here’s what’s possible instead.” The right answer to the last two is to ignore the bait.

Machine checks first, model graders second

Run deterministic checks before anything subjective. They’re cheap, repeatable and impossible to argue with.

• Grounding. Every named place resolves to a record in your supplier and venue data, with a status that’s open on the proposed date. Anything that doesn’t resolve is a phantom. Our note on phantom venues covers how they creep in.
• Time feasibility. Recompute every leg with a real routing service. Flag any gap shorter than travel time plus a buffer, and any visit outside opening hours.
• Pacing. Score each day against the traveler’s stated pace and an energy budget. A day can work on the clock and still be impossible for a 70-year-old at 7,000 feet.
• Constraint adherence. Every stated need maps to a check. Step-free means every venue on the route has a step-free entrance on record; a nut allergy means every meal stop has a confirmed allergy policy.
• Availability language. Words like “booked,” “reserved” or “guaranteed” without a matching hold are automatic failures.
• Invented specifics. Prices, hours or phone numbers in the text that don’t match source data fail the itinerary.

Feasibility and pacing checks are where a route engine earns its keep: the same constraint model that builds days can audit them.

Only after the deterministic checks pass do model-graded checks come in, for fit and voice. Calibrate them against human reviewers on a sample first, and keep re-checking agreement. A model grader that disagrees with your advisors a quarter of the time is measuring something, but not quality.

The human rubric

Experienced advisors review a sample of generated itineraries on a four-point scale. Anchors keep scores consistent from one reviewer to the next.

Dimension	A 1 looks like	A 4 looks like
Fit	Generic; ignores half the brief	Clearly built for this traveler; nothing irrelevant
Feasibility (sense check)	An advisor wouldn’t send it	Would send it as is
Pacing	Exhausting, or empty	Rhythm matches the stated pace, with slack
Honesty	States guesses as facts	Flags uncertainty; never overpromises
Voice	Off-brand or overwritten	Sounds like your best advisor on a good day

Two reviewers score each sampled itinerary. Where they disagree by two points or more, the pair discusses it in a weekly calibration session. Those disagreements are often where the rubric itself needs to change.

We also track substantive edits: in a draft-don’t-decide workflow, how many meaningful changes an advisor makes before sending. It’s the most honest quality signal we know, because it measures what an expert actually did rather than what they said. Teams of travel advisors usually trust this number long before they trust any dashboard.

Release gates

Gates turn evaluation into decisions. These are the thresholds we start from. Tune them to your own risk tolerance, but write them down before you see the results.

Metric	Gate to shadow mode	Gate to travelers
Phantom venues	0 in 300 briefs	0 in 1,000 briefs, plus live monitoring
Hard-constraint violations (access, diet, fixed reservations)	0	0
Time-infeasible days	≤3%	≤1%
Days over pacing budget	≤8%	≤4%
Adversarial pass rate	≥90%	≥97%
Human rubric, average fit	≥3.0	≥3.3
Substantive advisor edits	Tracked	Median ≤2 per itinerary

Some gates are zero by design. A generator that occasionally invents a restaurant is not “98% accurate.” It’s a liability with good prose.

Why 300, then 1,000

Zero failures in a sample doesn’t mean a zero failure rate. A useful rule of thumb from statistics, the rule of three, says that if you see no failures in n independent trials, you can be about 95% confident the true rate is below 3/n. Zero phantoms in 300 briefs bounds the rate at roughly 1%, which is fine for an internal pilot and too loose for travelers. Zero in 1,000 brings the bound down to about 0.3%. That’s why the second gate needs the bigger sample, and why live monitoring has to continue after launch.

Shadow mode: real briefs, no travelers

Passing the test set earns shadow mode, not launch. For four to six weeks, the generator runs on live inquiries alongside the advisors. Its output goes to advisors, never to travelers, and advisors note whether they would have sent each itinerary and why not.

Shadow mode reliably finds failures the test set missed: a new supplier with an unusual naming convention, a local holiday the calendar didn’t know about, a style of request nobody anticipated. Each becomes a new test case before it becomes a traveler’s problem.

For a destination organization putting a planner on its public site, shadow mode matters even more, because there is no advisor between the generator and the visitor once it launches. The destination organizations we work with tend to run it longer for that reason.

After launch, evaluation becomes operations

Launch changes the cadence, not the discipline.

• Regression on every change. Any change to the model, the prompt or the supplier data reruns the full test set against the gates.
• Weekly sampling. Review around 2% of live itineraries with the human rubric.
• Seasonal drift checks. Before each season, add briefs for its edge cases: shoulder-season hours, monsoon afternoons, holiday closures.
• Incidents with teeth. A phantom venue in production is traced to its cause, fixed and added to the test set within the week.

This is the work our managed AI operations service runs for teams that would rather not staff it themselves, and it sits inside the wider practices described under responsible AI and traveler data governance.

The one-page version

Before an itinerary generator reaches a traveler, you should be able to say yes to each of these:

• A stratified test set built from real briefs, with constraint-dense and adversarial slices
• Deterministic grounding, feasibility, pacing and constraint checks on every output
• Model graders calibrated against human reviewers
• A written rubric with anchors, and a weekly calibration session
• Release gates written down before results were seen
• At least four weeks of shadow mode with advisors
• A regression suite that runs on every change, and an incident process that adds tests

If any box is empty, the generator isn’t ready, however good the prose sounds. When we put an itinerary generator in front of travelers at Nova Adventure, the empty boxes are where the work starts.