Evaluations
A test set for every AI feature, covering itinerary feasibility, policy accuracy, grounding, tone and refusals. It runs before every change and on a schedule after.
Services · Managed AI Operations
Launch day is when the AI's real job starts
We run evaluations, monitoring, drift detection, cost control and incident response for the AI features in your travel business, whether we built them or not. Every month we sit down with you to show what changed and why.
AI features fail quietly
Traditional software breaks loudly: an error page, a failed payment, an alert at 3 AM. AI features mostly don’t. They drift.
A model provider ships an update, and your concierge’s tone goes from warm to clipped. A supplier renames its room types, and the proposal generator keeps offering a suite that no longer exists. A trail closes for the season, and the assistant keeps recommending it because the closure lives in an email instead of a field. We call these phantom venues, and grounding travel AI in live data is how you prevent them. Meanwhile a retry loop doubles the month’s model spend before anyone opens the invoice.
None of this sets off an alarm. Travelers notice before you do.
Most travel businesses don’t have, and shouldn’t need, a full-time team watching for this. But someone has to, with a test set, a dashboard, a budget and a phone number. That holds whether the AI is a planner on a destination organization’s website, a guest assistant at a lodge or a proposal tool for travel advisors.
What we run for you
Monitoring
Latency, errors, groundedness, hand-off rates and traveler feedback, sampled and scored, with alerts tied to thresholds you approve.
Drift detection
Shifts in what travelers ask, what supplier data says and how models answer, flagged before they turn into complaints.
Cost control
Budgets per feature, caching, model routing and usage alerts, so spend follows value instead of surprises.
Incident handling
Severity levels, runbooks and rollback for AI-specific failures: a wrong policy quoted, a phantom venue, a prompt-injection attempt.
Monthly reviews
A working session with your team covering what we measured, what we changed, what we recommend and what it cost.
How it works
Getting started
- 01
Baseline
2 weeksWe inventory your AI features, read their prompts and data flows, and measure current quality and cost, including features another vendor built.
- 02
Instrument
2–3 weeksWe add tracing, build evaluation sets with your team, agree thresholds and connect alerts.
- 03
Operate
OngoingDaily checks, weekly evaluations and on-call incident handling under agreed severity levels.
- 04
Review
MonthlyThe session where findings become decisions.
Checklist
The operating rhythm
What a typical month looks like once we're running.
0 of 10 ready
What to expect after the first quarter
Managed operations is quieter than a build. Success mostly looks like things not going wrong. By the end of the first quarter, clients typically have:
- A quality score for each AI feature that trends over time, with every dip explained
- Model spend that's predictable month to month, and often lower once caching and routing are in place
- Calmer incidents, because problems are caught by checks instead of by travelers
- A test set that grows from real conversations, so a fixed problem stays fixed
- Clear evidence for the next investment, or for switching something off
Side by side
Unwatched AI versus managed AI
AspectUnmanagedManaged by Nova
QualityJudged by the last complaintScored every week against a fixed test set
Model updatesArrive unannounced and untestedEvaluated before you adopt them
SpendDiscovered on the invoiceBudgeted per feature, with alerts
IncidentsWhoever notices, fixes, or doesn'tSeverity levels, runbooks and an on-call rotation
DecisionsMade on anecdotesMade in a monthly review, with evidence
We used to find out the planner was wrong when a visitor emailed us a screenshot. Now we hear about it on Tuesday morning, with the cause and a proposed fix already attached.
What we operate most often
- PlatformConciergeIn-trip assistants on web chat, WhatsApp and SMS, where tone and grounding matter most.
- PlatformItinerary StudioProposal drafting, evaluated for feasibility, accuracy and brand voice.
- PlatformSignalsForecast models checked for drift as seasons and booking patterns shift.
- PracticesSecurity & trustOur practices for data handling, access and incident response.
Questions
Frequently asked questions
Will you manage AI features you didn't build?
Yes. Much of the work is the same whoever built the feature: test sets, monitoring, cost and incidents. We start with a baseline, so you know what we're inheriting before we commit to quality targets.
What exactly is an eval?
A repeatable test. It's a fixed set of realistic inputs, such as a family asking for a gentle day near Santa Fe or a traveler asking about a cancellation, scored for what a good answer must and must not contain. How we evaluate an itinerary generator walks through one.
How fast do you respond to incidents?
Response targets depend on severity and are set in your service terms. Our support page explains severity levels and channels. For urgent AI incidents, a feature can roll back automatically while a person investigates.
Can you help us switch model providers?
Yes, and the evaluation sets make it safer. We run the candidate model against the same tests, compare quality and cost, and switch only when the evidence supports it.
Who can see our data and traveler conversations?
Only the engineers assigned to your account, under least-privilege access, with traveler data minimized or redacted in logs where possible. Our security practices and Responsible AI service go into detail.
Related
Keep exploring
- CompanySupportSupport for Nova Adventure platform customers: email support@novadventure.com or call (505) 303-9060. Severity levels, response targets and business hours.
- PlatformSecurity & trustHow Nova protects traveler data and keeps AI grounded: encryption, least-privilege access, prompt-injection defenses, PII minimization and incident response.
- Field Notes · Strategy & TrustHow We Evaluate an Itinerary Generator Before Travelers See ItOur evaluation protocol for AI itinerary generators: a stratified test set, deterministic feasibility and grounding checks, a human rubric and release gates.
- Field Notes · Strategy & TrustPhantom Trails and Closed Cafés: Grounding Travel AI in Live DataWhy travel assistants recommend places that closed, moved or never existed, with a taxonomy of phantom venues and a grounding design that catches them first.
- ServicesResponsible AI & Traveler Data GovernanceConsent, minimization, careful handling of mobility, health and dietary details, bias reviews, red-teaming and clear policy for the AI in your travel business.
Put someone on watch
Tell us which AI features you run today and what worries you about them. We'll suggest what a baseline would measure first.