Blog

AI’s biggest wins in 2026 won’t come from new models.
They’ll come from the discipline of evaluating, testing, measuring, and proving what actually works for the user.

Behind the scenes, something is shifting: contracts, budgets, even compliance are all starting to demand evidence, not demos.

We’ve lived through three fast years:

• 2023: The LLM rush (everyone worshipped the models).
• 2024: The POC flood (everyone “tried” AI).
• 2025: The Agent year (everyone’s wiring tools and workflows).

2026 will be the year evals go from “nice to have” to contractual – the thing buyers and regulators ask for before you deploy, and the thing finance teams ask for before they fund.

But let’s get precise about what I mean by “evals,” because this word is overloaded.

---

Product evals ≠ model evals (and ≠ observability)

In this blog, when I mention “Evals”, I am mainly pointing towards product evals and obervability in a blanket term. Let’s understand the difference first.

• Model (LLM) evals measure capability in controlled tasks (reasoning, safety, accuracy). These are useful for model selection, not sufficient for business sign-off.
• Product evals measure outcomes in a live product: customer impact, risk, cost, reliability. Think: A/B results, guardrail pass-rates, time-to-resolution, cost-to-serve, incident rates, and audit-ready traces.
• Observability watches operations in real time (latency, errors, spend, drift alerts). It’s how you keep the system healthy after you ship.

One-line rule of thumb: Evals decide “ship/keep.” Observability answers “what’s happening right now?” They work together, but they aren’t the same.

Enterprises buy product outcomes, not leaderboard wins. If your “evals” don’t connect to customer experience, risk, compliance, and ROI, you’re not building for the right outcome.

This is why 2026 tilts toward product evals. We’ll still run LLM evals, but they’ll be one input to a bigger, product-centric evidence loop.

---

A short timeline through the Eval lens

2023 – Leaderboards and lab metrics. We had a an explosion of models and academic benchmarks. Helpful for science, less helpful for CFOs. What did change: the conversation about transparent, reproducible evaluation started getting louder in the public sphere. Stanford’s HELM work on broader, reproducible benchmarking is a good marker of that shift.

2024 – Institutions formalize “measure before you trust.” NIST released a Generative AI Profile alongside its AI Risk Management Framework – explicitly pushing organizations to govern, measure, and manage risks with evaluation and monitoring built in. Translation: “trust” now requires evidence, not vibes.

The UK’s AI Safety Institute launched Inspect, an open platform to publish and run evaluations – primarily model-level, but the bigger signal is public bodies treating evaluation as infrastructure, not a one-off.

2025 – Evals slip into product workflows. While labs keep refining model tests, product companies keep doing what they’ve always done – experiment, measure, ship – just with AI in the loop now. Netflix, Uber, DoorDash, Booking.com, and LinkedIn have written openly for years about rigorous experimentation at product scale; that playbook is exactly what the AI era needs: tie changes to outcomes, at velocity, with guardrails.

2026 – Regulation + Procurement + Finance. The EU AI Act becomes fully applicable on August 2, 2026 (with gradations by risk). That puts conformity assessment, ongoing monitoring, and documentation in scope for many systems. Buyers in regulated sectors will ask for eval-derived evidence by default. This is the year product evals become the control plane for AI deployments.

---

Why Evals become non-negotiable in 2026

1. Regulators are asking for proof, not promises. NIST is telling organizations to measure and manage AI risks with concrete tests and monitoring. The EU AI Act puts time-bound obligations on evaluation and documentation. If you can’t show your tests, thresholds, and traces, you don’t have a compliance story.
2. Procurement teams want predictable outcomes. They’ll ask: “What’s your policy pass-rate? What happens when it fails? How fast can you detect drift? Show me the audit trail.” That’s product eval territory: live metrics, gates, fallbacks, and exportable proof bundles.
3. Finance wants the delta. Cost-to-serve, time-to-resolution, defect rate, and risk-adjusted loss. If evals can’t roll up into those numbers, budgets stall. .
4. Change never sleeps. Agents, prompts, and tools mutate weekly. Without eval gates and continuous checks, you ship regressions in the dark. Product evals are your headlights.

---

The market is signaling the same

Industry report states enterprises are losing an estimated $1.9 billion annually due to undetected LLM failures. This suggests the market problem is real and large, but also that current solutions haven’t fully solved it yet.

AI evaluation startups are experiencing rapid growth, with companies like Arize AI raising $70 million, Galileo raising $45 million, Braintrust securing $36 million, and newer entrants like Scorecard AI ($3.75 million) and Trismik (£2.2 million) attracting significant funding in 2025.

These products serve major enterprises including Notion, Stripe, BCG, Microsoft, AstraZeneca, and Thomson Reuters, demonstrating strong enterprise adoption across finance, healthcare, and technology sectors.

Model providers are also moving from quiz-style benchmarks to economically grounded evaluations. OpenAI’s recent announcement around evaluating models on “economically valuable, real-world tasks” is a signal of where the industry is heading: evaluations that look like work, scored in ways executives can understand. I’m not using it as a yardstick in this piece, just noting the shift in mindset: evals as evidence for real work, not just leaderboard points.

Public bodies are pushing too: the UK’s AI Safety Institute open-sourced evaluation tooling (Inspect) to make it easier for the whole ecosystem to measure consistently. Again, the signal is the same: evaluation is infrastructure.

---

The enterprise playbook for 2026

Step 1 – Define success in business terms. Pick the top one or two workflows. Baseline: cost-to-serve, time-to-resolution, defect rate, incident likelihood. This is important, if you skip this, you can’t show ROI later.

Step 2 – Turn policies into tests. Privacy, safety, factuality, refusal correctness, brand tone. Automate checks where you can; keep human review for what genuinely needs judgment. Look at NIST’s guidance to move beyond documentation, and measure and manage. I love Hamel’s guidance on Evals, they are practical and makes sense.

Step 3 – Build the gate. Ensure no change ships without passing scenario tests that mirror the real workflow. Treat every model/prompt/tool update as a release candidate.

Step 4 – Deploy with canaries and a kill switch. Expose to a small slice. Compare against baseline. Auto-rollback if guardrails trip or metrics regress. I would take inspiration from Netflix’s Sequential Testing principles.

Step 5 – Log everything. Prompts, versions, model/tool hashes, data lineage, evaluator settings, results, sign-offs. You’re building your audit pack as you operate, so log everything.

Step 6 – Report like an owner. Every month, share a simple one-pager: how policies performed, what it cost vs. expected, where risks went down, and what you changed as a result. That’s how you build trust and keep the budget flowing.

---

What changes in 2026

• What’s changing: Evals are moving from “nice to have” to contractual. RFPs don’t just ask “does it work?” – they now demand policy pass-rates, fallback design, drift detection speed, and evidence. CFOs aren’t satisfied with burn rates – they want cost, time, and risk deltas straight from your evaluation ledger. And compliance isn’t waiting for annual PDFs anymore – they expect continuous monitoring.
• What’s not changing: The simple truth: great products still come from disciplined experimentation. The companies that learned this muscle memory in Web2 – measure, learn, ship – are about to lap everyone in AI. Because eval-led AI is just that same playbook, turned up to eleven.

---

2026 will be about adding confidence to every decision AI makes

The coming year is shaping up to be the year of AI evals. Not as an academic curiosity, not as a side-note in model papers, but as the backbone of how AI gets built, bought, and trusted. Budgets, contracts, and compliance are all shifting to an eval-first mindset.

The companies that master this shift will build safer, smarter systems, they’ll build faster, learn faster, and win faster.

The real question is: will you treat evals as a checkbox, or as the operating system of your AI strategy?

I have been playing with Zerve.
It’s a game-changer for Data Scientists.

Try it here: https://bit.ly/3VGSaO8

It’s changing how Data Science projects are done.

I asked it to compare two models on a Diabetes dataset to predict diabetes risk.

Here’s what happened 👇

🔹 Zerve ingested a dataset
with patient health metrics
(age, glucose, BMI, etc.)

🔹 Preprocessed for missing values

🔹 Built and trained two models in parallel
→ Logistic Regression & Random Forest

🔹 Evaluated results with accuracy scores,
confusion matrix, ROC curve, feature importances

🔹 Saved all outputs + code
as tracked artifacts for reproducibility

And all of this was orchestrated by 𝐙𝐞𝐫𝐯𝐞 𝐚𝐠𝐞𝐧𝐭𝐬
in a modular, transparent workflow.

👉 Each step ran as a separate block
→ easy to inspect,
→ change,
→ re-run without losing context

👉 Models ran in parallel using distributed compute
→ no manual setup

👉 All artifacts (data, code, results)
→ were versioned and traceable

👉 It kept me in the loop,
→ I steered the whole process,
→ while agents handled the heavy lifting

This isn’t about replacing the Data Scientist.
It’s about accelerating while keeping you in control.

That’s why Zerve feels so different.

If you’re working in data science or AI,
this is one product to watch.

Why not give it a try?

𝐇𝐞𝐫𝐞 𝐚𝐫𝐞 𝟏𝟎 𝐩𝐨𝐰𝐞𝐫𝐟𝐮𝐥 𝐰𝐚𝐲𝐬 𝐭𝐨 𝐜𝐨𝐧𝐭𝐫𝐨𝐥 𝐀𝐈 𝐫𝐞𝐬𝐩𝐨𝐧𝐬𝐞𝐬 𝐟𝐫𝐨𝐦 𝐭𝐨𝐧𝐞 𝐚𝐧𝐝 𝐝𝐞𝐩𝐭𝐡 𝐭𝐨 𝐜𝐫𝐞𝐚𝐭𝐢𝐯𝐢𝐭𝐲 𝐚𝐧𝐝 𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞:

Most people think prompt writing is just about typing a question. It is not.
If you want to control how your AI agent thinks, talks, and behaves, you need to go deeper.

𝐇𝐞𝐫𝐞 𝐚𝐫𝐞 𝟏𝟎 𝐩𝐨𝐰𝐞𝐫𝐟𝐮𝐥 𝐰𝐚𝐲𝐬 𝐭𝐨 𝐜𝐨𝐧𝐭𝐫𝐨𝐥 𝐀𝐈 𝐫𝐞𝐬𝐩𝐨𝐧𝐬𝐞𝐬 𝐟𝐫𝐨𝐦 𝐭𝐨𝐧𝐞 𝐚𝐧𝐝 𝐝𝐞𝐩𝐭𝐡 𝐭𝐨 𝐜𝐫𝐞𝐚𝐭𝐢𝐯𝐢𝐭𝐲 𝐚𝐧𝐝 𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞:

1. Step-by-Step Mode: Force structured, logical reasoning by breaking tasks into steps.

2. Format (Markdown): Decide how outputs are presented bullet points, tables, or paragraphs.

3. Top-p (Nucleus Sampling): Tune how creative or focused the AI’s word choices should be.

4. Stop Sequences: Tell the AI when to stop, perfect for structured outputs like code or JSON.

5. Frequency Penalty: Prevent repetitive answers by penalizing repeated words or phrases.

6. Temperature: Control creativity: low for factual answers, high for bold, inventive responses.

7. Max Tokens: Set how long the answer should be: short and crisp or detailed and deep.

8. Presence Penalty: Push the AI to explore new ideas instead of sticking too close to the prompt.

9. Instruction Framing: Define the role, goal, and constraints to shape the response (e.g., “Act as a data scientist…”).

10. Tone Parameter: Change the voice and personality from teacher to CEO to journalist.

These controls transform a basic chatbot into a precision tool one that answers exactly how you want.

It’s called 𝐏𝐫𝐨𝐦𝐩𝐭 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠 – not “prompt typing.”
Because prompts must be
➛ designed,
➛ tested,
➛ deployed,
➛ monitored,
➛ and secured
➛ just like any production system.

𝐏𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 𝐢𝐬 𝐧𝐨𝐭 𝐭𝐲𝐩𝐢𝐧𝐠
Prompts need to be repeatable, testable and maintainable, not one-offs.

𝐃𝐞𝐬𝐢𝐠𝐧
Prompt design is modular: role, task, constraints, format.
Good design enables reuse and governance.

𝐓𝐞𝐬𝐭 𝐭𝐨 𝐝𝐞𝐩𝐥𝐨𝐲
You must A/B test and regression-test prompts before production.
Data wins over intuition.
Text becomes executable logic: version it, bake in policies, and release with CI/CD guardrails.

𝐌𝐨𝐧𝐢𝐭𝐨𝐫
Track token usage, latency, failure modes and semantic drift with an observability layer for LLMs.

𝐒𝐞𝐜𝐮𝐫𝐞
Defend prompts from injection, unfiltered tool-calls, and data leakage – treat them like a security boundary.

𝐃𝐞𝐬𝐢𝐠𝐧. 𝐓𝐞𝐬𝐭. 𝐃𝐞𝐩𝐥𝐨𝐲. 𝐌𝐨𝐧𝐢𝐭𝐨𝐫. 𝐒𝐞𝐜𝐮𝐫𝐞.

That’s why we call it engineering,
because text becomes systems.

I have been using AI for work, managing my life, running my community, and learning ever day.

Here are 𝟏𝟎 𝐡𝐨𝐧𝐞𝐬𝐭 𝐜𝐨𝐧𝐟𝐞𝐬𝐬𝐢𝐨𝐧𝐬 𝐟𝐫𝐨𝐦 𝐦𝐲 𝐨𝐰𝐧 𝐣𝐨𝐮𝐫𝐧𝐞𝐲 𝐨𝐟 𝐮𝐬𝐢𝐧𝐠 𝐀𝐈 every single day – from building strategies to ideating content, running communities, and managing life.

𝟏. 𝐀𝐈 𝐨𝐯𝐞𝐫𝐮𝐬𝐞.
It’s tempting to delegate everything to it. But I’ve learned that AI works best when I start with intent and context.

I ask myself, “Do I need AI for this?”

𝟐. 𝐋𝐨𝐬𝐞 𝐲𝐨𝐮𝐫 𝐯𝐨𝐢𝐜𝐞.
AI can sound too polished, too generic. Now, I use it for structure and ideation. I make sure I am in control here. I love my imperfections more.

𝟑. 𝐏𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 𝐚𝐥𝐥 𝐝𝐚𝐲.
It’s easy to keep tweaking prompts instead of producing outcomes. You have to measure outcome – track your progress.

Don’t drown in “prompt experimenting”.

𝟒. 𝐀𝐈 𝐜𝐚𝐧 𝐦𝐚𝐤𝐞 𝐭𝐡𝐢𝐧𝐤𝐢𝐧𝐠 𝐥𝐚𝐳𝐲.
If you’re not careful, it’ll hand you answers before you’ve formed the question. Now, I come up with ideas and ask AI to challenge me. I brainstorm with it, rather than just following what it says.

𝟓. “𝐓𝐨𝐨 𝐝𝐞𝐩𝐞𝐧𝐝𝐞𝐧𝐭” 𝐨𝐧 𝐀𝐈.
I’ve realized leaders of the future won’t be the ones avoiding AI; they’ll be the ones who integrate with it responsibly. It is just like eating healthy, or driving responsibly.

Know when to use AI.

𝟔. 𝐀𝐈 𝐦𝐚𝐤𝐞𝐬 𝐲𝐨𝐮 𝐢𝐦𝐩𝐚𝐭𝐢𝐞𝐧𝐭.
When results come instantly, you expect brilliance instantly. Now I treat AI like a colleague – the better the context, the better the output. Voice prompt help me provide more context easily wothout typing much.

𝟕. 𝐘𝐨𝐮 𝐜𝐚𝐧 𝐭𝐫𝐮𝐬𝐭 𝐢𝐭 𝐭𝐨𝐨 𝐦𝐮𝐜𝐡.
Facts, tone, nuance – they all need your human filter. AI is a brilliant assistant, not an authority. Treat it such.

𝟖. 𝐀𝐈 𝐜𝐚𝐧 𝐜𝐥𝐮𝐭𝐭𝐞𝐫 𝐲𝐨𝐮𝐫 𝐰𝐨𝐫𝐤𝐟𝐥𝐨𝐰.
So many tools. So much noise. Simplicity wins here. Learn the basics, spend on one or two tools that can do most of the work.

𝟗. 𝐀𝐈 𝐜𝐚𝐧 𝐫𝐞𝐝𝐮𝐜𝐞 𝐫𝐞𝐟𝐥𝐞𝐜𝐭𝐢𝐨𝐧.
I use AI intentionally to reflect. I use AI to find patterns I don’t notice in my meeting notes, my podcast transcripts, my writing. That helps me improve.

𝟏𝟎. 𝐀𝐈 𝐢𝐬 𝐫𝐞𝐝𝐞𝐟𝐢𝐧𝐢𝐧𝐠 𝐰𝐡𝐚𝐭 ‘𝐰𝐨𝐫𝐤’ 𝐦𝐞𝐚𝐧𝐬.
It’s shifting me from doing tasks to designing systems of work. That’s the real transformation – and the one I’m most excited about.

AI isn’t replacing us. It’s revealing where our human advantage truly lies.

In clarity, creativity, and critical thinking.

What’s your biggest confession about using AI so far?

Every week, Data Engineers tell me the same thing
‘𝐀𝐈 𝐢𝐬 𝐜𝐨𝐦𝐢𝐧𝐠 𝐟𝐨𝐫 𝐦𝐲 𝐣𝐨𝐛.’
But here’s the truth:
AI doesn’t replace Data Engineers.
It depends on them – just in new ways.

The Data Engineer of yesterday
moved data from one place to another.

The Data Engineer of tomorrow?
➛You’ll make data ready for AI.
➛You’ll focus on meaning, not just movement.
➛You’ll make sure data isn’t just available,
it’s understandable to both humans and AI agents.

If you’re a Data Engineer today:
Start learning how AI uses your data
➛embeddings,
➛vector databases,
➛feature stores.

Because soon, your pipelines won’t just feed dashboards, they’ll feed AI agents that reason, plan, and make decisions.

𝐇𝐞𝐫𝐞’𝐬 𝐰𝐡𝐞𝐫𝐞 𝐲𝐨𝐮𝐫 𝐫𝐨𝐥𝐞 𝐢𝐬 𝐡𝐞𝐚𝐝𝐢𝐧𝐠:

🟧 𝐃𝐚𝐭𝐚 𝐏𝐫𝐨𝐝𝐮𝐜𝐭 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫 – You’ll build reusable, AI-ready data products. Think “data as an API.”

🟧 𝐕𝐞𝐜𝐭𝐨𝐫𝐎𝐩𝐬 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫 – You’ll manage embeddings, retrieval, and optimization, the new ETV (Extract, Transform, Vectorize) era.

🟧 𝐃𝐚𝐭𝐚 𝐓𝐫𝐮𝐬𝐭 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫 – You’ll own accuracy, explainability, and compliance
because bad data doesn’t just break dashboards, it breaks trust.

(You might not see these job titles,
but naming them gives you some direction)

🚀 𝐖𝐡𝐞𝐫𝐞 𝐭𝐨 𝐬𝐭𝐚𝐫𝐭 (𝐲𝐨𝐮𝐫 𝐀𝐈 𝐫𝐞𝐚𝐝𝐢𝐧𝐞𝐬𝐬 𝐫𝐨𝐚𝐝𝐦𝐚𝐩):

1. Learn how AI consumes data –
embeddings, context, and retrieval.
Study how the 𝐒𝐞𝐦𝐚𝐧𝐭𝐢𝐜 𝐋𝐚𝐲𝐞𝐫 evolves with AI.

2. Add an AI-ready feature to your pipeline
maybe a vector store or feature store.

3. Share your learnings –
become the voice that connects
data and AI in your team.

𝐇𝐞𝐫𝐞’𝐬 𝐰𝐡𝐚𝐭 𝐈 𝐬𝐞𝐞:
Across enterprises adopting AI, data engineers are being asked to design vector pipelines alongside ETL ones.

So don’t chase every new AI tool –
focus on the fundamentals.

Understand how data becomes knowledge –
that’s where your real edge lies.

Data Engineers aren’t being replaced –
𝐲𝐨𝐮’𝐫𝐞 𝐛𝐞𝐢𝐧𝐠 𝐮𝐩𝐠𝐫𝐚𝐝𝐞𝐝.

➛Learn how AI consumes your data.
➛Build pipelines that power LLMs.
➛Become the bridge between data and intelligence.

Because the AI revolution still runs on one thing:
You, the 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐬. 🚀

𝐌𝐋 ≠ 𝐃𝐋 ≠ 𝐀𝐈 ≠ 𝐆𝐞𝐧𝐀𝐈 ≠ 𝐑𝐀𝐆 ≠ 𝐀𝐈 𝐀𝐠𝐞𝐧𝐭𝐬

𝐇𝐞𝐫𝐞’𝐬 𝐚 𝐜𝐥𝐞𝐚𝐫 𝐛𝐫𝐞𝐚𝐤𝐝𝐨𝐰𝐧:𝟏. 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 (𝐌𝐋)Extract features manually. Train models to classify patterns.Example: Is this a cat or not?𝟐. 𝐃𝐞𝐞𝐩 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 (𝐃𝐋)Learns features + classification end-to-end. No hand-engineering.Built with neural networks and hidden layers.𝟑. 𝐀𝐫𝐭𝐢𝐟𝐢𝐜𝐢𝐚𝐥 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 (𝐀𝐈)An umbrella term for ML, DL, NLP, robotics, vision, and more.Think automation with intelligence.𝟒. 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 (𝐆𝐞𝐧𝐀𝐈)Users interact with LLMs that use tools and data sources to generate smart outputs.Example: ChatGPT, Claude, Gemini.𝟓. 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥-𝐀𝐮𝐠𝐦𝐞𝐧𝐭𝐞𝐝 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 (𝐑𝐀𝐆)Adds memory to GenAI.Brings external data (documents) into the LLM via embeddings and vector search.𝟔. 𝐀𝐈 𝐀𝐠𝐞𝐧𝐭𝐬Takes GenAI to the next level.Agents act autonomously using tools, memory, logic, and reasoning.They don’t just respond, they do.