What Jev will do to data engineering

Every data team has a column they have promised sales for two years. This post is about that column.

You know the one. persona on the leads table, intent on support tickets, is_spam on signups, same_company_as on accounts. Everyone agrees it would be useful, there is a ticket for it (there has been a ticket for it for a long time, go look), and it still does not exist. It does not exist because its value is a judgment, and a warehouse has never had a good way to derive a judgment.

I think that is about to change, for the whole industry and fairly quickly, and I want to explain why. Bear with me.

What's hard about a judgment?

A warehouse runs on types. Every column has one, every downstream job trusts it, and a good part of data engineering is the business of making sure that a thing which claims to be a TIMESTAMP is in fact a TIMESTAMP. Judgments don't come with types. "Is this person a buyer or a practitioner?" is a question a human answers in half a second and a CASE WHEN answers badly in forty lines.

So over the years the industry has tried three things.

The first was the forty-line CASE WHEN. Somewhere in your warehouse there is one right now, cosplaying as a machine learning model. It matches '%vp%' (and catches the VP of Engineering along with every “MVP”). It was written years ago by someone who has since left, nobody will touch it, and it has a comment that says -- temporary.

The second was an actual, real life machine learning model. This works! It also needs labels, a training run, somewhere to serve it, and, well, a machine learning engineer. Most data teams can't spare that person for one column, which is why, a decade into "ML for everyone", most warehouses contain almost no ML.

The third is the one everybody reached for over the last three years: ask an LLM. It’s pretty ironic: data engineers spent a decade evangelizing that strings are not a type, and then LLMs showed up and returned nothing but strings. You ask for one of eight labels and you get a paragraph, or the label with a friendly parenthetical attached, or, on a bad day, NULL. It's also slow and expensive enough that you can't run it on every row, so you sample, and a sampled column isn't a column.

Price and trust are the same problem

Everyone’s caught onto this being useful; every major warehouse now ships “judgement” as a SQL function. Snowflake has AI_CLASSIFY, BigQuery has AI.CLASSIFY, Databricks has ai_classify. Researchers have a name for the whole family, semantic operators: filter, map, join and aggregate, by meaning instead of by value. Nobody is confused about whether people want a judgment in a column. The vendors settled that.

What they haven't settled is price and trust, and the two turn out to be the same problem.

We run one of these in production, a job that sorts about 250,000 job titles by persona and by function with AI_CLASSIFY. It works, it's one function call, and the data never leaves the warehouse, which matters a great deal. It costs 1.39 credits per million tokens (a Snowflake credit lists at $2 to $4 on demand), and the labels and rules are billed again on every row. In July a bug made the model return NULL for some labels, and a retry path helpfully sent every NULL back to the model, every half hour, in a dev environment. Two weeks. 44 billion tokens. About 61,600 credits. For the record, that is a job that sorts job titles, in dev. You can do the multiplication yourself. We did.

Today most of the code in that job has nothing to do with job titles. It's a cache, a rule for inputs the model refuses to answer, a test that fails the run when too many answers come back empty, and a breaker task that stops everything before it spends money. I'd bet that every team running an LLM in a pipeline has written some version of that code, or is about to find out why they should. All of it exists because the thing in the middle is expensive and can't be trusted with a type.

A model that judges quickly and efficiently

Last week TypeSafe released a model called Jev, and it goes the other way. It can't write. You give it text and a set of typed questions, and it gives back a bool, an enum or a score, each with a confidence, and that is the complete list of things it can do.¹ It's small and fast: about a fifth of a second a call in our runs, at $0.042 per million input tokens. Output is free, because there isn't any output to speak of.

Plenty of people had noticed that pipelines want decisions and not prose. TypeSafe is the team that went and trained a model for it, and I expect others to follow them, which is usually the sign that someone got it right. I've started calling the result a semantic column: the value is a judgment, and the type is still strict.

class LeadScore(BaseModel):
    persona: Literal["executive", "manager", "practitioner", "student", "other"]
    fit: Literal[1, 2, 3, 4, 5]

That’s pretty much it - this is what lands in the warehouse:

Look at the last row. A junk title still gets a valid value, because the type demands one, but its confidence is 0.41. Hold on to that number, because most of what follows is about it.

OK, but does it work?

I didn't want to write a whole post off somebody else's benchmark. So I took 4,000 of those job titles, gave Jev the same labels and the same rules we give Cortex, and hit run. It finished in 66 seconds and cost sixteen cents.

(The Cortex column is an estimate at the same token count, at list price. At that price, our two weeks in July would have cost about $1,900.)

The two agreed on 88% of personas and 73% of functions. Agreement doesn’t necessarily mean it was correct, because nobody ever promised that Cortex was right, so I just read the 44 cases where Jev was confident and the two disagreed, and held each against our own written rules. The rules sided with Jev in 41, with Cortex in 1, and 2 could go either way. Jev isn't perfect either. It sees the word "Chief" and starts handing out corner offices. Chief Data Scientist? Executive. A field CTO at a giant company? Executive. (TypeSafe if you could please put this in your next training run I’d be very appreciative.)

But the result I care about most is this one: When Jev said 0.99 or more, it agreed with Cortex 98% of the time. When it said under 0.50, it agreed 40% of the time. This means the confidence number gives you real signal about the decision, and you can use it accordingly (more on this below).

This isn’t a super rigorous benchmark per se, but it’s enough for me to speculate on where I think this goes.

Judgment becomes a column type

Here is how the persona column gets built today: someone files a ticket, the ticket waits for an ML engineer, and the ML engineer is busy until Q3. With a model like this, the person who owns the table writes the question, in plain words, and nobody labels or trains anything. When sales asks for a second column, the change is one line and a backfill that costs a few dollars.

 class LeadScore(BaseModel):
     persona: Literal["executive", "manager", "practitioner", "student", "other"]
     fit: Literal[1, 2, 3, 4, 5]
+    industry: Literal["software", "finance", "retail", "health", "other"]

Worth noting it’s not as straightforward as “just add a column with a couple values and Jev has suddenly, magically solved all your problems”. There’s still a lot of tuning you need to do, but Jev at least gives you the opportunity to do so without requiring training your own model, or depending on another team to train a model for you.

To illustrate this, as I was catching up on how people were already using Jev for data work (it’s been pretty crazy to see how quickly it’s been adopted), I came across an article from Southbridge, a startup working on data & AI problems. They used Jev to accelerate an entity resolution pipeline to join donors and committees across Ohio campaign finance data. Their first criterion was vague: "This specific relationship is supported between distinct compatible entities." Jev got 0 of 13 test questions right. They rewrote it to say what counts as enough proof: "Both records give the same residential street address (same house number and street), and the two are distinct persons” and suddenly Jev got 13 of 13. Writing a precise question, and keeping a labelled sample to test it against, is about to be a core data engineering skill.

Sampling goes away

Ask any data engineer how they'd run an LLM over a 2 million row table and you'll get the same answer: "I wouldn't. I'd sample 5,000 and extrapolate." That answer just expired. At these prices the habit stops making sense: a lead with its questions is about 200 tokens, so 2 million leads cost about $17. Keep your expectations sane, though. Across a whole system the saving is closer to 5 times than 500, because the hard cases still go to a bigger model as a second line of defense.

Data people worked this out within days. Within a week of the launch there were DuckDB extensions, and a SQLite library, jevsql, that puts jev_choice() in a SELECT. Hamilton Ulmer wrote that his DuckDB extension does "about 10sec for 1k rows". He called it "better than using an LLM, way more ergonomic than a classifier".

We’ve also deployed this to Astro and Apache Airflow as it’s the place people deploy data code (including DuckDB queries) to production. It adds a whole layer of orchestration on top of individual jobs, giving you scheduling, a pool that respects the rate limit, retries and backfills, amongst other things. It ends up being pretty simple:

@task(pool="jev_api", retries=3)
def score_leads(chunk: dict):
    hook = PydanticAIHook(llm_conn_id="jev_default")
    agent = hook.create_agent(output_type=LeadScore, instructions="Score this lead.")
    for row in read(chunk):
        stage(row.id, agent.run_sync(row.profile).output)

score_leads.expand(chunk=get_chunks())

Confidence becomes an interesting signal

Remember the 0.41 confidence number from above? Every answer Jev produces comes with one, which means you don't have to trust every answer the same amount. I think confidence ends up as a column next to every judgment, the way updated_at sits next to every row, and pipelines start to branch on it. High-confidence rows can go straight to the warehouse. The middle goes to an LLM for review. The rest goes to a person.

Code for the easy rows, a small model for the fuzzy rows, a person for the unsure ones.

This is also where orchestration has to grow up a little. A pipeline that makes decisions needs a bar, somewhere for the unsure cases to wait, and a record of who decided what. My colleague Kaxil Naik added the first version of that to Airflow's common.ai provider last week, and it works with any model the provider supports. A DecisionPolicy sets a minimum confidence, which I'll call the bar. An answer under the bar doesn't fail and doesn't guess. It opens a review form.

LLMBranchOperator(
    task_id="route_ticket",
    prompt=ticket_text,
    llm_conn_id="jev_default",
    model_id="typesafe:jev-1.13.0",
    branches={
        "billing_queue": "A question about an invoice, a charge or a refund.",
        "support_queue": "A product problem that needs an engineer.",
        "close_as_spam": BranchOption("Not a real request.", min_confidence=0.9),
    },
    decision_policy=DecisionPolicy(min_confidence=0.9, on_uncertain="review"),
)

I know what you're thinking: a confidence score is just a number the model made up. I thought so too, but I spent a bunch of time playing with Jev and it seems pretty reliable. I gave that task a ticket that mixes a billing question with a product problem. Jev leaned toward support, 0.63 to 0.37, with a confidence of 0.45. That is well under the bar, so Airflow stopped and asked.

Southbridge measured a split like this on their donor data. Jev did most of the work, and a second model ran a review stage. They report a cost 226 times lower than a frontier model alone, with accuracy within half a point of it. I expect that cascade to become the standard shape, with the frontier model as the specialist you call in, not the workhorse.

The jobs we gave up on come back

Some data jobs never got done because the only tool that worked was a person reading rows.

Entity resolution is the obvious one. Is "Acme Corp" the same company as "ACME Corporation, Inc."? SQL narrows the table to likely pairs, which SQL is good at, and the model judges each pair, which SQL is terrible at. High-confidence matches merge, and low-confidence pairs wait for review.

Watch the Umbrella pair. Its confidence is 0.56, so nothing merges and the pair waits for a person.

Documents are another. A model that can't write can't pull a clause out of a contract for you, but it can answer questions about one. Split the contract into sections, ask each section the same questions, and roll the answers up to one row per document.

Your scheduler gets a vote

Everything so far puts the typed answer in a table. The same kind of answer can also steer the pipeline itself, and I think this is the part people will be slowest to see.

Today if you set retries=5 on your DAG, it retries everything, and everything includes the PermissionError that is going to fail in exactly the same way five times, twenty minutes apart, while somebody waits for a table. You have watched this task. You knew on the first failure. You retried it anyway, because it was 6pm.

Airflow's ClassifierRetryPolicy asks a model one question when a task fails: what kind of failure is this? You own the table of answers so you can tune it to how you expect it to fail (and add to it as your DAGs surprise you). By default a network error retries in 10 seconds, a rate limit waits 60, and a permissions error fails at once. When the confidence is under the bar, plain rules decide, and either way the decision lands in retry_reason where you can read it later.

policy = ClassifierRetryPolicy(llm_conn_id="jev_default", min_confidence=0.6, fallback_rules=RULES)

@task(retries=5, retry_policy=policy)
def load_orders(): ...

This path runs on every task failure, which is exactly why it never made sense before. An LLM that takes eight seconds and costs a cent is a strange thing to put there. A typed answer that comes back in a fifth of a second is not.

Before you get too excited

A strict type is not the truth. A model like this can't return an invalid value, and it can absolutely return a valid one that is wrong. The confidence tells you how sure it is, never why. The only calibration evidence I have is the chart above, so set your bar from your own labelled sample, and keep that sample around. Jev is a week old, so test it on your own data before you trust it, the way I did. And the math, the dates, the joins and the writes all stay in code, where they belong.

Where I land

TypeSafe named the model after William Stanley Jevons, the economist who noticed in 1865 that more efficient steam engines made England burn more coal, not less. I think that is the right prediction. Nobody is going to take the three judgments in their pipeline today and make them cheaper. They're going to find the three hundred they never bothered to ask for.

TypeSafe deserves real credit here. They trained a model for the dull, enormous job the rest of the field walked past, priced it so you can run it on every row, and shipped something that held up on our data the first afternoon I tried it. It is a week old, and it is already the most interesting thing to happen to my corner of data engineering in a long while.

Will there be other models like it in a year? I'd bet on it. But TypeSafe got here first, they got the shape right, and I'm fairly sure it all starts with that column you promised sales.

Try it in Airflow

I’d be remiss if I didn’t talk about how we’re supporting Jev. Jev support ships in the upcoming version of the Airflow common.ai provider package.

pip install 'apache-airflow-providers-common-ai[typesafe]'

airflow connections add jev_default \
    --conn-type pydanticai \
    --conn-password "$TYPESAFE_API_KEY" \
    --conn-extra '{"model": "typesafe:jev-1.13.0"}'

The LLM operators, the branch operator and the retry policy all read that one connection.

• Classifier models guide, added in apache/airflow #73363
• Decision policy and the review flow: apache/airflow #73368
• Retry policies guide
• TypeSafe docs

We’re actively working on this integration! So if you have feedback on the shape, or try it and find something you feel is unintuitive, just reach out.

Also, we totally get that some teams cannot send data to a new vendor. For our customers, Astronomer's coming model gateway will offer Jev under a zero data retention agreement. TypeSafe won’t keep your prompts or the answers after the request completes. Hit me up if you want early access.

¹ Jev does not do math or dates, and it does not write text. Community reports put the median gain at about 7 times faster and 30 times cheaper than the LLM call it replaces. Both are below the launch figures. The numbers come from a community catalog.