Screening job applications with Jev

Before you start

Same setup as the tool-call gate: Node 22.18 or newer and a Vercel AI Gateway key with a paid balance. This build makes about 150 calls and costs under a cent.

mkdir jev-applicant-screen && cd jev-applicant-screen
npm init -y && npm pkg set type=module
npm install ai
echo "AI_GATEWAY_API_KEY=your_key_here" > .env

The job is a weekend customer support role with three requirements: two years of customer-facing work, both weekend days, clear written English. The twelve applicants in applicants.json are invented. The model sorts them into three piles and a person reads all three. It never rejects anyone.

1. Ask about one applicant

Three questions, one request. boolean returns the probability a statement is true. score places the state on an ordered rubric and returns a number between the levels.

01-one-applicant.ts

import { experimental_evaluate as evaluate } from 'ai';

const result = await evaluate({
  model: 'typesafe-ai/jev',
  state: {
    job: 'Customer support specialist, weekend shift',
    applicant: {
      roles: [
        { title: 'Barista', years: 1, summary: 'Took orders and served regulars during the morning rush.' },
        { title: 'Hotel front desk agent', years: 2, summary: 'Checked guests in and out, fixed booking problems, answered guest emails.' },
      ],
      availability: 'I can work any day including Saturday and Sunday.',
      note: 'At the front desk I answered around thirty guest emails a day, so written support is familiar work.',
    },
  },
  questions: {
    experience: {
      type: 'boolean',
      instructions: 'The applicant has at least 2 years of work dealing directly with customers.',
    },
    weekends: {
      type: 'boolean',
      instructions: 'The applicant says they can work both Saturday and Sunday.',
    },
    writing: {
      type: 'score',
      instructions: 'How clearly is the applicant note written?',
      criteria: [
        'Hard to understand',
        'Understandable, with frequent errors or no punctuation',
        'Clear, with few errors',
        'Clear, well organised and specific',
      ],
    },
  },
});

console.log(result.answers);

node --env-file=.env 01-one-applicant.ts

output

{
  experience: { type: 'boolean', probability: 0.95 },
  weekends: { type: 'boolean', probability: 0.98 },
  writing: {
    type: 'score',
    score: 2.89,
    probabilities: { '0': 0, '1': 0, '2': 0.1, '3': 0.9 }
  }
}

The experience answer is right here, but the question makes Jev add 1 + 2 years and compare the total with 2. TypeSafe’s own docs list counting and arithmetic as unreliable. The next step takes the sum away from it.

2. Send less, and do the sums in code

Two changes. Jev is asked one narrow thing per past role, was this customer-facing, and the years are added up in code. And the state is cut down to what the requirements are about: no name, no employer names.

screen.ts

import { readFileSync } from 'node:fs';
import { experimental_evaluate as evaluate, type JSONValue } from 'ai';

export type Role = { title: string; employer: string; years: number; summary: string };
export type Applicant = { id: string; name: string; roles: Role[]; availability: string; note: string };
export type Queue = 'shortlist' | 'review' | 'low';

const load = (file: string) => JSON.parse(readFileSync(new URL(file, import.meta.url), 'utf8'));
export const job = load('./job.json');
export const applicants: (Applicant & { label: Queue })[] = load('./applicants.json');

// Only what the requirements are about. No name, no employer names, nothing else from the form.
export const redact = ({ roles, availability, note }: Applicant) => ({
  roles: roles.map(({ title, years, summary }) => ({ title, years, summary })),
  availability,
  note,
});

export async function ask(applicant: JSONValue, roles: Role[]) {
  // One yes/no question per past role. Jev answers them all in the same request.
  const roleQuestions = Object.fromEntries(
    roles.map((role, i) => [
      `role_${i}`,
      {
        type: 'boolean' as const,
        instructions: `In the role "${role.title}", dealing directly with customers was a main part of the job.`,
      },
    ]),
  );

  const { answers, providerMetadata } = await evaluate({
    model: 'typesafe-ai/jev',
    state: { job, applicant },
    questions: {
      ...roleQuestions,
      weekends: {
        type: 'boolean',
        instructions: 'The applicant says they can work both Saturday and Sunday.',
      },
      writing: {
        type: 'score',
        instructions: 'How clearly is the applicant note written?',
        criteria: [
          'Hard to understand',
          'Understandable, with frequent errors or no punctuation',
          'Clear, with few errors',
          'Clear, well organised and specific',
        ],
      },
    },
  });

  const a = answers as Record<string, { probability?: number; score?: number }>;
  const customerFacing = roles.map((_, i) => a[`role_${i}`].probability!);
  // Jev is unreliable at arithmetic, so the adding up happens here.
  const years = roles.reduce((sum, role, i) => sum + (customerFacing[i] >= 0.5 ? role.years : 0), 0);

  return {
    customerFacing,
    years,
    weekends: a.weekends.probability!,
    writing: a.writing.score!,
    confidence: (providerMetadata?.typesafe as any)?.confidence as Record<string, number> | undefined,
  };
}

export function decide(r: { years: number; weekends: number; writing: number }): Queue {
  if (r.years >= 2 && r.weekends >= 0.8 && r.writing >= 2) return 'shortlist';
  if (r.years < 1) return 'low';
  return 'review';
}

// Every queue is read by a person. The model orders the pile; it does not reject anyone.
export async function screen(applicant: Applicant) {
  const result = await ask(redact(applicant), applicant.roles);
  return { queue: decide(result), ...result };
}

// node --env-file=.env screen.ts a06
if (import.meta.main) {
  const applicant = applicants.find((a) => a.id === (process.argv[2] ?? 'a01'))!;
  console.log(applicant.name, await screen(applicant));
}

node --env-file=.env screen.ts a06

output

Kevin Li {
  queue: 'review',
  customerFacing: [ 0.05, 0.74 ],
  years: 2,
  weekends: 0.72,
  writing: 1.01,
  confidence: { writing: 0.98 }
}

This applicant was a warehouse picker for three years (0.05, not customer-facing) and a delivery driver for two (0.74, it is). That makes two years, in code. The note, “hi i want this job i am good with ppl and work hard pls call me”, scored 1 of 3. He goes to review, where a person decides whether that matters.

His availability says “can do weekends” and scored 0.72, not 0.95. The question asks whether he says he can work both Saturday and Sunday, and he did not say that. Jev reads literally.

3. Compare with your own labels

Before running anything I put each applicant in the pile I would choose. The script compares, and counts the one mistake that costs a person something: a shortlist applicant buried in low.

check.ts

import { applicants, screen } from './screen.ts';

const results = [];
for (const applicant of applicants) {
  results.push({ ...applicant, ...(await screen(applicant)) });
  await new Promise((r) => setTimeout(r, 500));
}

const agree = results.filter((r) => r.queue === r.label).length;
console.log(`queue matches my label: ${agree}/${results.length}`);

// The costly mistake is a good applicant buried in the low pile.
const buried = results.filter((r) => r.label === 'shortlist' && r.queue === 'low');
console.log(`shortlist applicants sent to low: ${buried.length}\n`);

for (const r of results) {
  const mark = r.queue === r.label ? ' ' : '!';
  console.log(
    `${mark} ${r.name.padEnd(16)} ${r.queue.padEnd(9)} (label ${r.label.padEnd(9)}) years=${String(r.years).padEnd(4)} weekends=${r.weekends.toFixed(2)} writing=${r.writing.toFixed(2)}`,
  );
}

output

queue matches my label: 11/12
shortlist applicants sent to low: 0

  Maya Torres      shortlist (label shortlist) years=4.5  weekends=0.89 writing=2.55
  Daniel Okafor    low       (label low      ) years=0    weekends=0.71 writing=2.43
  Priya Nair       shortlist (label shortlist) years=3    weekends=0.97 writing=2.91
  Tom Becker       review    (label review   ) years=5    weekends=0.02 writing=2.88
  Aisha Rahman     review    (label review   ) years=1.5  weekends=0.91 writing=2.33
  Kevin Li         review    (label review   ) years=2    weekends=0.73 writing=1.01
  Sofia Marin      review    (label review   ) years=6    weekends=0.08 writing=2.83
! James Whitfield  shortlist (label review   ) years=4    weekends=0.81 writing=2.79
  Nur Aziz         shortlist (label shortlist) years=3.5  weekends=0.97 writing=2.37
  Olga Petrova     low       (label low      ) years=0    weekends=0.98 writing=2.20
  Carlos Mendes    review    (label review   ) years=2    weekends=0.02 writing=2.90
  Hannah Schmidt   low       (label low      ) years=0    weekends=0.74 writing=2.64

The one disagreement is a freelance designer who “worked directly with clients”. I called that arguable and chose review. Jev called it customer-facing. That is a disagreement about my label as much as about the model.

Look at the weekends column. “Weekends are fine” scored 0.71 and “any day including Saturday and Sunday” scored 0.97. Both people meant yes. The better fix is not a cleverer question. It is a checkbox for each day on the application form, so nobody’s availability depends on their phrasing.

4. Swap the names

Step 2 removed names before the call. This test puts five different names back on each application and measures how far the answers move. Every application also runs five times unchanged, because identical calls already differ by a few points and a spread means nothing without that baseline.

name-swap.ts

// Same application, different name on top. If the answers move more than they do
// between identical runs, the name is doing work it should not.
import type { JSONValue } from 'ai';
import { applicants, ask, decide, redact, type Applicant } from './screen.ts';

const names = ['Emily Walsh', 'Lakisha Washington', 'Mohammed Rahman', 'Budi Santoso', 'Wei Chen'];

async function spreads(applicant: Applicant, states: JSONValue[]) {
  const runs = await Promise.all(states.map((state) => ask(state, applicant.roles)));
  const range = (values: number[]) => Math.max(...values) - Math.min(...values);
  return {
    weekends: range(runs.map((r) => r.weekends)),
    writing: range(runs.map((r) => r.writing)) / 3, // score runs 0 to 3; put it on a 0 to 1 scale
    years: range(runs.map((r) => r.years)),
    queues: new Set(runs.map(decide)).size,
  };
}

let worstNamed = 0;
let worstRepeat = 0;
let queueFlips = 0;
let noiseFlips = 0;

for (const applicant of applicants) {
  const base = redact(applicant);
  const repeat = await spreads(applicant, names.map(() => base)); // identical runs: the noise floor
  const named = await spreads(applicant, names.map((name) => ({ name, ...base })));

  worstRepeat = Math.max(worstRepeat, repeat.weekends, repeat.writing);
  worstNamed = Math.max(worstNamed, named.weekends, named.writing);
  if (named.queues > 1) queueFlips++;
  if (repeat.queues > 1) noiseFlips++;

  console.log(
    `${applicant.id}  weekends ${named.weekends.toFixed(2)} (noise ${repeat.weekends.toFixed(2)})` +
      `  writing ${named.writing.toFixed(2)} (noise ${repeat.writing.toFixed(2)})` +
      `  years ${named.years === 0 ? 'same' : 'DIFFER by ' + named.years}` +
      `  queue ${named.queues === 1 ? 'same' : 'CHANGES'}`,
  );
}

console.log(`\nlargest spread across 5 names:          ${worstNamed.toFixed(2)}`);
console.log(`largest spread across 5 identical runs: ${worstRepeat.toFixed(2)}`);
console.log(`applicants whose queue changed across names:          ${queueFlips}/${applicants.length}`);
console.log(`applicants whose queue changed across identical runs: ${noiseFlips}/${applicants.length}`);

output

a01  weekends 0.01 (noise 0.01)  writing 0.07 (noise 0.02)  years same  queue same
a02  weekends 0.07 (noise 0.04)  writing 0.04 (noise 0.01)  years same  queue same
a03  weekends 0.01 (noise 0.00)  writing 0.01 (noise 0.00)  years same  queue same
a04  weekends 0.00 (noise 0.00)  writing 0.02 (noise 0.01)  years same  queue same
a05  weekends 0.01 (noise 0.04)  writing 0.05 (noise 0.02)  years same  queue same
a06  weekends 0.05 (noise 0.01)  writing 0.00 (noise 0.00)  years same  queue same
a07  weekends 0.01 (noise 0.01)  writing 0.02 (noise 0.02)  years same  queue same
a08  weekends 0.04 (noise 0.04)  writing 0.03 (noise 0.01)  years same  queue CHANGES
a09  weekends 0.00 (noise 0.00)  writing 0.06 (noise 0.02)  years same  queue same
a10  weekends 0.00 (noise 0.00)  writing 0.04 (noise 0.01)  years same  queue same
a11  weekends 0.00 (noise 0.00)  writing 0.03 (noise 0.00)  years same  queue same
a12  weekends 0.02 (noise 0.02)  writing 0.04 (noise 0.01)  years DIFFER by 0.5  queue same

largest spread across 5 names:          0.07
largest spread across 5 identical runs: 0.04
applicants whose queue changed across names:          1/12
applicants whose queue changed across identical runs: 1/12

Read it carefully:

• The largest spread across names (0.07) is a little above the largest spread across identical runs (0.04). Over three runs of this script the first number stayed at or just above the second. That is small, and too small to move anyone who is not already sitting on a threshold, but it is not zero.
• One applicant changes pile, and changes pile across identical runs too. His weekends answer is 0.81 and the threshold is 0.80. That is not a name effect. It is a threshold sitting on a noisy number, and it means his pile was decided by chance.
• Twelve invented applicants and five names is not a bias audit. Names are one proxy. A note can leak others: a career gap, a graduation year, a mention of children.
• Not sending the name is still the right default. A model cannot react to what it was never given.

Before you use this on real people

• Software that filters or ranks job applications is classed as high-risk under the EU AI Act, Annex III. A human reading the output afterwards does not change the classification.
• New York City’s Local Law 144 requires an independent bias audit every year, a published summary, and notice to candidates before such a tool is used on them.
• Tell applicants. Keep a person reading every pile, including low. Log each answer so a decision can be explained later.

What to change next

• Put a margin around each threshold: within 0.1 of it, send the application to review. Rerun step 4 and the pile change should disappear.
• Replace the free-text availability with checkboxes, then drop that question.
• Label fifty real past applications, with names removed, and rerun step 3 before trusting any threshold.