Can Jev redact PII?

Jev scores 1.43× GPT-6 Luna’s F2 on PII masking.

24 Sept 2026 · 400 PIIMB sentences · Jev 1.13, GPT-6 Luna, GLiNER Multi PII

Mean masking F2

Loading measured scores…

Mean across four sources. Higher is better. F2 = 5PR / (4P + R), where P is precision and R is recall.

Background

Redacting personal data means catching names, emails, IDs, and other sensitive spans without blanking the whole document. We tested whether Jev’s yes/no judgments do that better than an LLM, a dedicated PII model, or a few regexes.

Methodology

We sampled 400 short English sentences from PIIMB, 100 from each of four sources. Jev marked words at probability ≥0.5, GPT-6 Luna returned character offsets, and GLiNER Multi PII ran on CPU at threshold 0.3. Regex and mask-all set cheap baselines, and no method saw the answer key. We scored character-level F2, which weights recall over precision, then averaged the four sources.

Results

Jev reached 78.1% mean F2, ahead of GLiNER at 70.3% and Luna at 54.6%. It led on three of four sources; GLiNER led on Privy, 58.6% to 51.1%. All 400 sentences cost about $0.015 in API usage with Jev and $0.017 with Luna. GLiNER had no API fee and ran on an existing CPU VM.

Discussion

The average hides a weak spot: on Privy, Jev’s precision was 24.4%, so it masked plenty of text it should have kept. F2 rewards recall heavily, so masking every word scores 56.0%, above Luna. Luna returned 19 invalid spans or JSON outputs, scored as misses. The sentences are short, English, and public, with possible training overlap, and each method used a different workflow.

Conclusion

Jev is worth testing as a cheap PII detector, especially on text like the three sources it led. Don’t ship unattended redaction on this score alone. Check misses and over-masking on your own documents before trusting any of these methods.

Data and run details

Loading progress…

Results

Source	Jev	GPT-6 Luna	GLiNER PII	Regex	Mask-all

Masking F2 by source. Higher is better.

Partial cells show n/100. Compare only when every cell reaches 100.

Precision and recall

Source	Jev P	Jev R	Luna P	Luna R	GLiNER P	GLiNER R

P is precision, R is recall.

Method

PIIMB (CC BY-NC 4.0): 100 short English sentences each from Ai4Privacy, Nemotron-PII, Gretel, and Privy. This is a pilot, not the full benchmark. Labels include dates, places, and demographics as well as names and IDs.

Jev picks whitespace-separated words at probability ≥0.5. Luna returns character offsets in JSON. GLiNER Multi PII v1 runs on CPU at threshold 0.3. All three get the same text and are scored against the same masks, but use different workflows. No method sees the answer key.

Regex masks emails, URLs, and long numeric strings. Mask-all shows the most recall word-level masking allows, not an F2 ceiling. We score character-level F2 and ignore label names.

Sampling, models, and cost

We hash sentence IDs with seed pii-jev-v1, skip duplicate IDs, and keep nonempty sentences of at most 240 characters and 32 words. Dataset revision 7d797b9fc8dc1942cef60fbe532e7d1a0e31b655; sample SHA-256 9230a9d372021759ed89b237f1cba9994cdee59c393e416b0c8c9332c76801ef.

Jev 1.13 asks one yes/no question per candidate word, all in one call. Luna uses the 22 Sept 2026 revision through OpenRouter Azure US, reasoning off, with PIIMB’s published field names. Two setup calls on broad Azure routing were excluded before we pinned the provider. GLiNER is Apache-2.0 and uses the same published labels. Jev, regex, and mask-all keep their original audited scores.

Jev: 364706 input tokens, estimated $0.0153. Luna: $0.0171 provider-reported API usage; failed calls may add unobserved cost. GLiNER: no API fee, CPU time on an existing VM.

Example

Synthetic sentence, not part of the evaluation. We sent it to Jev 1.13 with the Ai4Privacy scope.

Ava Chen emailed ava.chen@example.com.

Request:

const text = "Ava Chen emailed ava.chen@example.com.";
const scope = "Mask every annotated personal data field: names, title, age, gender, dates, city, address, phone, email, IDs, tax, passport, payment numbers.";
const spans = tokenize(text); // whitespace tokens, edge punctuation trimmed
const request = {
  model: "jev-1.13.0",
  state: { text },
  questions: Object.fromEntries(spans.map(({ start, end }, i) => [
    `s${i}`, { type: "noul",
      instructions: `In \`text\`, should characters ${start}-${end} (${JSON.stringify(text.slice(start, end))}) be masked? ${scope} Answer yes only if this span is part of a field to mask.`
    }
  ]))
};

Response:

{
  "model": "jev-1.13.0",
  "answers": {
    "s0": { "type": "noul", "noul": 0.88 }, // Ava
    "s1": { "type": "noul", "noul": 0.92 }, // Chen
    "s2": { "type": "noul", "noul": 0.02 }, // emailed
    "s3": { "type": "noul", "noul": 0.98 } // ava.chen@example.com
  },
  "usage": { "input_tokens": 568, "output_tokens": 72 }
}

Downloads

Aggregate results Audit

No sampled PIIMB sentences or per-case predictions are published.