The dataset is the asset

The code for a retrieval-based conversation-safety guardrail is nearly trivial. Embed a text window. Store the vectors. Compute similarity. Rank neighbours. Apply a threshold. A competent engineer writes it in an afternoon, and it will be the least interesting thing they build that week.

The hard part — the part that actually determines whether the system works, whether it can be published, and whether anyone should believe its numbers — is the dataset.

Not the size of the dataset. Its contract: what a case is allowed to contain, what it must contain, what can never appear in it, and what the evaluation is permitted to conclude.

The experiment that taught this

The first version was a small prototype guardrail with a deliberately boring architecture, built to test one narrow question: can cheap semantic retrieval catch conversations drifting toward known harmful trajectories, fast enough to run continuously in parallel with normal generation?

Three stages, presented in the demo walkthrough as three panels over each staged scenario:

1. A keyword gate — fast, and brittle. It passes a conversation that uses ordinary IT vocabulary while heading somewhere it shouldn't.
2. Semantic retrieval — a rolling conversation window scored against a store of sanitized cases, surfacing nearest risky and nearest benign neighbours. "Unauthorized reconnaissance disguised as inventory" sits next to "authorized asset inventory," and the distance between them is the signal.
3. An adjudicator — a slower, policy-aware second stage that decides whether the retrieved similarity actually means misuse, weighing authorization, real-world targets, stealth, persistence, credential collection, weaponization, and repeat attempts after refusal — then routes: allow, monitor, review, or block_or_route.

The latency argument for this shape holds up. Strong safety models are expensive and slow; a vector lookup is cheap enough to run continuously. It should not decide everything, but it can decide what deserves more attention.

Then an early check against a then-current frontier model (mid-2026) produced a result that reframed the whole project: the model already caught several of the sanitized risky scenarios on its own, without retrieval at all.

That is a correction worth publishing rather than burying. The point of a retrieval layer is not that frontier models are blind without vector search. They are not. The point is that retrieval makes the safety decision auditable, comparable, and dataset-driven — grounded in cases you own, can inspect, and can update without retraining anything. The evidence, not the confidence, is the product.

Which puts the entire weight of the system back onto the dataset.

What the contract requires

A public safety dataset has two failure modes, and they pull in opposite directions. Include too much operational detail and publishing it is irresponsible — you have written an instruction manual with a research label on it. Make it too vague and the detector is useless, because nothing in it has enough semantic shape to recognize.

The contract threads that: enough shape to recognize intent, never enough detail to execute it. Every case carries:

Severity is deliberately not a confidence score. It is a routing band: 1 benign or allowed, 2 monitor, 3 ambiguous dual-use, 4 high-risk review, 5 route or block. A number that maps to an action is auditable in a way that a number that maps to a feeling is not.

And the exclusion list is absolute. No working exploit chains. No code usable for malware, evasion, phishing, credential theft, or unauthorized access. No real organizations, domains, people, addresses, keys, or credentials. No instructions for bypassing safety systems. No hidden chain-of-thought extraction templates copied from real abuse traffic. No private user logs, ever.

What remains is abstract descriptions of intent and trajectory. That is not a watered-down version of the real dataset. It is the dataset — because trajectory is what the system is meant to detect, and operational detail was never the signal.

False-positive discipline is the real work

Here is the part that separates a safety dataset from a list of bad examples.

A store of risky cases alone will flag a penetration tester with a signed scope, an incident responder mid-triage, an engineer rotating CI secrets, a student learning defensive security, and an author writing a licensed benchmark. All of them look, semantically, quite a lot like the thing you're watching for. That is not a bug in the embedder. It is the actual structure of the domain: security work resembles security misuse, because it is about the same subject matter.

So the taxonomy carries benign classes as first-class citizens, not as an afterthought:

benign_defensive_security · benign_security_learning · benign_ctf_lab · benign_incident_response · benign_model_eval · benign_dataset_generation

In the current seed store those benign controls are 10 of 29 cases — roughly a third of the corpus spent on examples the system is supposed to let through. Every dual-use topic that gets a risky case gets a benign counterpart written with the same care, by the same rule: could a reader use this row to cause harm? If yes, rewrite it higher-level.

This is what makes the scoring work at all. A margin between the nearest risky and the nearest benign neighbour is only meaningful if the benign side is populated with genuinely hard negatives. Curate the risky cases carefully and the benign ones lazily, and you have built a detector whose real decision boundary is "does this text mention security."

Where the contract matured

The successor project, Risk Tree, is where this stopped being a design document and started producing measurements — and where the discipline paid off in a way that would have been invisible without it.

Its evaluation sets were built to the same contract: 30 mixed cases (12 hidden-intent, 6 obvious-risk, 8 authorized-defensive controls, 4 genuinely ambiguous), plus a 14-case stress set where every last turn was hand-checked to read as an innocent question containing no risk vocabulary at all.

Those 8 control cases are the reason the project's central result exists. Risk Tree forecasts a bounded tree of plausible next intents and scores the reachable states; when that look-ahead is aggregated by worst reachable branch, it flags 87.5% of the authorized-defensive controls. Probability-weighted aggregation brings that to 37.5%, all soft. Without a deliberately curated benign-control block in the evaluation, the worst-branch version would have shown 100% recall on the risky cases and looked like a triumph. The controls are what turned a flattering result into a real one.

The same contract also disciplined the scoring. Raw embedding similarity, measured against these sets, routed every case to review — 100% recall and 100% false positives — a failure only detectable because the benign side was populated well enough to expose it.

Honest scope

This line of work is early and in progress, and the numbers should be read with that in front of them.

The corpus is small: 29 seed cases, 30 + 14 evaluation cases, 10 staged demo scenarios. That is a portfolio-scale probe, not a production benchmark, and no percentage computed on 8 control cases should be treated as an estimate of anything. The natural next steps are the obvious ones — expand the sanitized corpus by two orders of magnitude, add adversarial paraphrases, measure false positives on a much larger body of legitimate defensive work, and add privacy-preserving retention rules — and none of them are done.

What we publish, therefore, is the contract and the method, not a production system: the schema, the taxonomy, the exclusion list, the generation prompt pack and its manual review checklist, the benign-control discipline, and an evaluation design built to surface its own failures. The classifier around it is replaceable, and probably should be replaced. The contract is the part worth keeping.

It is also the part that determines whether this kind of work can be shared at all. A safety dataset that cannot be published teaches nobody anything; a sanitized one can be inspected, criticized, extended, and disagreed with by people who will find its holes faster than we will. That trade — less operational fidelity in exchange for being publishable — is the one design decision here we would not revisit.

Similarity is evidence, not guilt. A dataset is a claim about what your system believes the world looks like. Writing it down explicitly, with its benign half taken as seriously as its risky half, is most of the safety engineering.