Zero-Shot Classification for Intelligence Triage: Getting Useful Signal Without Labeled Training Data

Labeled training data is the quiet tax that slows every intelligence ML project. You need examples, and those examples need to come from analysts who are already overloaded. The result is a familiar stalemate: the ML team waits on labels, the analysts wait on tools, and the queue grows.

Photo by Google DeepMind on Pexels.

Zero-shot classification offers a way around that bottleneck. The technique lets a model categorize text into predefined classes without ever having seen labeled examples of those classes during training. Feed it a raw intelligence report and a set of candidate labels, and the model returns probability scores for each. No fine-tuning required. No six-week labeling sprint.

The question worth asking is not whether this works in a demo. It clearly does. The question is where it holds up under operational pressure and where it quietly fails.

How the Underlying Approach Works

Most production zero-shot classifiers are built on natural language inference (NLI) models trained on large entailment datasets. The model is asked, in effect: does this text entail the hypothesis that it belongs to class X? A document about port infrastructure and a hypothesis label like "maritime logistics disruption" either entail each other or they don't. The model scores that relationship.

Larger models like cross-encoder variants of DeBERTa, or zero-shot endpoints exposed through APIs, tend to outperform smaller BART-based classifiers on dense intelligence text. The domain specificity of intelligence reporting matters here: jargon, abbreviations, fragmented sourcing notes, and passive constructions all degrade performance relative to clean news text.

A rough pipeline for an operational triage system looks like this:

graph TD
    A[/Raw Intelligence Report/] --> B(Preprocessing & Chunking)
    B --> C{Zero-Shot Classifier}
    C --> D[Priority Label + Score]
    C --> E[Secondary Labels]
    D --> F(Analyst Queue Router)
    E --> F
    F --> G[Human Review]

Chunking matters more than most practitioners expect. A six-page cable contains multiple distinct topics, and classifying the document as a whole averages out signals that should be routed separately. Splitting on paragraph boundaries and scoring each chunk independently, then aggregating with a max-pool or attention-weighted scheme, consistently outperforms document-level classification on heterogeneous reporting.

What Breaks in Practice

Label design is where most zero-shot deployments quietly fall apart. Analysts will specify labels that feel intuitive but are semantically overlapping: "WMD proliferation," "dual-use technology transfer," and "sanctions evasion" share enough language surface area that probability mass distributes erratically across them. The model isn't wrong; the label space is underspecified.

The fix is deliberate label engineering. Each label should be converted into a short descriptive hypothesis: instead of the tag "WMD proliferation," use "this document describes the transfer or acquisition of materials usable in weapons of mass destruction." Verbose labels consistently outperform short tags on NLI-based classifiers. Analysts resist writing them. Make it mandatory anyway.

Confidence calibration is the second failure mode. Raw probability scores from zero-shot classifiers are not well-calibrated in the statistical sense. A score of 0.82 on "terrorist financing" does not mean the model is 82% confident. Temperature scaling or isotonic regression on a small held-out set (even 200 examples) dramatically improves score reliability and makes routing thresholds defensible to oversight bodies.

The third problem is adversarial language. Threat actors who understand classification systems will use circumlocution, code language, or deliberately fragmented text to suppress topic scores. Zero-shot models trained on standard corpora have no exposure to this kind of deliberate obfuscation. The failure mode is invisible: the document simply scores low across all categories and falls out of priority routing. Monitoring for low-confidence documents as a separate queue, rather than discarding them, catches most of this.

Where Zero-Shot Classification Actually Earns Its Place

The strongest use case is new collection streams where no labeled data exists yet. When a new source comes online, zero-shot classification can provide immediate triage while labeled data accumulates in parallel. After three to four weeks of analyst-validated outputs, that data feeds a fine-tuned classifier that will outperform the zero-shot approach on that specific domain.

Think of it as a warm-start mechanism, not a permanent solution. The goal is to make the labeling process itself more efficient: analysts review pre-classified documents rather than blank reports, and their corrections become training signal.

For multilingual reporting, zero-shot models based on multilingual NLI checkpoints (XLM-RoBERTa derivatives, for example) extend triage to languages where labeled intelligence data is essentially nonexistent. Performance degrades against English baselines, but triage-level discrimination between "likely relevant" and "likely irrelevant" holds across most major languages with operational intelligence volume.

Zero-shot classification will not replace a well-trained domain-specific model. What it does is compress the time between "we have a new collection requirement" and "analysts are seeing prioritized, routed reporting" from months to days. In intelligence operations, that compression has real consequences.