Structured Analytical Techniques (SAT) are a set of methods used in intelligence analysis to minimize cognitive shortcuts and biases. When working on cases like attribution analysis, geopolitical CTI or forecasting, applying these is a must for accurate analysis.

But as is also well known, applying these techniques requires people who are trained in SAT and who work through these methods as a group. Meaning analysis done this way takes a very long time and is costly. In the private sector, there is rarely enough time to devote to this kind of structured analysis.

To solve this problem, I wondered: could I build an analyst team made up of AI agents? If it worked, I would have an automation that could apply these time consuming, costly structured techniques on its own and produce analysis rather quickly. This post is about the path I took, the things I tried and my thoughts on why it didn’t work.

The idea was a pipeline of specialized agents, each owning a distinct role in the intelligence cycle. From intake all the way to finished reporting. Here's what I designed:

The analyst layer deserves a closer look. I wanted the agents to run through a structured reasoning loop, one that encodes the actual analytical steps that make SAT valuable in the first place:

1. Scenario generation

   Start by generating a broad list of possible scenarios using a creative SAT. Brainstorming without pre-filtering. The goal here is to avoid anchoring on the obvious explanation too early.

2. First refutation pass

   Go through the scenario list and eliminate any that the evidence already in hand directly contradicts. This is an active pruning step.

3. Collection gap identification

   For each scenario that survived, articulate what evidence or conditions would refute it. Then send that list back to the collection planner as new intelligence requirements. This circular loop intends to make the architecture more reliable.

4. Second refutation pass

   When the new collection comes back, revisit the remaining scenarios. Does any of the new information rule something out? If so, eliminate it. Repeat until no further elimination is possible.

5. ACH pass

   For surviving scenarios, flip the question: what in the evidence supports each one, and how strongly? This is Analysis of Competing Hypotheses; systematically evaluating how well the available information fits each remaining explanation.

6. Evidence weighting

   Rank the remaining scenarios by the combined quality of the evidence supporting each. This is where the trust scores assigned during collection finally come into play.

7. Key assumptions check

   Identify the baseline assumptions underpinning each scenario. If any of those assumptions turned out to be wrong, would the scenario collapse? If multiple scenarios share the same critical assumption, that assumption itself becomes a collection target. Loop back to the planner, collect new information and try to verify or refute it.

8. Scenario selection

   Converge on 2–3 non-overlapping, mutually alternative scenarios that have survived both refutation rounds and hold up under the assumptions check. These become the candidates for the final assessment.

9. Devil’s advocate

   Deliberately stress-test each of the remaining scenarios. What are its weakest points? Then apply alternative thinking: offer a plausible alternative explanation for each one that the evidence could also support. This is the check against overconfidence.

10. Final assessment

    Choose the most likely scenario based on evidence quality and resilience to assumption failure. Rank the others below it. Document supporting evidence, key weaknesses, and plausible alternatives for all of them.

This architecture attempts to operationalize the structured analysis process. The design was also made modular to make adding new SATs easy in the future. But, how did the user experience look like?

I tested it with the following question: “if China moves on Taiwan, what cyber operations should we expect, directed at whom, and with what objectives?”

The intake experience was exactly what I'd hoped for. Rather than dumping a requirement into a form, the tool opened a conversation. Two short exchanges and it had a focused requirement.

Looking at the session logs told a different story. I checked out the primary question and the sub-questions it had generated and the problems were visible immediately.

First, these questions are redundant and heavily overlapping. Good sub-questions should be mutually exclusive and collectively exhaustive: answer each one independently, and together they should give you everything needed to answer the primary. These don’t do that. They all require the same information to answer.

Second, all the questions are anticipatory. “what is China likely to do,” “what are the anticipated effects.” And that is the wrong question type for collection. You cannot answer anticipatory questions by collection. You might get a rough answer through analysis, by reasoning from known doctrine, history, and capabilities. But collection gathers facts about what exists, what has happened, what has been observed. Sending an anticipatory question to a collection planner is like asking a researcher to go find evidence for something that hasn’t happened yet. The framing is wrong at the source.

The collection planner's output was next. I'm not going to walk through every single one. Instead, you can go through them and try to spot the mistakes yourself. But let me point out the first few that immediately caught my eye.

First thing that jumps out is the “without specific attribution to China” instruction, which appears on several of the technical collection tasks. This prompt doesn’t meaningfully blind the collector; it just makes the phrasing awkward while leaving the direction of inquiry completely transparent.

The second problem is scope. The technical collection tasks ask for TTPs, malware and attack vectors associated with state-sponsored operations as a general category. No collection effort can meaningfully execute against a requirement that wide. Good collection tasks are specific enough that a researcher knows when they’ve found what they’re looking for and when they haven’t.

Now, let’s have a look at the final report together.

Looking at the session logs, the collected intelligence contained nothing about China, Taiwan, APT41, PLA doctrine or any of the actors and targets that appear in the final report. These came from the model’s own training data. Which is a problem, because training data is internet data. And the internet’s coverage of cyber warfare is dominated by the hype and FUD’s. Breathless threat reports, vendor marketing dressed as intelligence, apocalyptic scenario planning that gets clicks.

That’s the substrate the model is drawing from. So the scenarios read exactly like that: APT41 proxies activating on PLA SSF signal, simultaneous ransomware across Taiwanese banks and media, wiper attacks against Chunghwa Telecom’s submarine cables. All choreographed in neat phases with named actors, named targets, named objectives. Dramatically coherent. Completely ungrounded. Garbage in, garbage out. Just with a structured analytical framework wrapped around it.

This is a core problem with language models. You cannot reliably force a model to ground its judgements exclusively in the provided material. It will infer with the same confidence it cites real sources and there is no reliable mechanism to prevent that. The collection pipeline, the trust scores, the verbatim retrieval requirements; these are instructions to the model, not constraints on it.

I built something that looked like it should work, ran it against a real intelligence question, and got a report that was structurally correct and analytically hollow. The problems showed up at every stage: the sub-questions were wrong, the collection plan was flawed, the collected intelligence was thin, and the analyst agents filled the gaps with their own training data rather than stopping to say they didn’t have enough to work with.

But I want to be careful about how firmly I land this. There are too many variables I didn’t control for. Maybe my prompts were not optimized . Perhaps my scaffolding was not sufficient. Maybe my model of choice (gemini-flash) was incorrect. A more capable model running the same architecture might behave differently. Or it might not. I genuinely don’t know.

What I do think is that the core problems are hard in a way that better prompts alone won’t fix. But I could be wrong, and I’d rather find out than assume. I’d genuinely like to know what you find. Better prompts, a different model, a different scaffolding approach. If you get something that actually works, share it.

Thanks for reading