AI agents are making software development faster.
That is great.
But there is a problem I do not think we are talking about enough:
testing is not speeding up in the same way.
In many teams, testing is still held together by spreadsheets, meeting notes, screenshots, chat messages, and the memory of a few experienced QA engineers.
That worked when delivery was slower.
It becomes fragile when one developer can use multiple agents to change code across several modules in a single afternoon.
The bottleneck is no longer "can we write more test cases?"
The bottleneck is:
Can the team prove what was tested, why it was tested, what failed, what was fixed, and whether the release is safe?
That is the problem I built testboat for.
The Most Dangerous Sentence Before A Release
The sentence I worry about most is not:
We did not test this.
At least that is honest.
The dangerous sentence is:
I think we tested this.
That sentence usually means the team has test artifacts, but they are disconnected:
- requirements live in a doc
- test cases live in a spreadsheet
- automation scripts live somewhere in the repo
- execution results live in CI logs or chat
- bugs live in an issue tracker
- release reports are written manually before sign-off
Each piece may be useful on its own.
But when a Tech Lead asks, "Which requirements are not covered?" or a founder asks, "Can we release today?", the team has to reconstruct the answer manually.
That is not a testing process.
That is institutional memory under pressure.
AI Makes This Gap Worse
AI agents are very good at increasing throughput.
They can:
- implement a feature faster
- refactor code faster
- generate UI faster
- write automation faster
- fix bugs faster
But faster change creates more testing uncertainty.
If an agent changes the authentication module, what should be rerun?
If a test fails, is it a product bug, a flaky automation script, or an environment issue?
If a developer says "fixed", has the failed test actually been rerun?
If a release report says "main flows passed", where is the evidence?
Without a structured system, QA becomes the human buffer. Tech Leads become risk translators. Founders buy uncertainty with every release.
That is not sustainable.
Testing Needs To Become An Engineering System
testboat treats test artifacts like code.
It creates a .testboat/ directory in your project:
.testboat/
.active
draft/
strategy.yaml
tags.yaml
cases/
TC-001.yaml
bugs/
BUG-001.yaml
executions/
plans/
results/
execution-matrix.yaml
automate/
reports/
The important part is not "YAML is nice."
The important part is connection.
A requirement connects to a test case through req_id.
A test case connects to an execution plan.
An execution plan connects to an automation script.
A result connects back to the test case.
A bug can connect to both the test case and the failing result.
The latest execution state is summarized in an execution matrix.
Reports are generated from the same artifacts, not written from memory.
That changes the conversation.
Instead of asking:
Did we test login?
You can ask:
Show me every auth test case, its latest result, open bugs, and whether the release exit criteria passed.
What QA Gets
QA should not have to be the team's memory database.
With testboat, a test case is a structured file:
id: TC-001
title: Login with wrong password returns 401
status: ready
priority: P1
automation: to-automate
tags:
sprint: v1.0
type: functional
module: auth
req_id: STORY-001
steps:
- action: Enter wrong password
expected: API returns 401
expected_result: User sees a clear error message
It is diffable.
It is reviewable.
It has a state:
draft -> ready -> pass / fail / blocked / skipped
That means QA can maintain testing facts instead of constantly answering questions from memory.
What Tech Leads Get
Tech Leads need quality gates, not just good intentions.
testboat validate runs pre-report checks:
- format validation
- requirements coverage
- execution completeness
- exit criteria compliance
That last part matters.
Your strategy.yaml can define severity rules and exit criteria. For example, P0 and P1 bugs must be zero before release.
So the report is not just a nice HTML page.
It is generated after the system checks whether the release evidence is healthy enough.
This is the kind of thing that can eventually belong in CI.
What Founders And Managers Get
Founders do not need to read every test case.
But they do need release confidence.
"Main flows passed" is not enough.
The useful questions are:
- how many test cases exist?
- how many passed?
- how many were not executed?
- which requirements are uncovered?
- how many open P0/P1 bugs remain?
- did the release satisfy exit criteria?
- can we trace failures back to bugs and fixes?
testboat generates strategy, sprint, and closure reports from the actual test artifacts.
That gives leadership evidence instead of vibes.
Where AI Agents Fit
The goal is not to replace QA.
The goal is to give AI agents a testing workflow they can follow.
testboat enable creates agent-specific instructions for tools like Claude, Copilot, Cursor, Kiro, and others.
An agent can then follow a repeatable SOP:
- check the active test version
- read the strategy
- inspect registered tags
- create or update test cases
- validate test cases
- create execution plans
- run automation or guide manual testing
- record results
- file bugs on failures
- rerun affected tests after fixes
- validate before reporting
That is the difference between "AI wrote some tests" and "AI participated in the testing lifecycle."
A Small Example
If the auth module changed, you should not ask:
Can someone test login?
You should be able to do this:
testboat case list --module auth
testboat matrix show
Then rerun the affected tests and record results:
testboat result record TC-001 pass --type automated --by "AI"
If a failure appears:
testboat bug add \
--title "Wrong password returns 500 instead of 401" \
--tc TC-001 \
--severity major \
--priority P1
And after the fix, the bug should not jump straight to "closed."
It should move through retest:
fixed -> pending-retest -> verified -> closed
That is the loop teams need when development is moving faster.
Why I Think This Matters Now
AI is making code cheaper to produce.
That does not automatically make releases safer.
If anything, it makes weak testing systems more visible.
The next layer of AI engineering is not just faster code generation.
It is turning the surrounding engineering practices into systems that agents can participate in.
Testing is one of those practices.
That is why I built testboat.
Not to generate more test cases.
To make testing traceable, reviewable, versioned, validated, and reportable.
Try It
pip install testboat
testboat init
testboat enable cursor
testboat strategy create
Project:
https://github.com/lijma/testboat
Docs:
https://lijma.github.io/testboat/
Question
How does your team know a release is actually ready?
Is that answer stored in a system, or mostly in people's heads?