I built an AI incident copilot that does not store your production logs
Every engineer has done some version of this:
- Something breaks in production.
- You grab the logs.
- You paste them into an AI chat app.
- You ask: “What is going on here?”
- The model gives you a useful answer.
- Everyone pretends this was normal.
It is not normal.
It is insane.
Production logs are not harmless text. They can contain customer IDs, stack traces, auth errors, request paths, internal service names, database fields, infrastructure details, feature flags, API responses, payment metadata, and occasionally secrets that should never have been logged in the first place.
And yet the fastest debugging workflow in 2026 is still:
Paste sensitive production context into a chat box and hope the privacy policy is friendly.
I wanted the AI help without turning incident response into a data leak waiting to happen.
So I built a small internal app for debugging incidents where the main product requirement was simple:
The app should be useful even if we refuse to store the user’s prompts.
The app: an incident copilot for messy production failures
The app is basically an AI “war room” for debugging.
You paste in logs, traces, errors, alerts, or incident notes. Then it helps with things like:
- Summarizing what changed
- Finding the likely failure point
- Grouping noisy logs into themes
- Explaining stack traces
- Suggesting rollback or mitigation steps
- Turning messy Slack updates into clean incident notes
- Drafting a postmortem timeline
The obvious way to build this would be:
User input → backend → database → LLM provider → database → UI
That is also the terrifying way to build it.
Because now my app owns a durable archive of every production failure someone pasted into it.
That means I have to worry about admin access, debug logs, database backups, support tooling, analytics, retention policies, deletion flows, and breach impact.
I did not want that.
I wanted the app to be closer to a private scratchpad than a SaaS dashboard.
The privacy rule: do not store the damn logs
This became the core design rule:
Incident data should be processed, not collected.
The app does not need to remember what happened last month unless the user explicitly exports it.
It does not need a searchable backend history.
It does not need to store raw logs “for analytics.”
It does not need to keep prompts so I can maybe build some future feature.
It just needs to help the user reason through the current mess.
So the architecture became:
- Chat history lives locally in the browser.
- The backend does not persist raw prompts.
- The backend does not persist raw model responses.
- Each request is treated as disposable.
- Users can export a postmortem if they want a durable record.
- Sensitive cleanup happens before the model call where possible.
- The UI is explicit about what is local and what is sent for inference.
This is not perfect privacy.
But it is a much better default than quietly storing everything forever.
Where the Icelake API fits in
For the model layer, I used the Icelake AI API because it gave me an OpenAI-compatible interface while fitting the privacy posture I wanted for the app.
That mattered because I did not want the model provider integration to become the whole project.
I wanted to keep the product architecture simple:
const response = await fetch("/api/analyze-incident", {
method: "POST",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({
messages,
redactionMode: "strict",
}),
});
Then the server endpoint does three things:
- Redacts obvious sensitive values.
- Sends the minimized prompt to the model API.
- Returns the answer without storing the raw request or response.
The important part is not that this is technically fancy.
It is not.
The important part is that the app is designed around data restraint.
Most AI apps start with:
What can we collect?
This app starts with:
What can we avoid collecting?
That one question changes a lot.
The basic architecture
Here is the rough flow:
Browser
|
| 1. User pastes logs
v
Local session state
|
| 2. Optional client-side redaction preview
v
Backend API route
|
| 3. Server-side redaction and prompt shaping
v
LLM API
|
| 4. Response returned
v
Browser
|
| 5. Local-only chat history
The backend is deliberately boring.
No incident_messages table.
No chat_history table.
No “save everything now, figure out privacy later.”
Just request in, request out.
Redaction is useful, but it is not a privacy strategy
A lot of developers treat redaction like a magic shield.
It is not.
Redaction helps, but it will never catch everything.
You can strip obvious things:
function redact(input: string) {
return input
.replace(/[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}/gi, "[EMAIL]")
.replace(/\b(?:\d[ -]*?){13,16}\b/g, "[CARD_NUMBER]")
.replace(/\bsk-[a-zA-Z0-9]{20,}\b/g, "[API_KEY]")
.replace(/\bAKIA[0-9A-Z]{16}\b/g, "[AWS_ACCESS_KEY]")
.replace(/\b\d{1,3}(?:\.\d{1,3}){3}\b/g, "[IP_ADDRESS]");
}
That is better than nothing.
But it is not enough.
Production data is messy. Sensitive values do not always look like secrets. Internal service names can be sensitive. Customer IDs can be sensitive. Database fields can be sensitive. Even a stack trace can reveal more than you intended.
So the real privacy win is not redaction.
The real privacy win is reducing how much data your app keeps after the request is done.
Redaction lowers risk during inference.
Not storing the logs lowers risk forever.
I also stopped pretending analytics needed message content
This is where a lot of AI apps quietly go wrong.
They say:
We need logs for product analytics.
No, you probably do not.
For this app, I care about events like:
- User ran an analysis
- User used strict redaction
- User copied a mitigation plan
- User exported a postmortem
- Model response failed
- Request took too long
I do not need the actual production logs.
I do not need the exact prompt.
I do not need the model’s full answer.
Product analytics should tell me how the product is used, not capture the private contents of the work.
So the event payload looks more like this:
track("incident_analysis_completed", {
redactionMode: "strict",
inputLengthBucket: "10k-50k",
latencyMs: duration,
model: selectedModel,
});
Not this:
track("incident_analysis_completed", {
rawLogs: logs,
prompt: fullPrompt,
response: modelResponse,
});
The second version is easier.
It is also reckless.
Local-first history made the UX better
I expected local-first history to be a compromise.
It ended up making the product feel better.
Users understood the mental model immediately:
This is your temporary incident workspace. It lives on your machine unless you export it.
That changed how people used it.
They were more willing to paste messy logs. They were more willing to think out loud. They were more willing to use it during real incidents instead of only sanitized demos.
Privacy was not just a compliance feature.
It improved the product.
Because the user did not feel like every half-baked debugging thought was being permanently filed away in my backend.
The annoying parts
Local-first is not free.
Some things got harder:
- Cross-device sync
- Shared incident rooms
- Long-term search
- Support debugging
- Recovery after clearing browser storage
- Collaboration during larger incidents
But those features should require explicit user intent.
If someone wants to create a shared incident room, fine. Store that room.
If someone wants to export a postmortem, fine. Save the postmortem.
If someone wants cloud sync, fine. Make it opt-in.
But do not use those edge cases as an excuse to store every private debugging session by default.
The default matters.
The principle I keep coming back to
AI makes it incredibly easy to build products that feel magical.
It also makes it incredibly easy to build products that collect horrifying amounts of sensitive context.
The dangerous part is that the data does not look scary at first.
It just looks like text.
But that text might be a production outage, a security issue, a customer escalation, a legal concern, a private business plan, or a personal question someone would never want sitting in your admin panel.
So my rule now is simple:
If the user would be uncomfortable seeing the prompt in your database, maybe it should not be in your database.
This sounds obvious.
But most AI apps violate it on day one.
The bigger lesson
The next wave of AI apps should not compete only on model quality.
They should compete on restraint.
Not just:
- How smart is the model?
- How fast is the response?
- How good is the UX?
But also:
- What do you refuse to store?
- What do you refuse to log?
- What do you make impossible for yourself to access?
- What does the user control?
- What disappears when the session ends?
For an incident copilot, this is not some abstract privacy ideal.
It is the difference between a useful engineering tool and a production data liability.
AI apps are going to sit directly on top of our most sensitive workflows.
Debugging. Legal. Finance. Health. Hiring. Strategy. Personal writing. Security.
If we build all of them with the default SaaS instinct of “store everything,” we are going to create a surveillance layer over work itself.
I do not want that.
I want AI tools that are useful precisely because they are designed not to remember everything.