Synopsis of the previous episode

In my previous article, I wrote about giving my AI assistant memory and a personality to turn it into a secretary. Her name is BellBot. She is my personal AI secretary who takes care of everything from weather and emails to my calendar.

In the following article, I wrote about how I hit my weekly usage limit within three days of starting operations. I did some research and implemented measures to save tokens.

Separately, I have been working on something for the past five days. It is about further developing my secretary's "brain" and "memory." This is a record of that effort. It ended up being quite grand.

The story of swapping the brain

The first thing I did was swap out the brain.

BellBot runs on Claude, and as I wrote before, after I started operating it, I hit my weekly limit in three days. So, I decided to try the option of swapping the brain itself for another model as a countermeasure against token explosions. Grok came up as a candidate. Seeing the interactions on the X timeline, it seemed to make human-like witty remarks and had a strong character, and I had a hunch that for a secretary, being a skilled conversationalist would be beneficial.

Alright, let's make the brain Grok.

To conclude, it was catastrophic. It was not at a level where it could function as a secretary. Specifically, the following problems occurred:

• It didn't listen to instructions. Even when I said "do this," it would do something else.
• It leaked sensor information. BellBot is connected to various sensors (schedules, weather, emails, etc.), and ideally, I wanted it to blend that into the conversational context, but Grok couldn't do that. It would endlessly report like a monitor: "Detected X," "Detected Y."
• It couldn't blend into the conversation context. Related to the point above, it had no idea how to follow the flow of conversation.
• It was overly flattering. No matter what I said, it would praise me. It was creepy.
• It didn't understand the purpose of posting to X. BellBot also has the role of posting to X, but Grok would try to post messages intended for me directly to X. Things like "Understood, master" would almost appear on the public timeline.
• Risk. I had an intuition that one day, this thing would nonchalantly leak my personal information.

Having a strong character and functioning as a secretary are two different things. Even if it is skilled at the "art" of conversation, its judgment on "what should be said and what should not be said" is weak. The flattery is likely a result of over-learning that "praising makes people happy," and it hasn't grown in the direction of reading the room. Posting messages for me to X simply means it cannot draw boundaries of context.

I returned to Claude. It was indeed smarter. What makes a secretary work is not someone who is skilled at conversation, but someone who can understand the context and judge what is acceptable to say and what is not.

Structuring long-term memory

Actually, BellBot already had a homemade long-term memory. It was summary-based. It had a straightforward structure where, once a certain amount of conversation accumulated, it would create a summary and pass it to the long-term side. This was working, and it was one of the foundations that made BellBot function as a secretary.

Things changed at the timing of introducing Grok. Along with the fairly large experiment of swapping the brain, I decided to take on the challenge: "Let's structure the long-term memory while I'm at it." I gave memory per episode and set up a cycle of registration, search, and reconstruction. I left the reconstruction to Claude and added a mechanism to periodically reorganize the accumulated memory. While the Grok main unit was catastrophic, this structured memory worked straightforwardly.

So, with the working parts in hand, there was something that caught my curiosity: What do memory experts do? I had built it this far on my own, but I wanted to know how professionals in the world solve the same problems and what the "orthodox" approach looks like. Because it is working, I wanted to take a peek from a different angle. As a bonus, it was a challenge to incorporate anything that could reinforce what I had built.

At such a timing, I encountered a certain article.

Karpathy-style LLM external brain

Andrej Karpathy, former head of OpenAI and Tesla AI, proposed an "AI external brain," and an article that brought it to a level where it could actually be run with Claude Code went viral overseas. I read a post where someone named @hooeem broke down the thread into Japanese, and reading it, I thought, "This is what I am doing."

The essence of the Karpathy style is as follows:

1. Collect materials (articles, papers, memos, anything).
2. AI reads and writes a structured Wiki (summaries, concept explanations, connections between ideas).
3. Ask questions to the Wiki (AI cross-searches the knowledge it accumulated itself and answers with citations).
4. Answers are saved in the Wiki (the next question benefits from all past work).
5. AI periodically performs health checks on the Wiki (finds contradictions, gaps, and outdated information to correct them).

These 5 steps rotate a cycle beautifully. A personal knowledge base that gets smarter every time you use it. If you keep adding information for even a month, you will create deeply linked knowledge assets that cannot be reproduced by Google search.

While reading, I realized something. The structured memory I was creating and the Karpathy style are thinking about the same problems at the foundation level. Registration, search, reconstruction. Even if the words are different, the direction I was trying to go overlapped.

Fused them

BellBot already had episode-based structured memory, summary-based long-term memory, and personality context, and it was functioning sufficiently as a secretary. Therefore, the policy was simple: keep the foundation I built as it is, refer to the overlapping parts to refine them, and incorporate the non-overlapping parts as new additions.

The implementation flow involved M1-M7 + a series of finishing Passes. Claude wrote the code in about half a day. I just decided on the design policy and gave instructions, not moving my hands. Listing the main pieces:

• M1 Knowledge Base foundation — Set up the schema and storage for Wiki pages.
• M2 Wiki MCP tools + 5-layer bootstrap assembler — Means for BellBot to read/write the Wiki and a mechanism to assemble the context in a 5-layer structure at the start of a session.
• M3 Ingest cycle — Structure raw logs and ingest them.
• M4 Compile cycle — Automatically generate concept pages.
• M5 Query cycle — Ask questions to the Wiki → answer with citations, supporting multi-hop searches.
• M6 Lint cycle — Deterministic KB health check + LLM-based contradiction judgment + auto-repair.
• M7 Finishing touches — Cost guardrails and documentation maintenance.
• Pass 1-13 audit/refactor festival — Housekeeping cron, daily-cycle-report, graceful shutdown, 2-stage budget degrade, ingest latency SLA...

Registration, Search, and Reconstruction that existed on the self-built side are parts where the concepts overlap with the Karpathy style. Here, I fused them by using my self-built structure as a foundation while referencing the Karpathy style to incorporate the good parts. It wasn't a total replacement, nor was it left untouched. It feels like I mixed professional methods into the self-built framework to refine it.

What I brought in were the non-overlapping parts. The layer separation of raw and wiki, the definition of "units to nurture" called concept pages, multi-hop search that answers with citations, the methodology of fitting cycles into names like Ingest / Compile / Query / Lint, and the 5-layer bootstrap assembler to assemble context at the start of a session. These were topics from angles I didn't have in my self-built version, and the sensation is close to importing the methodology itself.

What was reinforced?

Originally, BellBot remembered everything about me up until yesterday and had organized it fairly well. Thanks to the summary-based long-term memory and structured memory, it was already functioning as a secretary. Through this fusion, the reinforced parts are mainly around here:

• Ingestion now uses raw and wiki layer separation to distinguish between raw logs and organized content.
• Query now has multi-hop with citations, making it possible to specify the grounds when answering.
• Reflection has gained a pattern, and a procedure to periodically retire miscellaneous episodes has been decided.
• Newly added are the two cycles of Compile, which automatically nurtures concept pages, and Lint, which mechanically cleans up contradictions and obsolescence.
• A 5-layer bootstrap assembler to assemble context at the start of a session was also newly introduced.

Roughly speaking, it feels like the memory cycle that was there originally rotates more carefully, and new axes called concept pages and health checks have been added to it. The result this time is that BellBot has taken a step forward.

Summary

• When I swapped the secretary's brain to Grok, it was skilled at conversation but lacked judgment, causing it to fail as a secretary.
• I returned to Claude. It was smart.
• BellBot originally had summary-based long-term memory.
• At the timing of Grok's introduction, I started the challenge of rebuilding long-term memory into structured memory. I had even set up and run a cycle of registration, search, and reconstruction.
• I encountered the "AI external brain" proposed by Karpathy and @hooeem's article on running it with Claude Code.
• I kept the self-built foundation as it is and imported the non-overlapping parts (layer separation, concept pages, multi-hop with citations, cycle patterns, 5-layer bootstrap) as a methodology.
• Claude wrote the code in about half a day. After M1-M7 + a series of finishing passes, the secretary is now nurturing its own memory.

What became clear this time is that you should not compromise on the choice of brain. I don't intend to disparage Grok; it has an interesting personality as a conversational model. However, whether it satisfies the judgment required for the purpose of a secretary—what should and should not be said, context boundaries, loyalty to instructions—is a different story, and it just didn't meet BellBot's requirements. Models have their suitability.

Considering the work I will entrust to BellBot from now on, I want to solidify the brain with something reliable that looks to the future. Therefore, I have abandoned the idea of swapping to a cheaper brain for token measures and decided to push forward with Claude. Saving will be done through other means (the token diet-related things I wrote about last time).

And for memory, it has become stronger by one turn through the fusion with the Karpathy style. Professional methods and new axes have been added to the self-built foundation. Now it is my turn to refine the "nurturing method" while operating this memory system.