Researchers at North Carolina State University have developed a new AI-assisted tool that helps computer architects boost processor performance by improving memory management. The tool, called CacheMind, is the first computer architecture simulator capable of answering arbitrary, interactive questions about complex hardware-software interactions.

The new tool focuses on caches, which are hardware or software components in a system that store data the system may need to use again soon – the idea being that it is faster to retrieve data from the cache than it would be to retrieve the data from elsewhere in the hard drive. However, caches can only store a limited amount of data. Computer architects use two complimentary techniques to improve cache performance: prefetching improves performance by selectively pulling the data most likely to be used into the cache before it is needed; and cache replacement policies are algorithms that determine which data should be removed from the cache in order to bring in new data.

“Optimizing a cache replacement policy is challenging, because it can be difficult to determine which blocks of data will be used in the immediate future,” says Kaushal Mhapsekar, first author of a paper on the work and a Ph.D. student at NC State. “Doing this well requires having a good understanding of the fine-grained details about what is happening within the system, such as which instructions rely on data that is not in the cache.”

“Currently, computer architects use simulators to estimate how changes to a cache replacement policy will affect system performance,” says Azam Ghanbari, co-author of the paper and a Ph.D. student at NC State. “Outputs from these simulators are aggregated statistics about data-block use. However, these outputs miss those fine-grained details that are essential to identifying the best ways to improve cache replacement policy.”

Basically, current approaches to improving cache performance take a trial-and-error approach: run a simulation, look at the numbers, try a change to the prefetcher or replacement policy, run the simulation again, and then see if things got any better.

“A better approach is to analyze what is happening, identify patterns that could be improved, determine what is causing those patterns, and then implement a fix,” says Samira Mirbagher Ajorpaz, corresponding author of the paper and an assistant professor of electrical and computer engineering at NC State. “CacheMind was developed to assist with this – it uses causal reasoning, not trial and error, to improve memory management.

“Our goal was to develop a user-friendly tool that could help computer architects understand not only what is happening inside their processors, but why,” Mirbagher Ajorpaz says. “And it’s important to note that CacheMind enables arbitrary questions that assist with human reasoning, allowing AI to work alongside humans in CPU design. Building such a tool was challenging because conventional AI models train on Q&A to answer specific questions, not arbitrary ones.”

The end result is a “conversational tool” that allows architects to ask natural language questions like, “Why is the memory access associated with PC X causing more evictions?”

In proof-of-concept testing, CacheMind improved both cache hit rate and speedup across all test cases.

Because CacheMind is the first LLM-based tool designed specifically to address cache replacement policies, the researchers also created a benchmark that can be used to compare CacheMind’s performance to that of future models designed to perform the same task.

“We created CacheMindBench, which consists of 100 queries about cache replacement policies with verified answers,” says Bita Aslrousta, co-author of the paper and a Ph.D. student at NC State. “CacheMind is the first tool of its kind, but it will not be the last. CacheMindBench should be useful for tracking the performance of future developments in the field.”

“This paper is focused on cache replacement policies, which is the case study we used to demonstrate CacheMind’s potential,” Mirbagher Ajorpaz says. “But the applications of CacheMind and CacheMindBench extend to broader computer architecture questions.

“CacheMindBench is the first LLM reasoning benchmark in microarchitecture. Verified reasoning benchmarks are essential because they serve as examples given to LLMs, which enable context learning. The machine learning approach known as ‘few-shot learning’ allows LLMs to respond to arbitrary questions and become grounded. Our benchmark gives LLMs the context they need to mimic reasoning. And this enables them to perform human-like reasoning in fields they have not been pre-trained on. CacheMind works as plug and play on any new configuration, new question, or new software workload challenge without having to be trained on it.”

The peer-reviewed paper, “CacheMind: From Miss Rates to Why – Natural-Language, Trace-Grounded Reasoning for Cache Replacement,” was presented March 25 at the ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) in Pittsburgh, Penn.

-shipman-

Note to Editors: The study abstract follows.

“CacheMind: From Miss Rates to Why – Natural-Language, Trace-Grounded Reasoning for Cache Replacement”

Authors: Kaushal Mhapsekar, Azam Ghanbari, Bita Aslrousta and Samira Mirbagher-Ajorpaz, North Carolina State University

Presented: March 25 at the ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Pittsburgh, Penn.

Abstract: Cache replacement remains a challenging problem in CPU microarchitecture, often addressed using hand-crafted heuristics, limiting cache performance. Cache data analysis requires parsing millions of trace entries with manual filtering, making the process slow and non-interactive. To address this, we introduce CacheMind, a conversational tool that uses Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs) to enable semantic reasoning over cache traces. Architects can now ask natural language questions like, “Why is the memory access associated with PC X causing more evictions?”, and receive trace-grounded, human-readable answers linked to program semantics for the first time. To evaluate CacheMind, we present CacheMindBench, the first verified benchmark suite for LLM-based reasoning for the cache replacement problem. Using the SIEVE retriever, CacheMind achieves 66.67% on 75 unseen trace-grounded questions and 84.80% on 25 unseen policy-specific reasoning tasks; with RANGER, it achieves 89.33% and 64.80% on the same evaluations. Additionally, with RANGER, CacheMind achieves 100% accuracy on 4 out of 6 categories in the trace-grounded tier of CacheMindBench. Compared to LlamaIndex (10% retrieval success), SIEVE achieves 60% and RANGER achieves 90%, demonstrating that existing Retrieval-Augmented Generation (RAGs) are insufficient for precise, trace-grounded microarchitectural reasoning. We provided four concrete actionable insights derived using CacheMind, wherein bypassing use case improved cache hit rate by 7.66% and speedup by 2.04%, software fix use case gives speedup of 76%, and Mockingjay replacement policy use case gives speedup of 0.7%; showing the utility of CacheMind on non-trivial queries that require a natural-language interface.