Overview
While validating MariaDB RSS stability under stored procedure workloads, I ran into unexpected memory growth. The goal was straightforward: confirm that MariaDB was not leaking memory when running TPROC-C stored procedure workloads. To do this, I used HammerDB TPROC-C SP and added improved RSS monitoring so I could watch memory usage over long-duration runs.
The results were clear. MariaDB RSS stayed stable. HammerDB 5.0 kept growing, and growing.
This led to a deeper investigation into HammerDB itself, and ultimately to an update in TAF to support HammerDB 6.0 and a new RSS logging script.
MariaDB Foundation’s Test Automation Framework (TAF-Perl)
Improved RSS Monitoring
The original RSS script in TAF logged memory usage, but the output was ambiguous. It did not clearly identify the process being monitored, timestamps were minimal, and the formatting made long-duration analysis harder than it needed to be.
I replaced the older script with a more useful version, rssLogger.sh, which improves clarity and usability:
– It resolves and prints the process name (for example, mariadbd or hammerdbcli).
– It prints timestamped entries.
– It prints PID and process name on every line.
– It generates a clear, auto-named log file.
– It makes long-duration memory tracking readable and unambiguous.
Each process still requires its own instance of the script, but the improved output makes it obvious what you are looking at.
Example output line:
2024-08-13 11:22:01 pid: 12345 (mariadbd) RSS: 512.34 MB
With this improved logging, it became immediately obvious that MariaDB RSS stayed flat, while HammerDB 5.0 RSS steadily increased over time.
HammerDB 5.0 Memory Growth
During long-duration TPROC-C SP runs, HammerDB 5.0 showed continuous RSS growth. MariaDB stayed flat. HammerDB did not.
After discussing this with Steve Shaw, the author of HammerDB, he explained the root cause:
HammerDB 5.0 does not limit the number of samples collected by the xtprof time profiler. With no reservoir limit, memory usage grows continuously as virtual users run.
This behavior was fixed in HammerDB 6.0.
HammerDB 6.0 Fix
I pulled HammerDB 6.0 into my TAF clone and updated the default HammerDB client executable:
hammerdb_tprocc.client_executable=client_source/HammerDB-6.0/hammerdbcli
Then I reran the same workload.
The difference was immediate:
– HammerDB 5.0 RSS: continuous growth.
– HammerDB 6.0 RSS: tops out around ~740 MB and stays flat.
Steve explained why:
HammerDB 6.0 introduces a new setting called xt_reservoir. It controls the maximum number of samples collected per virtual user. The default is 10000 samples. If set to 0, HammerDB reverts to the old unlimited behavior seen in 5.0.
This is important for users:
– xt_reservoir=10000 gives controlled, stable memory usage as it limits sampling to the last X number.
– xt_reservoir=0 restores the 5.0 behavior and will cause memory to grow without bound as it collects all samples.
– Users who want additional samples, or legacy behavior must explicitly change this value in the XML file.
The setting is located in:
client_source/HammerDB-6.0/config/generic.xml
Inside the <timeprofile> block:
<xt_reservoir>10000</xt_reservoir>
This is the fix for the memory growth seen in HammerDB 5.0, and users should be aware that changing this value directly affects HammerDB’s memory footprint.
TAF Updates
Based on these findings, I updated TAF as follows:
1. Added HammerDB 6.0 to the clone.
2. Updated default HammerDB properties to point to 6.0.
3. Left HammerDB 5.0 in place for reproducibility until the next official TAF release.
4. Removed the old RSS script.
5. Added the improved RSS script under taf/scripts/memory/.
These changes improve stability and make memory behavior easier to analyze without breaking existing test suites.
Summary
MariaDB RSS remained stable under stored procedure workloads. HammerDB 5.0 did not, due to unlimited xtprof sampling. HammerDB 6.0 fixes this with the xt_reservoir setting. Setting xt_reservoir to 10000 limits sampling to the last X entries and keeps memory stable. Setting xt_reservoir to 0 restores the old 5.0 behavior and will cause memory to grow without bound as it collects all samples. TAF has been updated to include HammerDB 6.0, improved RSS monitoring, and clearer defaults, while keeping HammerDB 5.0 available for reproducibility until the next major release.
Jonathan (Jeb) Miller has experience in military leadership, Fire/EMS leadership, computer operations leadership, teaching, and more than 26 years of database performance engineering (PervasiveSQL, MySQL, MariaDB). TAF is the third benchmarking framework he has helped design and build.