Not utilising the server to the fullest is a sin
It all started when a senior AEM Developer, after seeing his pipeline fail for the 10th time straight and wasted over 15 man hours just for a single Pull Request, reached out to me angrily complaining about the Maven Build time being too much for a Pull Request this small.
Context: We use AEM with Maven for Build check bundled with React & solid.js. Total Build time was 95 minutes for each iteration. Along with this, SonarQube & GitHub Advanced Security Scanning stretched the total time to 120 minutes.
Hell, yeah! This is frustrating. Even for me, to see my servers being exploited like this, for 2 hours straight.
Well, my initial reaction was “How dare he shout at me like this, as if I created this process and I am wasting his 2 hours of time”.
But then kicked in my instincts and I started talking with my own thoughts, “I didn’t setup this process, but i surely can improve it for better right?”
I analysed
- Project architecture and components of Build
- Server utilisation — cpu, memory, disk
- real-time resource monitoring and performance checks
Conclusion was, “We were the culprits all along”. because we were barely using the server power for Build check, Testing, Linting and SonarQube scanning. There are roughly 12000+ tests, and they were running Synchronously one by one. Can you believe it?
My runner Server is of size: c5a.4xlarge (16 vCPUs, 32 GiB of memory) and being used only 5%.
We decided to tweak the configurations. And after rounds of experimentation aiming for the perfect recipe. I found the proportions which were perfect.
They were
- Defining the heap size limits: 1024 –> 4096 MB of memory
- Parallel builds scaled to 1.5× CPU cores — on a 16-core machine, that’s 24 concurrent build threads.
- Reusing the forks — helps accelerate the JVM resources reusability for each test set.
- Tiered Compilation — Runtime profiling for compiling and warm up. Asynchronous module compilation, defying the dependency.
Drum Rolllllssssssssssss
29 minutes!!!! Yes, this combination of configurations brought down the total build time to 29 minutes. A reduction of ~60 minutes. Crazy enough.
What started as frustration quickly evolved into something crazy that my entire developer team loves now.
Most of the times, the processes are fine, but the systems are not, because we are not utilising them properly.
Utilise, what you pay for!!
P.S. 8 of my Runner Servers died(crashed) during the entire testing process in search of the right recipe of configurations.