Atlassian twice shunned AWS Graviton CPUs, but now runs Jira and Confluence on them
Bills fell 10 percent after granular tests suggested JVM tweaks that improved performance
Atlassian twice marked Amazon Web Services’ Graviton CPUs off-limits for production purposes, but recently relented and now uses the processors to power thousands of server instances that run its Jira and Confluence products. So what changed?
A recent post by Atlassian principal site reliability engineers Paulo Almeida and Jakub Gutkowski, plus software engineer Jędrzej Lepa, reveals that Atlassian was interested in Amazon’s CPUs after the 2020 release of Graviton 2.
“In the tech industry there are only a couple of things that can make engineers try something new. Either it’s because we can do something faster or because we can do something cheaper,” the Atlassian trio wrote. “When only one of those two benefits presents itself, we often resort to long trade-off conversations because we not only have to factor in the immediate gains but also the invisible costs of implementation, support, tooling ecosystem, edge cases, and so on.”
Atlassian noted AWS’s claims that Graviton would be both faster and cheaper, which made the processors worthy of investigation – but its early tests did not support Amazon’s assertions.
“There have been previous investigations into the feasibility of migrating Jira to Graviton 2 and Graviton 3 at Atlassian,” the trio wrote. Those efforts produced results that showed Graviton processors were not superior, but didn’t unearth a smoking gun that showed exactly how Amazon’s CPUs slowed Atlassian’s code.
“From a performance engineer’s view, this is a problem,” Atlassian’s post states. “When issues appear at specific parts of our code, at least you know where to investigate next. But when problems span multiple endpoints and areas of your code, pinpointing a fix is like finding a needle in a haystack.”
Atlassian doesn’t have unlimited resources, so after its site reliability engineering team could not reach consensus about why Graviton wasn’t delivering, the company parked it.
“The sentiment for a while across the SRE team was ‘Graviton is off limits for now; should we ever want to try it again, we need an unquestionable and reproducible evidence that explains why Jira and Confluence couldn’t run before with the desired performance (and what makes it possible now)’,” the post explains.
In September 2024, AWS switched on the EC2 instances running Graviton 4 and again claimed they delivered superior performance. Atlassian decided to take another look and this time devised tests it hoped would yield more granular data about the performance of Amazon’s chips with its code.
That effort produced insights into the way Graviton processors use their L3 cache.
“This finding was valuable because it helped us move beyond the biased ‘Graviton-doesn’t-have-enough-horse-power’ knee-jerk reaction and pointed us to something more tangible, consistent and reproducible across multiple tests,” Atlassian’s authors wrote. It also suggested ways that Atlassian could optimize its use of the Java virtual machine (JVM) so that Graviton’s caching quirks didn’t negatively impact performance.
Those JVM tweaks produced improved performance.
“In theory all we would to have to do is to change the instance type to a Graviton-based equivalent and happy days, right?” the post states, before answering that question with an observation that when AWS launches a new instance type, the incidence of Insufficient Capacity Errors (ICE) increases.
“Atlassian has worked with the cloud long enough to realise that what might be an exception for smaller customers is common for us. ICE errors occur between 10,000 to 15,000 times per hour worldwide for Atlassian,” the post states.
The company therefore devised a scheme that sees it use EC2 instances powered by Graviton 4 processors for user-facing tasks and older Gravitons for other workloads, while ensuring workloads can fallback to x86 instances when required.
Atlassian has since migrated over 3,000 Jira and Confluence instances to Graviton, and after has chalked up “around 9.8 percent savings across the board, while improving the experience for our customers with ... meaningfully lower latencies.”
The post says Atlassian’s cost to operate some of its cloudy infrastructure has fallen 25 percent, but achieving that on all systems “will require a bit of reorganisation of our fleet to unlock.”
Atlassian users will probably hope the company gets that re-org done quickly, as the company hiked prices in October and has promised to start charging for its Rovio AI at an unspecified future point. ®