Introduction

As most of our large consumer base already knows, Eclipse Adoptium is the community 'build, distribution and verification' project for the upstream OpenJDK project. We serve up millions of binaries per week, as can be seen in the download trends in our dashboard. We deliver these binaries in the most transparent manner. There is no 'mystery meat' in our recipe. In the spirit of such great transparency, this post serves to share our insights of the post mortem analysis of our most recent release activities.

We follow a strict regime of continuous improvement at the project, which means after each of our release cycles, conducting thorough retrospectives on what went well, what did not go well, and what actions are needed to improve in the future.

Goals and Metrics

All good retrospective activities start with "what were we trying to achieve"? During the July Critical Patch Update (CPU), we were targeting to deliver 42 different 'products', which is essentially the platform/version combinations of Eclipse Temurin that we build, test and distribute at the project.

Based on download numbers, we have divided these products into 'primary' and 'secondary' platforms. We focus our efforts on the primary platforms first to get these into the hands of our users as quickly as possible. Currently, x64 Linux, x64 Windows and x64 Mac are the leading downloaded platforms, so they are categorized as 'primary' platforms. The remaining platforms are categorized as 'secondary' platforms. Within the secondary platforms, some may even be considered 'best effort' platforms which as the name suggests are put at the bottom of the priority list.

To keep our goals simple, we aim to release the primary platforms within 2 days of the final OpenJDK source code being available. For secondary platforms, our target is to release within 7 days of the source code availability. These are summarized in a release status issue to communicate on-going status to the community. The July release activities were tracked in adoptium/issues/153.

For the original GA tags for July, you can find our 'scorecard' on how we did. 64.3% were released within the targeted timeframes, 37.7% were not released within targets. Of the ones that were not released within targets, 7 of them missed the targets by more than 5 days, which I will retrospectively refer to as "the Sickest Seven". For a detailed breakdown of these targets per platform and version, see adoptium/issues/157.

The Sickest Seven

As issue 157 describes, 7 of the 42 products released badly missed the target goals (by 5 or more days).

Let's look at the Sickest Seven, and determine next actions for improvement. From the initial July release activity, the Sickest Seven (7 products that took more than 5 business days over the 2 or 7 day target to publish), plus the two outliers from the respins:

What is encouraging about this list is that most of these issues have actionable responses, many of them completed during the release, as evident in our improved times for the respins.

New Features Need New Tests

A project that is changing, is a project that is going to occasionally break. This is will always be the case, but how do we pro-actively protect ourselves in the tug-of-war between adding new features and remaining stable and secure? The answer is of course 'testing'. While we have an ever-evolving suite of tests for verifying the OpenJDK binaries we produce via the AQAvit project, we additionally need to be better at adding tests to verify the changes we are making to our own build and distribution scripts.

Several changes went in ahead of the July CPU that introduced delays during the release. 2 of the changes are enhancements as part of our Secure Software Development Framework (SSDF) activities, GPG signing and the addition of SBOM artifacts. These features introduce new artifacts that can be downloaded alongside of the JDK binaries.

During the release, there were some initial missteps when not all of the new artifacts were published. While this was easily rectified, it prompted the team to add checks for ensuring we are publishing the expected set of artifacts.

A Windows compiler upgrade was also attempted prior to the release, there were unwanted side-effects, in this case a missing runtime library. These went unnoticed because we had no tests for verifying the full set to be produced, and other testing did not spot the missing libraries either, because our test machines have the compiler installed in order to compile native test material. The immediate action was to revert the change, until the core issue is addressed. To future-proof against similar problems, we will be adding smoke tests to be run in a stripped down environment.

Respins

We have covered the issues introduced in our repositories that caused minor delays, now let's look at changes in the upstream OpenJDK repositories that resulted in the need to respin every JDK version of this release, a costly activity.

Let's look at the last 2 years and the set of unique issues identifying the need for OpenJDK to respin in the table below.

For the JDK 8u345 respin, the issue described in JDK-8290832 points to its root cause JDK-8194154, which describes the crash that would occur when changing the "user.dir" property. While changing user.dir is discouraged, some applications still do it, notably Gradle. This is an interesting dilemma, where the feature is highly discouraged, but still possible, and therefore it is used in the field by some number of applications.

For the JDK 11.0.16.1, JDK 17.0.4.1 and JDK 18.0.2.1 respins, these could have potentially been found within the AQAvit testing if we had the Wildfly testing enabled. This highlights the fact that we need to be testing with a broad set of external applications, especially ones that are used widely. We are actively incorporating a set of tests in the AQAvit suite for this purpose in our external test category, but need to progress this work.

For the 2 of the 5 that were introduced by CVE patches, it would be good to understand what level of testing the OpenJDK Vulnerability Group (OJVG) does with these CVE patches ahead of releasing them into the OpenJDK codebase, and assess what needs to be changed in order to identify these issues before delivery to avoid the churn of a respin.

Since our Eclipse Foundation project is not a member of the OJVG, we do not have access to these patches ahead of the final code being available. As such, it was less likely that we would have been able to catch these failures, as we did not have weeks to soak these changes and see the intermittent issues. The first time we have a chance to build and test with these CVE patches is the day the final code is made available, which is Day 1 of our release period.

On Day 1 of our release period, we run 9 required top-level targets for AQAvit verification. For JDK-8249677, if we add our extended.external test target as a 10th required target, we may not have seen it, as the lucene-solr application tag is at releases/lucene-solr/8.11.1 and the issue was reported on 8.2.0. This is yet a third case where our external test group may be of great value once we graduate it into the set of required AQAvit test targets (note to self: time to push this activity up the priority list).

In any event, in order for us to have access to these patches earlier we would need to join the OJVG. While there has been an effort for Eclipse Foundation to join the OJVG, Oracle has denied the request. From the Adoptium Steering Committee Minutes 2022-August-25, Oracle have indicated that a reason for declining is that there is a perceived lack of contributions to OpenJDK from members of the Adoptium community.

Side note, in the August 25th Steering Committee meeting, it was pointed out that there is a list of contributions including those related to reproducible builds in the reproducible builds blog post. One could argue that the OJVG would benefit from having Eclipse Adoptium as an additional member that could assist in testing these patches more heavily and across a very broad range of platforms to boot.

In absence of membership, we will have to look at soak testing our release builds a bit longer, balancing the benefits against the increased time it would take us to release products.

Positive Takeaways

While most of this blog post has been spent analyzing what could be improved, it is important to remember that many things went well and how we have continuously improved our deliveries. This past release is no exception. Even during this release, we were able to improve the speed with which we delivered the respun products for late changes in OpenJDK.

We have continued to improve the way we communicate the progress of the release to the community, and the automation related to all of the steps required to deliver the Eclipse Temurin builds. Let's celebrate these good things. I look forward to further refinements outlined in adoptium/issues/155 and invite any and all to participate at the project to help develop and deliver them. One additional plug for support, if you want to help but have no spare time for project work, you can also directly sponsor the project. We typically focus these funds to help ensure we have sufficient machine resources to reduce bottlenecks in our pipelines.