Over a month ago, we discussed a possible migration to the Node.js test runner. While we were sufficiently happy with Mocha, we are always looking to make our CI jobs faster.

Relying on a test runner baked inside our runtime had some advantages for our main monorepo:

• Two fewer dependencies to install and maintain in our monorepo: mocha and chai.
• Maintainability: there are more people involved in the Node.js project to maintain the Node.js test runner.
• Future benefits: we believe the test runner will improve with time, and eventually save some time in our CI workflows.

From an idea to a PoC

The Astro monorepo has more than 500 testing suites: between integration tests and unit tests, we have 664 suites, with a total of 1603 tests. The majority of these tests are integration tests.

An integration test, in our monorepo, means creating a tiny Astro project, building this project with a specific environment (development, static generation (SSG) or dynamic generation (SSR)), and then running assertions over the built pages. That’s right, each integration test requires vite to build and bundle the project.

Before deciding to undertake this migration, we wanted to make sure that moving away from Mocha was not a mistake. Despite its quirks, Mocha is a very capable test runner! It’s been around for a long time and is battle-tested. If you use Mocha, you are in good hands.

The idea of the PoC was to understand:

• The flexibility of the Node.js CLI arguments and how customizable the test reporters could be.
• The speed of execution of the testing suites.
• The overall developer experience.

How we started

We started by migrating only one of our packages that didn’t already use astro’s integration suite: create-astro. This was a good opportunity to play with the built-in assertion library node:assert, to learn about the options it offered, and evaluate its performance compared to Mocha.

Since create-astro only had a handful of tests, it was relatively easy to migrate the test files to use node:test and node:assert instead of mocha and chai. After that, the only thing left was to update the mocha command to node --test to execute the tests. However, we quickly ran into issues using the node --test command, including:

• It had trouble parsing the glob syntax when passing multiple arguments (e.g. node --test "test/*.test.js" --test-name-pattern="foo").
• It wasn’t possible to pass the --test-concurrency flag (only available in Node.js 21 and above), but could be worked around by using the programmatic API concurrency option.
• Nitpicking, but the argument names were verbose: --test-name-pattern instead of --match, -m arguments; --test-timeout instead of --timeout, -t arguments, etc.

Hence, to solve these issues, we created a custom script which can be called with the astro-scripts test command. This decision also proved useful in order to enable more workarounds as you’ll see later.

Opening Pandora’s box

After successfully migrating the first package, we then attempted to migrate the testing suites of the @astrojs/node package. This integration is one of our most downloaded integrations, so we have plenty of tests. Plus, the tests of this package all have integration tests, so it was a good opportunity to check the performance of the test runner.

Once the PR was ready, we noticed that Node.js test runner was significantly slower than Mocha. We investigated, and we discovered that Node.js spawns a new process for each test file to ensure that each testing suite is run in isolation. Running a testing suite in isolation is, generally, a good practice because it assures that tests run in an unpolluted environment.

However, our testing suites were already isolated. In fact, we were able to run our testing suites with Mocha using the main thread without running into any of the typical issues: side effects, polluted environments, etc. Unfortunately, Node.js didn’t provide an option to run all tests in the same thread. So, we had to come up with a solution to the slow test runs. (Aren’t we engineers, after all? We solve problems!)

Using our internal astro-scripts test command, we were able to work around this by creating a temporary file that imports all the testing suites, and we let Node.js test that single file. This way, only one process is spawned for the file and we reach the same level of performance as if we were using the main process.

However, this came with its own downside: if there was a test failure or a timeout, we weren’t able to tell which test was the cause. This was the main quirk we found, and while not every team may have made this choice, we accepted this trade-off in order to give us the benefits mentioned earlier. After all, we had previously accepted Mocha’s quirks!

Node.js assert and chai

During the migration, we had to remove the chai library for node:assert/strict. This task uncovered that with chai, you can execute the same check in different ways. For example, you can run an equality check at least in four different ways:

import { expect } from "chai";
expect("foo").to.eq("foo")
expect("foo").to.be.eq("foo")
expect("foo").to.equal("foo")
expect("foo").to.be.equal("foo")

On the one hand, it’s good to have this kind of flexibility. But on the other, the code for the tests becomes inconsistent. With the Node.js assertion module, there is only one way to perform this check:

import { assert } from "node:assert/strict";
assert.equal("foo", "foo")

The Node.js assertion module provides almost all of the functionalities we required, so the migration from chai wasn’t as painful as we thought it might be. Our use of chai was very minimal. However, we do miss the .includes shortcut of chai:

import { expect } from "chai";
expect("It's a fine day").includes("fine")

The Node.js assertion module doesn’t provide such a utility, so instead we ended up using the equality assertion with the String#includes function:

import assert from "node:assert/strict";
assert.equal("It's a fine day".includes("fine"), true)

Here come the dragons

As mentioned before, we have a lot of test files and we add new tests almost every day. Opening a one-off PR that does the migration of the whole monorepo is unfeasible. It would require a lot of work from one person, and keeping the branch updated would be stressful.

So we came up with a simple plan:

• Migrate first the small packages inside the monorepo.
• Slowly migrate the main package - astro - by having Mocha and Node.js test runner in the same CI.
• Remove Mocha.

In order to achieve that, we asked for help from our community. We thought this was the perfect opportunity to let people that aren’t familiar with Astro’s business logic contribute to the project. And, we could make the migration process go more quickly.

We created and pinned an umbrella issue to coordinate the efforts. Each contributor took ownership of the migration of an individual package, opening a separate PR for each one. Two new first-time contributors to the project even joined the efforts. It was a fantastic thing to see. In one week, we were able to migrate all packages!

Migrating the main package astro was a feat! It’s the package that contains by far the highest number of tests. In order to perform this migration slowly and carefully, we had to come up with an out-of-the-box solution.

We set up the Node.js test runner to test only the files called *.nodetest.js. Doing so allowed us to keep testing all files in the CI. Then, the rest was just a matter of coordinating our community by providing (and documenting!) a clear process for them to follow:

• Use the umbrella issue to tell other contributors which files you intend to migrate;
• Rename the files to migrate from *.test.js to *.nodetest.js;
• Migrate the files;
• Open a PR, wait for a review, and if successful, an Astro maintainer will merge the PR.

With the help of @log101, @mingjunlu, @VoxelMC, @alexnguyennz, @xMohamd, @shoaibkh4n, @marwan-mohamed12, @at-the-vr and @ktym4a we migrated almost 300 test suites in one week!

The results

We are quite happy with the results. We haven’t seen any significant regression in the performance of our tests. The assertion module that Node.js provides has all the utilities we need, and the describe/it pattern is supported, so the migration from Mocha was smooth.

There are, however, still a few hiccups regarding the developer experience compared to using Mocha.

For example, to run one single test suite in Mocha, using it.only is enough. With the Node.js test runner you have to:

• Run the CLI using the --test-only argument.
• Add .only to the describe that contains the it.only you want to run.
• If there are multiple instances of describe, all of them need to be marked with .only.

Another example is that using --test-name-patterns could be improved. This argument is used to run only the tests that match a particular name pattern. The DX isn’t great because the CLI litters the terminal with messages about tests that don’t match (which aren’t run). This makes it more difficult to understand which tests are actually run. Plus, the command is really slow just for running tests that match some pattern.

Node.js test runner is still young and with its active development, has all the right cards to become better. For example, the Node.js project is currently evaluating running tests using the main process after we voiced our use case.

In the spirit of true open-source collaboration, we are pleased that improving Astro by switching our tests to Node.js will, in turn, improve Node.js itself!