At PlanetScale, our backend API is built with Ruby on Rails. It’s a pretty standard Rails application. We use minitest for our test suite and FactoryBot for creating test data.
Everyone on our team has worked in the past on Rails applications with slow test suites and knew how much it hurts team productivity. As our app has grown, we have continually invested time into keeping our test suite fast. We know how much a quick feedback cycle pays off for our team and a little extra work on it makes every feature we build easier.
Local development
We never run all of our application’s tests in local development. It’s not a good use of time and will never be as fast as running them on CI. When working locally, we’ll run the tests for the single file we modified, or just a single test at a time. Then we push the commit and get feedback for the whole test suite quickly.
Our whole test suite locally takes around 12 minutes running serially on a MacBook Pro. We haven’t put much effort here because it’s not something our engineers ever run.
Parallel Tests on CI
Rails now can run tests in parallel with minitest. If you’re using another test framework, various gems enable this as well.
This had the biggest impact and is also the easiest step to improve your test suite speed. When we initially started this, we began by running our tests in parallel on 2 workers. You’re limited by the number of cores the machine you’re running on has.
This gave us some speed gains, but we wanted it really fast. Our infrastructure team set us up with some 64 core machines on Buildkite.
# Only run in parallel on CI
if ENV["CI"]
parallelize(workers: 64)
end
After this change, our test suite ran in around 3-4 minutes. We clearly still had some issues to figure out. The next step was improving the tests themselves.
Auditing FactoryBot
After a bit of digging, we noticed most of our test time was spent setting up test data. We use FactoryBot for this.
We began investigating this by putting a debugger in our tests to stop execution right after the test setup. We used pry here to look around at all the objects created and see if they matched our expectations. We found a few surprising places where we were creating up to 8× as many objects as we thought we were.
This is a common mistake in FactoryBot. The library makes it so easy to set up relationships between data that it’s possible to trigger the creation of more associated objects than you expect.
Fixing our Factories
Solving this was more straightforward once we knew the problem. We set up tests with our expectations for the amount of data our factories should create.
test "factory doesn’t create tons of databases" do
create(:database)
assert_equal 1, Database.count
end
These tests failed at first, but we worked through the factories and eventually got them down to creating the correct number of objects.
This gave us another huge gain, and after a few of these changes, the test run time dropped to ~1 minute.
We keep these tests in our models, protecting us from any regressions when making changes to our factories.