This Month in Our Test Infra: December 2024

Happy new year, dear reader! This is the last This Month in Our Test Infra issue for 2024.

This is a quick summary of the changes in the test infrastructure for the rust-lang/rust repository¹ for December 2024.

As usual, if you encounter bugs or UX issues when using our test infrastructure, please file an issue. Bugs and papercuts can't be fixed if we don't know about them!

Thanks to everyone who contributed to our test infra!

Highlights

rustc-dev-guide is now a josh subtree!

Previously, rustc-dev-guide was a submodule inside rust-lang/rust, and updates to rustc-dev-guide had to be done in a separate PR against the rustc-dev-guide repository.

Now, thanks to @Kobzol's efforts (which included overcoming many unforeseen obstacles), rustc-dev-guide is now a josh subtree. This is a significant improvement for contribution workflows because it means that documentation updates to rustc-dev-guide can accompany the implementation change in rust-lang/rust in the same PR. The reduction in contribution friction also encourages rustc-dev-guide updates because you no longer have to author and maintain two separate PRs.

compiletest will now show the difference between normalized output and actual output for differing lines

Previously, it can be very difficult to tell when a ui test fails what the actual output is pre-normalization, as you would have to resort to --nocapture or similar.

Now, compiletest will also show the pre-normalization mismatched output lines on failure to make this easier to tell. Example output:

failures:

---- [ui] tests/ui/layout/enum.rs stdout ----
diff of stderr:

-	error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIGN }
+	error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN }
2	  --> $DIR/enum.rs:9:1
3	   |
4	LL | enum UninhabitedVariantAlign {

Note: some mismatched output was normalized before being compared
-	error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: Align(8 bytes) }
-	  --> /home/jyn/src/rust2/tests/ui/layout/enum.rs:9:1
+	error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN }

compiletest now allows using a specific debugger when running debuginfo tests

For a long time, the tests/debuginfo test suite could only be successfully run if you had all the tested debuggers (being lldb, gdb, cdb). This is very annoying locally if:

• One or more of these debuggers are not available or don't work locally².
• You just wanted to look at the test failures for a given debugger.

Now, you are able to run the tests/debuginfo test suite with a specific debugger, to only run the tests against that specified debugger. Example usage:

$ ./x test tests/debuginfo -- --debugger gdb

ui tests now support //@ forbid-output

ui tests can now use the //@ forbid-output: REGEX directive to check for a pattern which must not appear in the stderr. If the REGEX pattern is matched, then the ui test will fail.

Please consult rustc-dev-guide for more details.

./x test now accepts a --no-capture flag which will be forwarded to compiletest (and thus libtest)

Previously, if you wanted to pass the --nocapture flag through bootstrap, through compiletest, to the underlying libtest runner, on Linux you have to write:

$ ./x test tests/ui -- --nocapture

On native Windows msvc it's even worse, I recall having to write

PS> ./x test tests/ui -- -- --nocapture

Which is hard to discover and a contributor papercut³.

Now, you can just write

$ ./x test tests/ui --no-capture

and bootstrap will forward this flag as --nocapture to the underlying libtest.

Notable changes

This section is intended to be like a "compatibility notes" but for human test writers.

FileCheck-based test suites no longer predefine MSVC and NONMSVC FileCheck prefixes

In the current setup, compiletest will register a FileCheck custom prefix for each compiletest //@ revision. However, for historical reasons compiletest also predefined MSVC and NONMSVC FileCheck prefixes depending on the host. But this is very surprising for two reasons:

1. It's "spooky action in a distance" because the user never declared these custom prefixes, and these prefixes are conditionally set based on the host. It's hard to debug too.
2. If the user also wants to add their own //@ revision: MSVC NONMSVC revisions, because compiletest would pass --check-prefix for those names twice, this will cause FileCheck to report an error about duplicate prefixes.

Therefore, in compiletest: don't register predefined MSVC/NONMSVC FileCheck prefixes #134463 we no longer predefine these two FileCheck prefixes.

If you want the previous MSVC vs NONMSVC behavior, you will need to explicitly write out

//@ revisions: MSVC NONMSVC
//@[MSVC] only-windows-msvc
//@[NONMSVC] ignore-windows-msvc

normalize-{stderr,stdout}-test directives are renamed to normalize-{stderr,stdout}

Mostly a cleanup, the -test suffixes provide no additionally useful information, and only make these two normalize-* directives hard to discover.

normalize-{stderr,stdout}-test directives are now renamed to normalize-{stderr,stdout}. normalize-{stderr,stdout}-{32,64}bit directives remain unaffected.

compiletest will now deny usages of builtin FileCheck suffixes as revision names (for FileCheck-based test suites)

For FileCheck-based test suites (assembly, codegen, mir-opt), compiletest will register a custom FileCheck prefix for each compiletest //@ revision. However, FileCheck also has some builtin suffixes such as:

// COM:
// CHECK:
// CHECK-NEXT:
// CHECK-SAME:
// CHECK-EMPTY:
// CHECK-NOT:
// CHECK-COUNT:
// CHECK-DAG:
// CHECK-LABEL:

When combined, this previously meant that the compiletest revision + builtin FileCheck suffix constructions such as

// next:
// same-SAME:
// not-NOT:
// next-NEXT:
// not-SAME:

are permitted by compiletest, which are incredibly confusing.

As such, compiletest will now reject CHECK, COM, NEXT, SAME, EMPTY, NOT, COUNT, DAG, LABEL as revision names in FileCheck-based test suites.

PR listing

Improvements

• rustc-dev-guide: Turn rustc-dev-guide into a josh subtree #134907
• compiletest: Show the difference between the normalized output and the actual output for lines which didn't match #133733
• compiletest: Explain that UI tests are expected not to compile by default #133813
• compiletest: Allow using a specific debugger when running debuginfo tests #134629
• compiletest, run-make-support: Improve compiletest and run-make-support symlink handling
• compiletest: Support forbid-output in UI tests #134738
• bootstrap: Add --no-capture/--nocapture as bootstrap arguments #134809
• bootstrap: Allow ./x check compiletest #134848
• compiletest: Deny usage of special FileCheck suffixes as revision names #134925
• tidy: Run Python formatting check in tidy on CI #134964
• tidy: Print how to rebless Python formatting in tidy #134968

Fixes

• compiletest: Fix --nocapture for run-make tests #134111
• compiletest: Remove empty 'expected' files when blessing #134808
• run-make: Fix assert_stderr_not_contains_regex #134113⁴

Cleanups

• compiletest: Don't register predefined MSVC/NONMSVC FileCheck prefixes
• compiletest: Remove the -test suffix from normalize-* directives #134759
• compiletest: Only pass the post-colon value to parse_normalize_rule #134840
• compiletest: Slightly simplify the handling of debugger directive prefixes #134849
• bootstrap: Consolidate the macros for declaring compiletest test suites #134876
• tidy: Replace black with ruff in tidy #133821

Documentation updates

• Document how to run the split Docker pipelines #134894
• Document the --dev flag for src/ci/docker/run.sh #134669
• rustc-dev-guide: compiletest: Document the --debugger flag #2170
• rustc-dev-guide: Document forbid-output for UI tests #2171
• rustc-dev-guide: Remove the -test suffix from normalize directives #2172
• rustc-dev-guide: Document x test --no-capture #2174
• rustc-dev-guide: Describe how to use rust-analyzer with rmake.rs #2191