This Month in Our Test Infra: October 2024

This is a quick summary of the changes in the test infrastructure for the rust-lang/rust repository¹ for October 2024². It also includes brief descriptions of on-going work.

As usual, if you encounter bugs or UX issues when using our test infrastructure, please file an issue. Bugs and papercuts can't be fixed if we don't know about them!

Thanks to everyone who contributed to our test infra!

Highlights

compiletest now supports bringing your own custom diff tool

compiletest (and bootstrap) now supports bringing your own custom diff tool (#131181).

This only affects the visual diff generation (i.e. maybe you like a different visual style). The .stderr snapshots and such are not affected.

If you want to use your favorite diff tool for generating the visual diffs, you can modify config.toml's build.compiletest-diff-tool option:

[build]
# What custom diff tool to use for displaying compiletest tests.
#compiletest-diff-tool = <none>

Thanks to @dev-ardi for the implementation!

minicore test auxiliary and //@ add-core-stubs directive

ui, assembly and codegen tests can now use the //@ add-core-stubs directive to conditionally build a minicore test auxiliary which provides core stubs (#130693). This is so that we can share core stubs between cross-compiling tests that only need to build for the cross-compile target and avoid having to reinvent and maintain duplicate minicore copies in each of such test³.

Previously, having to reinvent a minicore every time you want to add a distinct ui/assembly/codegen test (for checking e.g. cross-compile ABI) is a significant source of contributor friction and makes it more prone to introduce mistakes in the minicore copies. This is also particularly annoying for compiler contributors who want to introduce new lang items, as they found themselves having to update multiple copies of such core stubs.

Note that currently, the shared tests/auxiliary/minicore.rs test auxiliary is still quite minimal. The plan is to land the test infrastructure first, then we can incrementally add more core stubs to the shared test auxiliary.

Example usage:

// tests/ui/abi/my-abi-test.rs

//@ add-core-stubs
//@ compile-flags: --target x86_64-pc-windows-msvc
//@ needs-llvm-components: x86

#![crate_type = "lib"]
#![feature(no_core)]
#![no_std]
#![no_core]

extern crate minicore;
use minicore::*;

struct Meow;
impl Copy for Meow {} // `Copy` is provided by `minicore`!

See the context issue, MCP and tracking issue for more info on the original motivations. See the rustc-dev-guide chapter for example usage.

PR listing

Improvements

• General test infra: Add minicore test auxiliary and support //@ add-core-stubs directive in ui/assembly/codegen tests #130693
• compiletest: Add test infra to explicitly test rustc with autodiff/enzyme disabled #131470
• compiletest: Special case error message for a build-fail test that failed check build #131642
• compiletest: Document various parts of compiletest's lib.rs #131679
• compiletest: Fix unnecessary nesting in run-make test output directories #131764
• compiletest: Warn on redundant --cfg directive when revisions are used #131925
• compiletest: Disambiguate html-tidy from rust tidy tool #131941
• compiletest: Custom differ #131181
• compiletest: Don't allow test revisions that conflict with built in cfgs #131930
• compiletest: Dynamically link run-make support #132225
• compiletest: Improve robustness of LLVM version handling #132315
• compiletest: Add "reference" as a known compiletest header #131382⁴
• tests/run-make, CI: Add aarch64-gnu-debug job #131207
• meta: Tag PRs affecting compiletest with A-compiletest #131682

Fixes

• compiletest: Fix up-to-date checking for run-make tests #131681
• compiletest: Suppress Windows Error Reporting (WER) for run-make tests⁵
• compiletest: Error on trying to use revisions in run-make tests #131614
• tests/run-make, CI: Run the full stage 2 run-make test suite in x86_64-gnu-debug #131917
• bootstrap, tests/run-make: Don't stage-off to previous compiler when CI rustc is available #132006
• bootstrap, tests/mir-opt: Match std RUSTFLAGS for host and target for mir-opt test suite to fix double std build/rebuilds #131442
• emscripten: Fix bootstrap and compiletest handling of emscripten target tests as part of Fix most ui tests on emscripten target #131705

Cleanups

• compiletest: Extract auxiliary-crate properties to their own module/struct #131541
• compiletest: Rename directive needs-profiler-support to needs-profiler-runtime #131429
• compiletest: Move debugger setup code out of lib.rs #131638
• compiletest: Remove the one thing that was checking a directive's original_line #131585
• compiletest: Store test collection context/state in two structs #131870
• compiletest: Tidy up how tidy and tidy (html version) are disambiguated #131961
• compiletest: Make line_directive return a DirectiveLine #132033
• compiletest: Rename command-list.rs to directive-list.rs #132313
• compiletest: Remove the magic hacks for finding output with lto=thin #131524
• compiletest: Simplify the choice of --emit mode for assembly tests #131525
• compiletest: Move debugger setup code out of lib.rs #131638

Documentation updates

• rustc-dev-guide: Document compiletest directives ignore-coverage-map and ignore-coverage-run #2094
• rustc-dev-guide: Rename needs-profiler-support to needs-profiler-runtime #2095
• rustc-dev-guide: Fix and update docs for needs-force-clang-based-tests #2085
• rustc-dev-guide: Add redirects for integration-testing and headers #2092
• rustc-dev-guide: Describe minicore test auxiliary and directive #2097
• rustc-dev-guide: Fix minicore.rs link #2122
• rustc-dev-guide: Add a link for the reference compiletest header #2096

On-going efforts

Note: there are certainly many more spontaneous efforts, this is more what I know is "planned".

• Reworking compiletest directive handling to be more robust and improve test coverage