惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

P
Privacy International News Feed
CTFtime.org: upcoming CTF events
CTFtime.org: upcoming CTF events
Jina AI
Jina AI
T
Tailwind CSS Blog
WordPress大学
WordPress大学
Scott Helme
Scott Helme
C
Cybersecurity and Infrastructure Security Agency CISA
博客园 - Franky
C
CERT Recently Published Vulnerability Notes
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
雷峰网
雷峰网
Schneier on Security
Schneier on Security
博客园 - 聂微东
T
Tor Project blog
Hugging Face - Blog
Hugging Face - Blog
博客园 - 司徒正美
AI
AI
T
Troy Hunt's Blog
Security Latest
Security Latest
T
The Blog of Author Tim Ferriss
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org
C
Check Point Blog
T
Threat Research - Cisco Blogs
W
WeLiveSecurity
V
Vulnerabilities – Threatpost
Recorded Future
Recorded Future
Recent Commits to openclaw:main
Recent Commits to openclaw:main
Cisco Talos Blog
Cisco Talos Blog
C
CXSECURITY Database RSS Feed - CXSecurity.com
Cloudbric
Cloudbric
J
Java Code Geeks
罗磊的独立博客
C
Cyber Attacks, Cyber Crime and Cyber Security
aimingoo的专栏
aimingoo的专栏
L
LangChain Blog
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
P
Privacy & Cybersecurity Law Blog
Google DeepMind News
Google DeepMind News
Threat Intelligence Blog | Flashpoint
Threat Intelligence Blog | Flashpoint
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
L
Lohrmann on Cybersecurity
I
InfoQ
MongoDB | Blog
MongoDB | Blog
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
The GitHub Blog
The GitHub Blog
The Hacker News
The Hacker News
H
Help Net Security
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
P
Proofpoint News Feed
N
News and Events Feed by Topic

Lobsters

CIFSwitch: a non-universal Linux local root vulnerability RIPE NCC session fixation: poaching logins with an Atlas probe GNOME 2.20 but its Web Components Agentic Search for Context Engineering – Leonie Monigatti Garnix is shutting down [not OC] akashina.tngl.sh/jjc Concerning Emacs (and Jazz) Nitpicking the shell history scene in ‘Tron: Legacy’ What's cooking on SourceHut? Q2 2026 The tenth OpenPGP email summit Package managers that package package managers Clojure on Fennel part three: parsing WordPress at 23 Finding Miscompiles for Fun, Not Profit GitHub - creusot-rs/creusot: Creusot helps you prove your Rust code is correct. Announcing Rust 1.96.0 | Rust Blog A Love Letter to Neovim sqlite AGENTS.md Am I a Bad Friend? CSS vs. JavaScript • Josh W. Comeau Erlang Ecosystem Foundation - Supporting the BEAM community A brief note about slot access cost in Common Lisp Keyboard latency probe Rethinking the GNOME clipboard issues Back to the Building Blocks’ Building Blocks Tech Notes: Theseus: translating win32 to wasm Fast is better than slow Content-addressed Rust builds (or, what kache actually caches) Intent to Prototype: Embedding API Canada’s Bill C-22 and the security cost of collecting more data 5 PostgreSQL locking behaviors that trip people up okmij.org Stop advertising in your commits! | AksDev GitHub - mplsllc/macsurf: A modern web browser for Classic Mac OS 9 PowerPC. Real CSS3, ES5 JavaScript, native HTTPS — built with CodeWarrior on the Carbon API. Introducing DoomBench - Can Your Data Stack Run DOOM? What are some of your favourite developer tools? Building a Scalable Ingestion Pipeline with Temporal (Part 1) Converting shallow Git bundles into normal repositories Are you a member of any professional associations? What is a harmonic? An interactive comic about additive synthesis How Virtual Tables Work in the Itanium C++ ABI Using SwiftUI to Build a Mac-assed App in 2026 Rust (and Slint) on a jailbroken Kindle. ~jack/lambda-on-lambda - Serverless Haskell on AWS - sourcehut git Human proof for FOSS contributions Extremely simple internet radio controlled via IRC Announcing BABLR Splitting Konsole views from Helix to run tools | AksDev GitHub - yugr/rust-slides Serving files over HTTP three ways: synchronous, epoll, and io_uring update docs with information about building with build.py (#979) · astral-sh/python-build-standalone@c9c40c5 A Simple Makefile Tutorial On C extensions, portability, and alternative compilers Switching to Colemak | Pedro Alves Just How Bad Was The Intel IAPX432? Nix's Substituter List Is Not a Routing Table Accelerating copy_if using SIMD Lambda on Lambda: Serverless Haskell on AWS | Blog Announcing feed-repeat v1.0 Scaling Akvorado BMP RIB with sharding EYG news: A host of CLI improvements, new guides and new effects The social contract of writing JS Crossword C array types are weird; and related topics Flatpak will depend on systemd – OSnews Migrating from Go to Rust | corrode Rust Consulting A portentous reunion Vivado Licensing Options How my minimal, memory-safe Go rsync steers clear of vulnerabilities the entropy layer of a wavelet codec, on its own GitHub - nferhat/fht-compositor: A dynamic tiling Wayland compositor. Debian SE Linux and PinTheft Does bulk memmove speed up std::remove_if? (No.) 声明式部分更新 | Blog | Chrome for Developers Fully in-browser container builds Dianne Skoll's Web Site - Remind The Architecture of Open Source Applications (Volume 1)Berkeley DB Pardon MIE? - ironPeak Blog “Long-Term Support” doesn’t mean what you think Jira IS Turing-Complete May I recommend thinking of Emacs as your Fortress of Solitude hershey Floodgap Gopher-HTTP gateway gopher://thelambdalab.xyz/1cuneiforth/ HP QuickWeb, Singular And Pointless That one time I used Go panics for flow control A new suite of modern tools coming for editing and publishing RFCs From the Tabletop… The Digital Antiquarian Building a Host-Tuned GCC to Make GCC Compile Faster Are we self-sovereign PKI yet? Claw Patrol: an open-source security firewall for agents | Deno Revised^7 Report on Scheme, Large: Procedural Fascicle Draft is now public A Network Allow-List Won't Stop Exfiltration — André Graf From AFSK to Goertzel – µArt.cz Software For My New Home Server Introducing Neptune: Direct3D virtualization for QEMU AI Agent Bankrupted Their Operator While Trying to Scan DN42 - Lan Tian @ Blog mimalloc: A new, high-performance, scalable memory allocator for the modern era Making wl_shm fast The Soul of Maintaining a New Machine - Third Draft | Books in Progress What is Git made of?
Optimizing #[sqlx::test] rebuild time
kobzol.githu · 2026-06-22 · via Lobsters

You might find this post especially useful if you have a project with many #[sqlx::test] tests.

One of the upstream Rust projects that I worked on during the past few years was the rewrite of bors, the merge queue bot we use to merge all rust-lang/rust PRs. If you are interested in learning more about this bot, check out my talk from RustWeek 2026.

I’m quite proud of the integration test suite of bors, which I spent a lot of effort on, and thanks to which the bot has been working pretty much flawlessly since we deployed it to production in January 2026 (despite GitHub lately often having… troubles).

One thing that I’m not very happy about though is the incremental rebuild time of bors, and in particular its test suite. It takes a long time (~8-10s) to rebuild the tests after each change on my laptop, which is quite bad for productivity.

Recently I finally found some time to profile1 its build time, and learned that it is caused by a combination of several factors:

  • Generation of debug information takes a long time. This is a known issue, but in this case I didn’t want to give up debug info, because I actually debug and step through bors tests quite often.
  • rustc takes a long time to load and persist the incremental session. I plan to take a look into this.
  • Probably because of all the debuginfo (the final binary has like 220 MiB), it takes lld a whole second (!) to link the tests. With wild, it’s just ~200ms.
  • The sqlx::tests that I am using heavily in bors take a long time to compile. This is what I will focus on in this post.

Slow compilation of sqlx tests

As a frame of reference, for my benchmark I was using touch <test-file> && time cargo test --no-run. Even after a no-op change, it took ~7.5 seconds to recompile the tests, which is super slow.

Of course, it is very well known that sqlx’s proc macros can slow down compilation times, because of all the crimes interesting things that they do2. However, the case that I encountered might not be so obvious. In my case, sqlx did not actually even connect to a database! Because I’m compiling with SQLX_OFFLINE=1, unless I work directly on SQL queries. And yes, I am setting opt-level = 3 for the sqlx-macros crate, as recommended by the sqlx documentation.

So what is happening here? To find out, it is important to understand what is happening when you have a test like this:

#[sqlx::test]
async fn test_foo(pool: sqlx::PgPool) {}

The #[sqlx::test] attribute is super useful, because it creates a new database before the execution of the test, runs migrations on it, and then gives you a database connection pool, so that you can run your tests against an actual database, and not against a mocked HashMap3.

Wait, did I say migrations? Hmm, where does it find them? Well, from disk, of course! Each usage of #[sqlx::test] will gather all migrations from a directory on disk, and then read, parse, validate and hash each migration. Perhaps counter-intuitively, this part is not that slow! Turns out that Rust is actually quite fast (who knew, right??), and if you do not have gigabytes of migrations, I/O is probably also not a problem4.

What is worse is the generated output of those macros. For each such test, the macro will generate a complete list of migrations, including their text content and a checksum in the form of a byte array, in the Rust source code as a constant. So if you expand the macro, before each test you’ll find something like this:

args.migrator(&::sqlx::migrate::Migrator {
   migrations: ::std::borrow::Cow::Borrowed(&[
      ::sqlx::migrate::Migration {
        version: 20240517094752i64,
        description: ::std::borrow::Cow::Borrowed("create build"),
        migration_type: ::sqlx::migrate::MigrationType::ReversibleUp,
        sql: ::std::borrow::Cow::Borrowed("CREATE TABLE <skipped>)"),
        no_tx: false,
        checksum: ::std::borrow::Cow::Borrowed(&[193u8, 202u8, <skipped>]),
      },
      ::sqlx::migrate::Migration {
        <skipped>
      },
      <skipped>
   ]),
   <skipped>
});

The example above is shortened, and it skips a lot of stuff. The actual generated code will be much longer, and of course it scales with the number (and content) of your migrations.

Now, if you have something like this in your source code once, that’s not so bad. However, in bors, there are ~350 sqlx tests and 30 migrations. And at that point, it starts to add up rather quickly.

To test my hypothesis that migrations might be causing some of the build slowness, I tested what would happen if I had only one migration by deleting the rest of them. And sure enough, the rebuild time immediately went from ~7.5s to ~5s! What is perhaps even more telling is that the size of the output of cargo expand --lib --tests went from 32 MiB (!) with 30 migrations to “only” 6 MiB with a single migration. Compiling an additional 26 MiB of Rust code sure isn’t for free.

It wasn’t just about the compilation time of the generated code though. In the profiles, it looked like converting all the migration description data to tokens using the quote crate during the proc macro execution also takes a non-trivial amount of time.

This behavior can be pretty inconspicious, because at the start of the project, the rebuilds were fast (or at least, faster). But then, with each added test, and each added migration, the rebuild time slowly increases, so it creeps up on you.

I tried if the experimental proc macro caching feature that I landed in the compiler last year might help, but it didn’t. Probably the time needed to compile the generated code dwarfs the time to run the proc macro itself, so it does not help if the proc macro itself is cached.

Nevermind: #[sqlx::test] is an attribute, not a derive macro, so this flag doesn’t apply here. Thanks to @futile for the correction.

What can be done about it

First, I started thinking about reducing the size of the generated code, e.g. by representing the checksum byte array in a more compact form. While this would likely help, I realized that as long as we generate code for all migrations next to each test, there will still be too much code.

Then I tried to patch sqlx to move the loading of the migrations to (test) runtime from compilation time, to get rid of all the inlined migrations. This actually had the desired effect! The rebuild time went down to ~5s, and (at least in the case of bors), the cargo expand output was reduced to ~6 MiB, all while the execution time of the tests wasn’t affected in a measurable way. The change wasn’t even very complicated, all that’s needed is to generate code that will call the function to load migrations at runtime, rather than calling that function in the proc macro and then embedding the description of all migrations in the generated source code.

So essentially, you go from this (pseudo-code):

fn sqlx_proc_macro() -> TokenStream {
  let migrations = generate_migrations();
  quote! {
    Migrator {
        migrations: #migrations
    }
  }
}

to this:

fn sqlx_proc_macro() -> TokenStream {
  quote! {
    Migrator {
        migrations: ::sqlx::generate_migrations()
    }
  }
}

However, loading the migrations at runtime might have some disadvantages, e.g. the tests would no longer be self-contained.

I went to the sqlx Discord server and asked if people have any suggestions. My proposal was that sqlx could either load the migrations at runtime (which I described above), or that there could be a shared variable with the migrations that all the tests would reference, to avoid the generated code bloat. Funnily enough, one person responded and told me that the second solution is already implemented (well, sort of). You can actually specify a path to a variable containing the migrations to apply in #[sqlx::test]:

// The macro generates the migrations, we store it in a single variable.
const MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!();

// Each test just references the variable, instead of inlining all the
// migrations next to the test's source code.
#[sqlx::test(migrator = "crate::MIGRATOR")]
async fn test_1(pool: sqlx::PgPool) {}

#[sqlx::test(migrator = "crate::MIGRATOR")]
async fn test_2(pool: sqlx::PgPool) {}

I wasn’t sure if const or static would be better here, and I didn’t measure any rebuild time differences. But static makes more sense to me, to ensure that the data exists just once in the final binary.

This solution worked great for bors! After adding the migrator argument to all #[sqlx::test] instances with a bit of Find + Replace magic, the rebuild time went down to ~5s.

While this works, what I don’t like about it is the annoying migrator = "crate::MIGRATOR" attribute that I have to remember to add to all my tests to avoid the migration code bloat problem. I think that it would be quite elegant to specify a default value for the migrator argument in the sqlx.toml configuration file added in the 0.9 release of sqlx, so that the shared variable would be used by default, without having to think about it.

I opened an issue that proposes this feature. Let’s see what do sqlx maintainers think, although I would understand if they had some concerns about a feature like this.

In any case, even this manual solution was quite helpful, and it made my test rebuild times a bit faster. There is still a lot of space for improvements, as 5s is still slow, though.

Maybe it would be useful to mention this “footgun” in the #[sqlx::test] documentation. I suggested that here.

Conclusion

I think that this post offers two main takeaways:

  • If you have a project with many #[sqlx::test] tests, and you suffer from long rebuild times, try to use the migrator = "..." trick to see if that helps.
  • If you have proc macros that generate a lot of code, measuring the byte size of your code using cargo expand can be a relatively good predictor of compilation time :)

If you have further suggestions on how to optimize rebuild times when using sqlx, let me know on Reddit.