惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

aimingoo的专栏
aimingoo的专栏
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
博客园_首页
Recent Announcements
Recent Announcements
A
About on SuperTechFans
云风的 BLOG
云风的 BLOG
T
Troy Hunt's Blog
F
Fortinet All Blogs
Webroot Blog
Webroot Blog
S
Secure Thoughts
D
Docker
Attack and Defense Labs
Attack and Defense Labs
博客园 - 叶小钗
H
Heimdal Security Blog
S
Security Affairs
Microsoft Azure Blog
Microsoft Azure Blog
The GitHub Blog
The GitHub Blog
T
Tenable Blog
I
InfoQ
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
L
LangChain Blog
Project Zero
Project Zero
Cyberwarzone
Cyberwarzone
NISL@THU
NISL@THU
有赞技术团队
有赞技术团队
AWS News Blog
AWS News Blog
Threat Intelligence Blog | Flashpoint
Threat Intelligence Blog | Flashpoint
IT之家
IT之家
Vercel News
Vercel News
www.infosecurity-magazine.com
www.infosecurity-magazine.com
C
CXSECURITY Database RSS Feed - CXSecurity.com
T
Threatpost
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org
Forbes - Security
Forbes - Security
博客园 - 聂微东
Security Latest
Security Latest
T
Threat Research - Cisco Blogs
Latest news
Latest news
Exploit-DB.com RSS Feed
Exploit-DB.com RSS Feed
Stack Overflow Blog
Stack Overflow Blog
Cloudbric
Cloudbric
Google DeepMind News
Google DeepMind News
C
CERT Recently Published Vulnerability Notes
N
Netflix TechBlog - Medium
酷 壳 – CoolShell
酷 壳 – CoolShell
T
The Blog of Author Tim Ferriss
J
Java Code Geeks
C
Check Point Blog
Recorded Future
Recorded Future
Jina AI
Jina AI

Hacker News

Introducing Claude Opus 4.7 Qwen Studio The Future of Everything is Lies, I Guess: Where Do We Go From Here? GitHub - SeanFDZ/macmind: Single-layer transformer in HyperTalk for the classic Macintosh Show HN: Agent-cache – Multi-tier LLM/tool/session caching for Valkey and Redis Bonsai 1-bit WebGPU - a Hugging Face Space by webml-community Moving a large-scale metrics pipeline from StatsD to OpenTelemetry / Prometheus GitHub - Nightmare-Eclipse/RedSun: The Red Sun vulnerability repository GitHub - SethPyle376/hiraeth: Local AWS emulator focused on fast integration testing, with SQS support, SQLite-backed state, and a debug-friendly web UI. GitHub - macOS26/Agent: Any AI, replaces Claude Code, Cursor, OpenClaw. Over 18 LLM providers (Claude, OpenAI, Gemini, Ollama, Zai, HF, Qwen) wired into a native Mac app that writes code, builds Xcode projects, bumps versions, manages git, automates Safari, use AppleScript, JS or Accessibility, extend Agent! w/ MCP Servers, run tasks from your iPhone via Messages. YouTube now lets you turn off Shorts I Made a Terminal Pager Burgers | マクドナルド公式 Commands — HackerNews CLI documentation ChatGPT for Excel PiCore - Raspberry Pi Port of Tiny Core Linux Live Nation illegally monopolized ticketing market, jury finds Google Broke Its Promise to Me. Now ICE Has My Data. Founding Engineer at Adaptional | Y Combinator CRISPR takes important step toward silencing Down syndrome’s extra chromosome GitHub - saffron-health/libretto: The AI toolkit for building reliable browser automations US v. Heppner (S.D.N.Y. 2026) no attorney-client privilege for AI chats [pdf] Retrofitting JIT Compilers into C Interpreters IPv6 – Google The Accursèd Alphabetical Clock Cybersecurity Looks Like Proof of Work Now Fragments: April 14 Cal.com Goes Closed Source: Why AI Security Is Forcing Our Decision | Cal.com - Scheduling Software for Online Bookings Laravel raised money and now injects ads directly into your agent When moving fast, talking is the first thing to break Too much Discussion of the XOR swap trick – Heather Cafe Introduction to Spherical Harmonics for Graphics Programmers The Grand Line Building a Z-Machine in the worst possible language High-Level Rust: Getting 80% of the Benefits with 20% of the Pain GitHub - duguyue100/midnight-captain: Inspired by Midnight Commander, tailored to my taste. How to build a `git diff` driver · Jamie Tanna | Software Engineer Center for Responsible, Decentralized Intelligence at Berkeley The Local Universe’s Expansion Rate Is Clearer Than Ever, but Still Doesn’t Add Up - A new synthesis of astronomical measurements confirms a persistent mismatch that could point to physics beyond current models The air throughout our homes is infused with microplastics. But there are things you can do to breathe less of them The disturbing white paper Red Hat is trying to erase from the internet – OSnews The Future of Everything is Lies, I Guess: Annoyances ‘Abhorrent’: the inside story of the Polymarket gamblers betting millions on war Productive procrastination — Max van IJsselmuiden maps, territory and LMs 447 Terabytes per Square Centimetre at Zero Retention Energy: Non-Volatile Memory at the Atomic Scale on Fluorographane Show HN: Pardonned.com – A searchable database of US Pardons 20 Years on AWS and Never Not My Job The Seasons are Wrong Artemis II crew splashes down near San Diego after historic moon mission We gave an AI a 3 year retail lease in SF and asked it to make a profit | Andon Labs How a dancer with ALS used brainwaves to perform live On filing the corners off my MacBooks Installing every* Firefox extension OpenClaw’s memory is unreliable, and you don’t know when it will break Steve Blank Nowhere Is Safe Chimpanzees in Uganda locked in vicious 'civil war', say researchers watgo - a WebAssembly Toolkit for Go linux/Documentation/process/coding-assistants.rst at master · torvalds/linux GitHub - callumlocke/json-formatter: Makes JSON easy to read. Founding Product Engineer at Bild AI | Y Combinator A compelling title that is cryptic enough to get you to take action on it GitHub - Keychron/Keychron-Keyboards-Hardware-Design: Industrial design files for Keychron keyboards and mice. 100+ models with CAD assets in STEP, DXF, DWG, and PDF. Source-available, with commercial use allowed for original compatible accessories within the license terms. [ANNOUNCE] WireGuardNT v0.11 and WireGuard for Windows v0.6 Released 1D-Chess Helium Is Hard to Replace Cooperative Vectors Introduction | Evolve Keeping a Postgres queue healthy — PlanetScale Our response to the Axios developer tool compromise Do Americans read print books, e-books or audiobooks more? The Zettelkasten Method in Obsidian: A Practical Setup Guide Artemis II Is Competency Porn and We Are Starving For It WeakC4 Flight Viz — Cockpit View A Mexican surveillance giant you’ve never heard of is now watching the U.S. border Surelock: Deadlock-Free Mutexes for Rust RISC-V 101 – what is it and what does it mean for Canonical? | Ubuntu The Problem That Built an Industry How Much Linear Memory Access Is Enough? | Solidean Investigating Split Locks on x86-64 Simplest hash functions Sybilproof reputation mechanisms (2005) [pdf] What is a property? How Complex is my Code? Static code analysis in Kotlin — tools overview Toffoli gates are all you need PGLite evangelism dcmake: a new CMake debugger UI Clojure on Fennel part one: Persistent Data Structures Fragments: April 2 Python Release Python install manager 26.1 The Life and Death of the Book Review - Liberties Introducing Database Traffic Control — PlanetScale Bitcoin miners are losing $19,000 on every BTC produced as difficulty drops 7.8% God sleeps in the minerals Building slogbox Apple Silicon and Virtual Machines: Beating the 2 VM Limit Who was “Not Even Wrong” first? Pokemon Evolution Vs Darwinian Evolution The APL Programming Language Source Code
Optimizing #[sqlx::test] rebuild time
kobzol.githu · 2026-06-22 · via Hacker News

You might find this post especially useful if you have a project with many #[sqlx::test] tests.

One of the upstream Rust projects that I worked on during the past few years was the rewrite of bors, the merge queue bot we use to merge all rust-lang/rust PRs. If you are interested in learning more about this bot, check out my talk from RustWeek 2026.

I’m quite proud of the integration test suite of bors, which I spent a lot of effort on, and thanks to which the bot has been working pretty much flawlessly since we deployed it to production in January 2026 (despite GitHub lately often having… troubles).

One thing that I’m not very happy about though is the incremental rebuild time of bors, and in particular its test suite. It takes a long time (~8-10s) to rebuild the tests after each change on my laptop, which is quite bad for productivity.

Recently I finally found some time to profile1 its build time, and learned that it is caused by a combination of several factors:

  • Generation of debug information takes a long time. This is a known issue, but in this case I didn’t want to give up debug info, because I actually debug and step through bors tests quite often.
  • rustc takes a long time to load and persist the incremental session. I plan to take a look into this.
  • Probably because of all the debuginfo (the final binary has like 220 MiB), it takes lld a whole second (!) to link the tests. With wild, it’s just ~200ms.
  • The sqlx::tests that I am using heavily in bors take a long time to compile. This is what I will focus on in this post.

Slow compilation of sqlx tests

As a frame of reference, for my benchmark I was using touch <test-file> && time cargo test --no-run. Even after a no-op change, it took ~7.5 seconds to recompile the tests, which is super slow.

Of course, it is very well known that sqlx’s proc macros can slow down compilation times, because of all the crimes interesting things that they do2. However, the case that I encountered might not be so obvious. In my case, sqlx did not actually even connect to a database! Because I’m compiling with SQLX_OFFLINE=1, unless I work directly on SQL queries. And yes, I am setting opt-level = 3 for the sqlx-macros crate, as recommended by the sqlx documentation.

So what is happening here? To find out, it is important to understand what is happening when you have a test like this:

#[sqlx::test]
async fn test_foo(pool: sqlx::PgPool) {}

The #[sqlx::test] attribute is super useful, because it creates a new database before the execution of the test, runs migrations on it, and then gives you a database connection pool, so that you can run your tests against an actual database, and not against a mocked HashMap3.

Wait, did I say migrations? Hmm, where does it find them? Well, from disk, of course! Each usage of #[sqlx::test] will gather all migrations from a directory on disk, and then read, parse, validate and hash each migration. Perhaps counter-intuitively, this part is not that slow! Turns out that Rust is actually quite fast (who knew, right??), and if you do not have gigabytes of migrations, I/O is probably also not a problem4.

What is worse is the generated output of those macros. For each such test, the macro will generate a complete list of migrations, including their text content and a checksum in the form of a byte array, in the Rust source code as a constant. So if you expand the macro, before each test you’ll find something like this:

args.migrator(&::sqlx::migrate::Migrator {
   migrations: ::std::borrow::Cow::Borrowed(&[
      ::sqlx::migrate::Migration {
        version: 20240517094752i64,
        description: ::std::borrow::Cow::Borrowed("create build"),
        migration_type: ::sqlx::migrate::MigrationType::ReversibleUp,
        sql: ::std::borrow::Cow::Borrowed("CREATE TABLE <skipped>)"),
        no_tx: false,
        checksum: ::std::borrow::Cow::Borrowed(&[193u8, 202u8, <skipped>]),
      },
      ::sqlx::migrate::Migration {
        <skipped>
      },
      <skipped>
   ]),
   <skipped>
});

The example above is shortened, and it skips a lot of stuff. The actual generated code will be much longer, and of course it scales with the number (and content) of your migrations.

Now, if you have something like this in your source code once, that’s not so bad. However, in bors, there are ~350 sqlx tests and 30 migrations. And at that point, it starts to add up rather quickly.

To test my hypothesis that migrations might be causing some of the build slowness, I tested what would happen if I had only one migration by deleting the rest of them. And sure enough, the rebuild time immediately went from ~7.5s to ~5s! What is perhaps even more telling is that the size of the output of cargo expand --lib --tests went from 32 MiB (!) with 30 migrations to “only” 6 MiB with a single migration. Compiling an additional 26 MiB of Rust code sure isn’t for free.

It wasn’t just about the compilation time of the generated code though. In the profiles, it looked like converting all the migration description data to tokens using the quote crate during the proc macro execution also takes a non-trivial amount of time.

This behavior can be pretty inconspicious, because at the start of the project, the rebuilds were fast (or at least, faster). But then, with each added test, and each added migration, the rebuild time slowly increases, so it creeps up on you.

I tried if the experimental proc macro caching feature that I landed in the compiler last year might help, but it didn’t. Probably the time needed to compile the generated code dwarfs the time to run the proc macro itself, so it does not help if the proc macro itself is cached.

Nevermind: #[sqlx::test] is an attribute, not a derive macro, so this flag doesn’t apply here. Thanks to @futile for the correction.

What can be done about it

First, I started thinking about reducing the size of the generated code, e.g. by representing the checksum byte array in a more compact form. While this would likely help, I realized that as long as we generate code for all migrations next to each test, there will still be too much code.

Then I tried to patch sqlx to move the loading of the migrations to (test) runtime from compilation time, to get rid of all the inlined migrations. This actually had the desired effect! The rebuild time went down to ~5s, and (at least in the case of bors), the cargo expand output was reduced to ~6 MiB, all while the execution time of the tests wasn’t affected in a measurable way. The change wasn’t even very complicated, all that’s needed is to generate code that will call the function to load migrations at runtime, rather than calling that function in the proc macro and then embedding the description of all migrations in the generated source code.

So essentially, you go from this (pseudo-code):

fn sqlx_proc_macro() -> TokenStream {
  let migrations = generate_migrations();
  quote! {
    Migrator {
        migrations: #migrations
    }
  }
}

to this:

fn sqlx_proc_macro() -> TokenStream {
  quote! {
    Migrator {
        migrations: ::sqlx::generate_migrations()
    }
  }
}

However, loading the migrations at runtime might have some disadvantages, e.g. the tests would no longer be self-contained.

I went to the sqlx Discord server and asked if people have any suggestions. My proposal was that sqlx could either load the migrations at runtime (which I described above), or that there could be a shared variable with the migrations that all the tests would reference, to avoid the generated code bloat. Funnily enough, one person responded and told me that the second solution is already implemented (well, sort of). You can actually specify a path to a variable containing the migrations to apply in #[sqlx::test]:

// The macro generates the migrations, we store it in a single variable.
const MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!();

// Each test just references the variable, instead of inlining all the
// migrations next to the test's source code.
#[sqlx::test(migrator = "crate::MIGRATOR")]
async fn test_1(pool: sqlx::PgPool) {}

#[sqlx::test(migrator = "crate::MIGRATOR")]
async fn test_2(pool: sqlx::PgPool) {}

I wasn’t sure if const or static would be better here, and I didn’t measure any rebuild time differences. But static makes more sense to me, to ensure that the data exists just once in the final binary.

This solution worked great for bors! After adding the migrator argument to all #[sqlx::test] instances with a bit of Find + Replace magic, the rebuild time went down to ~5s.

While this works, what I don’t like about it is the annoying migrator = "crate::MIGRATOR" attribute that I have to remember to add to all my tests to avoid the migration code bloat problem. I think that it would be quite elegant to specify a default value for the migrator argument in the sqlx.toml configuration file added in the 0.9 release of sqlx, so that the shared variable would be used by default, without having to think about it.

I opened an issue that proposes this feature. Let’s see what do sqlx maintainers think, although I would understand if they had some concerns about a feature like this.

In any case, even this manual solution was quite helpful, and it made my test rebuild times a bit faster. There is still a lot of space for improvements, as 5s is still slow, though.

Maybe it would be useful to mention this “footgun” in the #[sqlx::test] documentation. I suggested that here.

Conclusion

I think that this post offers two main takeaways:

  • If you have a project with many #[sqlx::test] tests, and you suffer from long rebuild times, try to use the migrator = "..." trick to see if that helps.
  • If you have proc macros that generate a lot of code, measuring the byte size of your code using cargo expand can be a relatively good predictor of compilation time :)

If you have further suggestions on how to optimize rebuild times when using sqlx, let me know on Reddit.