Speed up Rust CI pipelines that use Tarpaulin
Rust is an awesome language. Not only does it provide you with runtime performance that is on par with languages like C and C++, but it also prevents you from shooting yourself in the foot thanks to a lot of compile time checks. This however means that the compiler has to do a lot more work compared to some other languages: Borrow checker, Types, Generics, Macros, LLVM Optimization… While those features do result in a better product, no one wants to sit around 30 minutes just to discover that a single test failed due to a typo.
In this article, we’ll go over some specific tips to minimize agonizingly slow recompilations while using the code coverage tool Cargo Tarpaulin.
Using caches to speed up pipelines
A simple solution to speed up most pipelines is to add a cache that persists across pipeline runs. The following example uses Gitlab caches to have a separate Rust-related cache for each git branch:
variables: CARGO_HOME: $CI_PROJECT_DIR/cargo cache: key: $CI_COMMIT_REF_SLUG paths: - cargo/ - target/ tests: script: - cargo test
Note that we also cache the cargo home directory by setting its environment variable. Doing this prevents each pipeline job from having to download and recompile all dependencies every time from scratch. Why recompile something that does not change, right? It turns out that’s not always that simple…
What is Tarpaulin?
Tarpaulin is a code coverage tool for the Cargo build system. It provides metrics for how many lines are covered by unit and integration tests, which can be extremely helpful in identifying possible bugs from missing tests. An example from our crypto library at Identeco could look like this:
> cargo tarpaulin Sep 09 13:26:47.297 INFO cargo_tarpaulin::report: Coverage Results: || Uncovered Lines: || src/credential_check/client.rs: 72, 88 || src/credential_check/server.rs: 97, 115 || ... || Tested/Total Lines: || src/credential_check/client.rs: 28/30 -6.67% || src/credential_check/server.rs: 63/65 -3.08% || ... 97.68% coverage, 547/560 lines covered, -0.6909937888198812% change in coverage
Here we can see that 13 lines are currently not covered.
Tarpaulin tells us where and how the test coverage changed compared to the last run.
cargo tarpaulin in place of
cargo test in our CI Pipeline, we can forward the change in coverage to be displayed within the merge request and help reviewers ensure that new changes are properly tested.
Integrating Tarpaulin into the pipeline
Now back to our original problem:
We want to speed up our pipeline and avoid recompilations while also using Tarpaulin.
The first thought might be to use a pipeline cache as demonstrated above, switch
cargo test for
cargo tarpaulin and call it a day.
But as it turns out, Tarpaulin uses the
--force-clean flag and recompiles everything by default regardless of whether an incremental compilation already exists.
This is done to avoid some minor coverage bugs with cargo, but ends up cleaning our cache at every pipeline run as well.
The confusing behavior of
cargo tarpaulin --skip-clean should speed everything up, right?
Well, in isolation, yes, but you might end up scratching your head why everything ends up even slower than before.
Let’s look at the following Gitlab pipeline snippet:
variables: CARGO_HOME: $CI_PROJECT_DIR/cargo cache: key: $CI_COMMIT_REF_SLUG paths: - cargo/ - target/ check: script: - cargo check tests: script: - cargo tarpaulin --skip-clean
We have two pipeline jobs:
Both use the globally defined pipeline cache with the
Here’s the catch: Not only does
tarpaulin seems to ignore the
--skip-clean flag, even
cargo check has to recompile everything now! This behavior can also be reproduced by running the commands locally, which is especially annoying when your IDE runs
cargo check regularly.
Recompilation because of Rustflags
As it turns out after some GitHub issue digging, Tarpaulin uses different
RUSTFLAGS for compilation:
> cargo tarpaulin --print-rust-flags Sep 09 14:11:55.260 INFO cargo_tarpaulin::config: Creating config Sep 09 14:11:55.367 INFO cargo_tarpaulin: Printing `RUSTFLAGS` RUSTFLAGS="-Cdebuginfo=2 --cfg=tarpaulin -Clink-dead-code"
Aha! Incremental compilation files can only be reused if running with the same
This explains the weird behavior above:
--skip-clean seems to be ignored since the initial build from
cargo check can’t be used by
cargo tarpaulin due to the different flags.
But now Tarpaulin overrides this build with it’s own coverage build, which in return can’t be used by
cargo check in the next pipeline run anymore, making our cache essentially useless.
At the time of initial writing, this recompilation behavior was not explicitly documented, but has since been added to the Tarpaulin docs:
As Tarpaulin changes the
RUSTFLAGSwhen building tests sometimes rebuilds of test binaries can’t be avoided. There is also a
--skip-cleanargument, the default has been changed at times to avoid issues with incremental compilation when changing
RUSTFLAGS. If you aim to reduce the amount of unnecessary recompilation attempting to add the
--skip-cleanflag should be the first step. After that you can either:
cargo tarpaulin --print-rust-flagsand use those flags for dev and coverage
--target-dirwhen running tarpaulin and have a coverage build and dev build
What’s the solution?
We have two possibilities:
Either we can add a separate pipeline cache to the
tests job, or we can set a different
--target-dir for Tarpaulin.
We’ve opted to set the target dir to
Instead of passing a flag, this can also be configured in
tarpaulin.toml to prevent confusion for developers running Tarpaulin locally.
After exchanging with the Tarpaulin maintainer, he noted that this isn’t the default due to the resulting bloated target directory. Additionally, this behavior should improve in the future with this issue that is waiting for the implementation of another RFC.
If you want to speed up general rust compilation times more, you can also check out this blog by Endler.dev.
A note on coverage misses
Remember the example output for Tarpaulin that I showed above? Well, the presumably uncovered lines are actually covered.
Whether you use
--skip-clean or not, Tarpaulin is still battling misses in coverage, which can cause some confusion for developers.
In our specific example, it seems that formatting can cause misses in line coverage, as the following unformatted one-liner code is covered:
let response = self.send_request(request).await.map_err(ClientError::Request)?;
While the formatted multi-line equivalent is flagged as uncovered despite explicit tests:
let response = self .send_request(request) .await .map_err(ClientError::Request)?;
Published of Fabian Odenthal