Speed up Rust CI pipelines that use Tarpaulin

In this article, we'll go over some specific tips to minimize agonizingly slow recompilations while using the code coverage tool Cargo Tarpaulin.

Reading time:

6 min

Share

Image by dashu83 on Freepik

Rust is an awesome language. Not only does it provide you with runtime performance that is on par with languages like C and C++, but it also prevents you from shooting yourself in the foot thanks to a lot of compile time checks. This however means that the compiler has to do a lot more work compared to some other languages: Borrow checker, Types, Generics, Macros, LLVM Optimization… While those features do result in a better product, no one wants to sit around 30 minutes just to discover that a single test failed due to a typo.

In this article, we’ll go over some specific tips to minimize agonizingly slow recompilations while using the code coverage tool Cargo Tarpaulin.

Using caches to speed up pipelines

A simple solution to speed up most pipelines is to add a cache that persists across pipeline runs. The following example uses Gitlab caches to have a separate Rust-related cache for each git branch:

variables:
  CARGO_HOME: $CI_PROJECT_DIR/cargo
cache:
  key: $CI_COMMIT_REF_SLUG
  paths:
    - cargo/
    - target/
tests:
  script:
    - cargo test

Note that we also cache the cargo home directory by setting its environment variable. Doing this prevents each pipeline job from having to download and recompile all dependencies every time from scratch. Why recompile something that does not change, right? It turns out that’s not always that simple…

What is Tarpaulin?

Tarpaulin is a code coverage tool for the Cargo build system. It provides metrics for how many lines are covered by unit and integration tests, which can be extremely helpful in identifying possible bugs from missing tests. An example from our crypto library at Identeco could look like this:

> cargo tarpaulin
Sep 09 13:26:47.297  INFO cargo_tarpaulin::report: Coverage Results:
|| Uncovered Lines:
|| src/credential_check/client.rs: 72, 88
|| src/credential_check/server.rs: 97, 115
|| ...
|| Tested/Total Lines:
|| src/credential_check/client.rs: 28/30 -6.67%
|| src/credential_check/server.rs: 63/65 -3.08%
|| ...
97.68% coverage, 547/560 lines covered, -0.6909937888198812% change in coverage

Here we can see that 13 lines are currently not covered. Tarpaulin tells us where and how the test coverage changed compared to the last run. Awesome! When running cargo tarpaulin in place of cargo test in our CI Pipeline, we can forward the change in coverage to be displayed within the merge request and help reviewers ensure that new changes are properly tested.

Integrating Tarpaulin into the pipeline

Now back to our original problem: We want to speed up our pipeline and avoid recompilations while also using Tarpaulin. The first thought might be to use a pipeline cache as demonstrated above, switch cargo test for cargo tarpaulin and call it a day. But as it turns out, Tarpaulin uses the --force-clean flag and recompiles everything by default regardless of whether an incremental compilation already exists. This is done to avoid some minor coverage bugs with cargo, but ends up cleaning our cache at every pipeline run as well.

The confusing behavior of --skip-clean

So, using cargo tarpaulin --skip-clean should speed everything up, right? Well, in isolation, yes, but you might end up scratching your head why everything ends up even slower than before. Let’s look at the following Gitlab pipeline snippet:

variables:
  CARGO_HOME: $CI_PROJECT_DIR/cargo
cache:
  key: $CI_COMMIT_REF_SLUG
  paths:
    - cargo/
    - target/
check:
  script:
    - cargo check
tests:
  script:
    - cargo tarpaulin --skip-clean

We have two pipeline jobs: check and tests. Both use the globally defined pipeline cache with the target/ and cargo/ directories. Here’s the catch: Not only does tarpaulin seems to ignore the --skip-clean flag, even cargo check has to recompile everything now! This behavior can also be reproduced by running the commands locally, which is especially annoying when your IDE runs cargo check regularly.

Recompilation because of Rustflags

As it turns out after some GitHub issue digging, Tarpaulin uses different RUSTFLAGS for compilation:

> cargo tarpaulin --print-rust-flags
Sep 09 14:11:55.260  INFO cargo_tarpaulin::config: Creating config
Sep 09 14:11:55.367  INFO cargo_tarpaulin: Printing `RUSTFLAGS`
RUSTFLAGS="-Cdebuginfo=2 --cfg=tarpaulin -Clink-dead-code"

Aha! Incremental compilation files can only be reused if running with the same RUSTFLAGS. This explains the weird behavior above: The --skip-clean seems to be ignored since the initial build from cargo check can’t be used by cargo tarpaulin due to the different flags. But now Tarpaulin overrides this build with it’s own coverage build, which in return can’t be used by cargo check in the next pipeline run anymore, making our cache essentially useless.

At the time of initial writing, this recompilation behavior was not explicitly documented, but has since been added to the Tarpaulin docs:

As Tarpaulin changes the RUSTFLAGS when building tests sometimes rebuilds of test binaries can’t be avoided. There is also a --clean and --skip-clean argument, the default has been changed at times to avoid issues with incremental compilation when changing RUSTFLAGS. If you aim to reduce the amount of unnecessary recompilation attempting to add the --skip-clean flag should be the first step. After that you can either:

1. Use cargo tarpaulin --print-rust-flags and use those flags for dev and coverage
2. Use --target-dir when running tarpaulin and have a coverage build and dev build

What’s the solution?

There are two possibilities: Either we can add a separate pipeline cache to the tests job, or we can set a different --target-dir for Tarpaulin. Instead of passing a flag, this can also be configured in tarpaulin.toml to prevent confusion for developers running Tarpaulin locally. We’ve opted to set the target dir to target/tarpaulin. We had some issues with setting the --target-dir, as some build artifacts were occasionally factored into the coverage results. So instead we opted for the separate cache. If you’re using GitLab-Ci, make sure to use different cache keys like rust-$CI_COMMIT_REF_SLUG and tarpaulin-$CI_COMMIT_REF_SLUG.

After exchanging with the Tarpaulin maintainer, he noted that a different target directory isn’t the default due to the resulting increase in disk space usage. Additionally, this behavior should improve in the future with this issue that is waiting for the implementation of another RFC.

If you want to speed up general rust compilation times more, you can also check out this blog by Endler.dev.

A note on coverage misses

Remember the example output for Tarpaulin that I showed above? Well, the presumably uncovered lines are actually covered. Whether you use --skip-clean or not, Tarpaulin is still battling misses in coverage, which can cause some confusion for developers.

In our specific example, it seems that formatting can cause misses in line coverage, as the following unformatted one-liner code is covered:

let response = self.send_request(request).await.map_err(ClientError::Request)?;

While the formatted multi-line equivalent is flagged as uncovered despite explicit tests:

let response = self
    .send_request(request)
    .await
    .map_err(ClientError::Request)?;