Speed up Rust CI pipelines that use Tarpaulin

Rust is an awesome language. Not only does it provide you with runtime performance that is on par with languages like C and C++, but it also prevents you from shooting yourself in the foot thanks to a lot of compile time checks. This however means that the compiler has to do a lot more work compared to some other languages: Borrow checker, Types, Generics, Macros, LLVM Optimization… While those features do result in a better product, no one wants to sit around 30 minutes just to discover that a single test failed due to a typo.

Speed up Rust CI pipelines that use Tarpaulin Image by dashu83 on Freepik

In this article, we’ll go over some specific tips to minimize agonizingly slow recompilations while using the code coverage tool Cargo Tarpaulin.

Using caches to speed up pipelines

A simple solution to speed up most pipelines is to add a cache that persists across pipeline runs. The following example uses Gitlab caches to have a separate Rust-related cache for each git branch:

    - cargo/
    - target/
    - cargo test

Note that we also cache the cargo home directory by setting its environment variable. Doing this prevents each pipeline job from having to download and recompile all dependencies every time from scratch. Why recompile something that does not change, right? It turns out that’s not always that simple…

What is Tarpaulin?

Tarpaulin is a code coverage tool for the Cargo build system. It provides metrics for how many lines are covered by unit and integration tests, which can be extremely helpful in identifying possible bugs from missing tests. An example from our crypto library at Identeco could look like this:

> cargo tarpaulin
Sep 09 13:26:47.297  INFO cargo_tarpaulin::report: Coverage Results:
|| Uncovered Lines:
|| src/credential_check/client.rs: 72, 88
|| src/credential_check/server.rs: 97, 115
|| ...
|| Tested/Total Lines:
|| src/credential_check/client.rs: 28/30 -6.67%
|| src/credential_check/server.rs: 63/65 -3.08%
|| ...
97.68% coverage, 547/560 lines covered, -0.6909937888198812% change in coverage

Here we can see that 13 lines are currently not covered. Tarpaulin tells us where and how the test coverage changed compared to the last run. Awesome! When running cargo tarpaulin in place of cargo test in our CI Pipeline, we can forward the change in coverage to be displayed within the merge request and help reviewers ensure that new changes are properly tested.

Pipeline #10166 passed for e3bff582 on 16-fix-and-improve-… – 5 hours ago – Test coverage 100.00% (+12.04%) from 1 job
Figure: The pipeline report shows +12.04% coverage

Integrating Tarpaulin into the pipeline

Now back to our original problem: We want to speed up our pipeline and avoid recompilations while also using Tarpaulin. The first thought might be to use a pipeline cache as demonstrated above, switch cargo test for cargo tarpaulin and call it a day. But as it turns out, Tarpaulin uses the --force-clean flag and recompiles everything by default regardless of whether an incremental compilation already exists. This is done to avoid some minor coverage bugs with cargo, but ends up cleaning our cache at every pipeline run as well.

The confusing behavior of --skip-clean

So, using cargo tarpaulin --skip-clean should speed everything up, right? Well, in isolation, yes, but you might end up scratching your head why everything ends up even slower than before. Let’s look at the following Gitlab pipeline snippet:

    - cargo/
    - target/
    - cargo check
    - cargo tarpaulin --skip-clean

We have two pipeline jobs: check and tests. Both use the globally defined pipeline cache with the target/ and cargo/ directories. Here’s the catch: Not only does tarpaulin seems to ignore the --skip-clean flag, even cargo check has to recompile everything now! This behavior can also be reproduced by running the commands locally, which is especially annoying when your IDE runs cargo check regularly.

Recompilation because of Rustflags

As it turns out after some GitHub issue digging, Tarpaulin uses different RUSTFLAGS for compilation:

> cargo tarpaulin --print-rust-flags
Sep 09 14:11:55.260  INFO cargo_tarpaulin::config: Creating config
Sep 09 14:11:55.367  INFO cargo_tarpaulin: Printing `RUSTFLAGS`
RUSTFLAGS="-Cdebuginfo=2 --cfg=tarpaulin -Clink-dead-code"

Aha! Incremental compilation files can only be reused if running with the same RUSTFLAGS. This explains the weird behavior above: The --skip-clean seems to be ignored since the initial build from cargo check can’t be used by cargo tarpaulin due to the different flags. But now Tarpaulin overrides this build with it’s own coverage build, which in return can’t be used by cargo check in the next pipeline run anymore, making our cache essentially useless.

At the time of initial writing, this recompilation behavior was not explicitly documented, but has since been added to the Tarpaulin docs:

As Tarpaulin changes the RUSTFLAGS when building tests sometimes rebuilds of test binaries can’t be avoided. There is also a --clean and --skip-clean argument, the default has been changed at times to avoid issues with incremental compilation when changing RUSTFLAGS. If you aim to reduce the amount of unnecessary recompilation attempting to add the --skip-clean flag should be the first step. After that you can either:

  1. Use cargo tarpaulin --print-rust-flags and use those flags for dev and coverage
  2. Use --target-dir when running tarpaulin and have a coverage build and dev build

What’s the solution?

We have two possibilities: Either we can add a separate pipeline cache to the tests job, or we can set a different --target-dir for Tarpaulin. We’ve opted to set the target dir to target/tarpaulin. Instead of passing a flag, this can also be configured in tarpaulin.toml to prevent confusion for developers running Tarpaulin locally.

After exchanging with the Tarpaulin maintainer, he noted that this isn’t the default due to the resulting bloated target directory. Additionally, this behavior should improve in the future with this issue that is waiting for the implementation of another RFC.

If you want to speed up general rust compilation times more, you can also check out this blog by Endler.dev.

A note on coverage misses

Remember the example output for Tarpaulin that I showed above? Well, the presumably uncovered lines are actually covered. Whether you use --skip-clean or not, Tarpaulin is still battling misses in coverage, which can cause some confusion for developers.

In our specific example, it seems that formatting can cause misses in line coverage, as the following unformatted one-liner code is covered:

let response = self.send_request(request).await.map_err(ClientError::Request)?;

While the formatted multi-line equivalent is flagged as uncovered despite explicit tests:

let response = self

Published of Fabian Odenthal

Show more

  1. We're starting the Identeco Blog

    We're starting the Identeco Blog

    At Identeco, we learn new things every day. This is true for our developers, but also for our colleagues in sales and marketing as well as for our management. Since our society and the economy are changing continuously, we must enjoy constantly expanding our knowledge. Digitalization is a huge factor in rapid change. We want to play a part in ensuring that these changes don’t catch you cold. Because we believe that it is essential living an open culture of information and knowledge sharing when working with our customers.

Get to the blog