Rustのテストは2種類のオプションがある

このエントリはRust Advent Calendarの1日目の記事です。

空いてる日を埋める担当のκeenです。1日目が空いてたので遡って記事を投稿します。 Rustでテストをするときに渡すオプションが2種類あるという話について。

テストを実行するときに cargo test を使うかと思います。この cargo test のヘルプメッセージは以下のようになっています。

cargo-test
Execute all unit and integration tests and build examples of a local package

USAGE:
    cargo test [OPTIONS] [TESTNAME] [-- ...]

OPTIONS:
    -q, --quiet                      Display one character per test instead of one line
        --lib                        Test only this package's library unit tests
        --bin ...              Test only the specified binary
        --bins                       Test all binaries
        --example ...          Test only the specified example
        --examples                   Test all examples
        --test ...             Test only the specified test target
        --tests                      Test all tests
        --bench ...            Test only the specified bench target
        --benches                    Test all benches
        --all-targets                Test all targets
        --doc                        Test only this library's documentation
        --no-run                     Compile, but don't run tests
        --no-fail-fast               Run all tests regardless of failure
    -p, --package ...          Package to run tests for
        --workspace                  Test all packages in the workspace
        --exclude ...          Exclude packages from the test
        --all                        Alias for --workspace (deprecated)
    -j, --jobs                    Number of parallel jobs, defaults to # of CPUs
        --release                    Build artifacts in release mode, with optimizations
        --profile      Build artifacts with the specified profile
        --features ...     Space or comma separated list of features to activate
        --all-features               Activate all available features
        --no-default-features        Do not activate the `default` feature
        --target ...         Build for the target triple
        --target-dir      Directory for all generated artifacts
        --manifest-path        Path to Cargo.toml
        --ignore-rust-version        Ignore `rust-version` specification in packages
        --message-format ...    Error format
        --unit-graph                 Output build graph in JSON (unstable)
        --future-incompat-report     Outputs a future incompatibility report at the end of the build (unstable)
    -v, --verbose                    Use verbose output (-vv very verbose/build.rs output)
        --color                Coloring: auto, always, never
        --frozen                     Require Cargo.lock and cache are up to date
        --locked                     Require Cargo.lock is up to date
        --offline                    Run without accessing the network
        --config ...      Override a configuration value (unstable)
    -Z ...                     Unstable (nightly-only) flags to Cargo, see 'cargo -Z help' for details
    -h, --help                       Prints help information

ARGS:
        If specified, only run tests containing this string in their names
    ...     Arguments for the test binary

Run `cargo help test` for more detailed information.

何やら色々オプションが並んでいますね。多くは cargo build などにもありますが、例えば --exclude--no-fail-fast などはテスト専用のオプションに見えます。

この影響でここにリストアップされているものが全てのような気がしますが、実はテストのオプションはまだまだあります。上記はCargo側のオプションで、コンパイルされたテスト側にもまだオプションはあるのです。テストバイナリ側のヘルプメッセージは cargo test -- --help-- を挟んでからオプションを渡してあげると見れます。

$ cargo test -- --help
Usage: --help [OPTIONS] [FILTERS...]

Options:
        --include-ignored 
                        Run ignored and not ignored tests
        --ignored       Run only ignored tests
        --force-run-in-process 
                        Forces tests to run in-process when panic=abort
        --exclude-should-panic 
                        Excludes tests marked as should_panic
        --test          Run tests and not benchmarks
        --bench         Run benchmarks instead of tests
        --list          List all tests and benchmarks
    -h, --help          Display this message
        --logfile PATH  Write logs to the specified file
        --nocapture     don't capture stdout/stderr of each task, allow
                        printing directly
        --test-threads n_threads
                        Number of threads used for running tests in parallel
        --skip FILTER   Skip tests whose names contain FILTER (this flag can
                        be used multiple times)
    -q, --quiet         Display one character per test instead of one line.
                        Alias to --format=terse
        --exact         Exactly match filters rather than by substring
        --color auto|always|never
                        Configure coloring of output:
                        auto = colorize if stdout is a tty and tests are run
                        on serially (default);
                        always = always colorize output;
                        never = never colorize output;
        --format pretty|terse|json|junit
                        Configure formatting of output:
                        pretty = Print verbose output;
                        terse = Display one character per test;
                        json = Output a json document;
                        junit = Output a JUnit document
        --show-output   Show captured stdout of successful tests
    -Z unstable-options Enable nightly-only flags:
                        unstable-options = Allow use of experimental features
        --report-time [plain|colored]
                        Show execution time of each test. Available values:
                        plain = do not colorize the execution time (default);
                        colored = colorize output according to the `color`
                        parameter value;
                        Threshold values for colorized output can be
                        configured via
                        `RUST_TEST_TIME_UNIT`, `RUST_TEST_TIME_INTEGRATION`
                        and
                        `RUST_TEST_TIME_DOCTEST` environment variables.
                        Expected format of environment variable is
                        `VARIABLE=WARN_TIME,CRITICAL_TIME`.
                        Durations must be specified in milliseconds, e.g.
                        `500,2000` means that the warn time
                        is 0.5 seconds, and the critical time is 2 seconds.
                        Not available for --format=terse
        --ensure-time   Treat excess of the test execution time limit as
                        error.
                        Threshold values for this option can be configured via
                        `RUST_TEST_TIME_UNIT`, `RUST_TEST_TIME_INTEGRATION`
                        and
                        `RUST_TEST_TIME_DOCTEST` environment variables.
                        Expected format of environment variable is
                        `VARIABLE=WARN_TIME,CRITICAL_TIME`.
                        `CRITICAL_TIME` here means the limit that should not
                        be exceeded by test.
        --shuffle       Run tests in random order
        --shuffle-seed SEED
                        Run tests in random order; seed the random number
                        generator with SEED


The FILTER string is tested against the name of all tests, and only those
tests whose names contain the filter are run. Multiple filter strings may
be passed, which will run all tests matching any of the filters.

By default, all tests are run in parallel. This can be altered with the
--test-threads flag or the RUST_TEST_THREADS environment variable when running
tests (set it to 1).

By default, the tests are run in alphabetical order. Use --shuffle or set
RUST_TEST_SHUFFLE to run the tests in random order. Pass the generated
"shuffle seed" to --shuffle-seed (or set RUST_TEST_SHUFFLE_SEED) to run the
tests in the same order again. Note that --shuffle and --shuffle-seed do not
affect whether the tests are run in parallel.

All tests have their standard output and standard error captured by default.
This can be overridden with the --nocapture flag or setting RUST_TEST_NOCAPTURE
environment variable to a value other than "0". Logging is not captured by default.

Test Attributes:

    `#[test]`        - Indicates a function is a test to be run. This function
                       takes no arguments.
    `#[bench]`       - Indicates a function is a benchmark to be run. This
                       function takes one argument (test::Bencher).
    `#[should_panic]` - This function (also labeled with `#[test]`) will only pass if
                        the code causes a panic (an assertion failure or panic!)
                        A message may be provided, which the failure string must
                        contain: #[should_panic(expected = "foo")].
    `#[ignore]`       - When applied to a function which is already attributed as a
                        test, then the test runner will ignore these tests during
                        normal test runs. Running with --ignored or --include-ignored will run
                        these tests.

こっちのオプションを使うとより精細にテストの実行をコントロールできます。

例えば --no-capture オプションを使えばテストが吐いた出力を見ることができます。

#[test]
fn foo() {
    println!("foo test");
}

#[test]
fn foo2() {
    println!("foo2 test");
}
$ cargo test -- --nocapture
cargo test -- --nocapture
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
     Running unittests (target/debug/deps/test_example-87b3befdea076842)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/tests.rs (target/debug/deps/tests-ed358733425f347a)

running 2 tests
foo test
foo2 test
test foo ... ok
test foo2 ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

あるいは正確に foo だけ実行したい場合に --exact フラグも有効です。 普通のテストの引数に名前を渡すものだと部分一致していればそのままテストされるので foofoo2foofoo_fail のようなテストが並んでいるときに foo だけを実行したくてもできません。

$ cargo test foo
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
     Running unittests (target/debug/deps/test_example-87b3befdea076842)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/tests.rs (target/debug/deps/tests-ed358733425f347a)

running 2 tests
test foo ... ok
test foo2 ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

そこでテストバイナリ側にある --exact フラグを使うと望んだものだけを実行できます。

$ cargo test -- --exact foo
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
     Running unittests (target/debug/deps/test_example-87b3befdea076842)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/tests.rs (target/debug/deps/tests-ed358733425f347a)

running 1 test
test foo ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

他にも実行スレッド数を制御したりといくつか必要になりそうな機能があるのでテストの実行のしかたを変えたいという人は一度バイナリ側のヘルプも読んでみて下さい。

Written by κeen