Bazel rules¶
load(
"@rules_requirements//rr:defs.bzl",
"rr_annotations_check",
"rr_annotations_test",
"rr_editor",
"rr_evidence",
"rr_golden_test",
"rr_model",
"rr_node_test",
"rr_py_test",
"rr_report",
"rr_rust_test",
"rr_sets_lock_test",
"rr_wrapped_test",
)
See Getting started for the MODULE.bazel setup. Besides the rules,
the module provides these targets:
Target |
What |
|---|---|
|
The Python library ( |
|
The |
|
The googletest hook ( |
|
Per-case JUnit for plain-assert C++ tests ( |
|
Flag, default |
|
The Rust hook crate ( |
|
The node:test |
|
The model’s JSON Schema. |
|
The JSON report’s schema ( |
Under bazel run, the CLI resolves relative paths against the directory you ran
Bazel from.
Model¶
rr_model¶
rr_model(
name = "model",
srcs = glob(["requirements/**/*.yaml"]),
strict = False,
lock = "requirements/verification.rrlock",
)
Declares the model files and, unless validate = False, a <name>_test that
runs rr validate on them. The test writes one JUnit case per check family
(rr.validate::shape, ::references, ::coverage-rules, ::claims,
::lock), so a model test claimed by a requirement is per-case evidence.
Attribute |
Default |
|
|---|---|---|
|
required |
Model files ( |
|
|
Treat validation warnings as errors in |
|
|
Create |
|
|
The verification-set lock ( |
|
Visibility of the model target. |
|
|
Forwarded to the validation test ( |
The target provides RrModelInfo(srcs, lock) and its files as DefaultInfo
(the lock is in its runfiles, not its files).
rr_annotations_test¶
rr_annotations_test(
name = "annotations_test",
model = ":model",
srcs = glob(["src/**/*.py", "src/**/*.rs"]),
)
Fails if an annotation in srcs names an id the model does not define
(Source annotations). Only the listed files are scanned, so the test is
hermetic and cached.
rr_annotations_check¶
rr_annotations_check(
name = "check_annotations",
model = ":model",
)
The non-hermetic counterpart for annotations that span many packages:
bazel run :check_annotations scans the whole source tree bazel run was
invoked in (git-aware, so ignored files are skipped) and fails on any
annotation naming an unknown id. Extra rr scan flags go after --
(-- --list, -- --exclude 'third_party/*').
rr_editor¶
rr_editor(
name = "editor",
model = ":model",
paths = ["requirements"], # optional: open the directory (one-object-per-file layouts)
)
bazel run :editor starts the web editor (Web editor) on the model.
Model and evidence paths are relative to the workspace root, so the editor edits
the real files, never runfiles copies.
Attribute |
Default |
|
|---|---|---|
|
required |
The |
|
the model’s files |
Workspace-relative files or directories to open instead. |
|
|
Workspace-relative evidence paths. |
|
|
Extra |
|
|
Extra Python dependencies: your pip hub’s |
Test hooks¶
rr_py_test¶
rr_py_test(
name = "controller_test",
srcs = ["tests/test_controller.py", "tests/conftest.py"],
deps = [":thermostat", requirement("pytest")],
)
A py_test that runs pytest with the rr marker plugin and writes JUnit to
$XML_OUTPUT_FILE. pytest is not bundled: add your workspace’s pytest to
deps.
Attribute |
Default |
|
|---|---|---|
|
required |
Files named |
|
|
Dependencies, including pytest. |
|
|
Extra pytest arguments, baked into the generated main (see below). |
|
|
Runtime data. |
|
Forwarded to |
rr_wrapped_test¶
rr_wrapped_test(
name = "parser_test",
test = ":parser_test_bin", # usually tagged "manual"
level = "sil",
)
Runs a test executable through rr wrap (Test hooks), converting its output
to traceability JUnit and preserving its exit code.
A runner that writes JUnit itself, to a fixed path (a Go or JavaScript test
runner, a hardware harness using CheckPlan), uses format = "junit"; the
wrapper passes its report on to Bazel and adds the exit-status taint:
rr_wrapped_test(
name = "bench_test",
test = ":bench_runner",
format = "junit",
junit_in = "${TEST_TMPDIR}/bench/junit.xml", # where the runner writes
level = "hitl",
)
Attribute |
Default |
|
|---|---|---|
|
required |
The test executable. |
|
|
Its output format: |
|
|
With |
|
|
Level for cases that do not declare one. |
|
|
Extra arguments for the executable. |
|
Forwarded to the wrapper |
rr_rust_test¶
load("@rules_rust//rust:defs.bzl", "rust_test")
rr_rust_test(
name = "setpoint_test",
rule = rust_test,
crate = ":setpoint",
deps = ["@rules_requirements//rust:rr"],
)
A Rust test whose rr::verifies!(...) calls become JUnit traces. Creates
<name>_bin — the real rust_test, tagged manual — and <name>, the wrapper
that bazel test runs. rules_requirements does not load rules_rust itself:
pass the rust_test rule as rule.
Attribute |
Default |
|
|---|---|---|
|
required |
The |
|
|
Level for cases that do not declare one. |
|
Applied to the wrapper test. |
|
|
|
Arguments for the test binary (after |
|
Environment of the wrapper, inherited by the test binary. |
|
|
Forwarded to the |
The wrapper also turns a crash into evidence: when the binary fails, a test
that recorded traces but never reported a result (an abort mid-run) becomes an
error case carrying its own declared id, and a binary that exits non-zero
after every test passed gets an exit-status error case. That case declares no
requirement: it is target-scope (rr.scope=target) and taints every case
claimed on the target.
It understands --nocapture output (the result on a line of its own). Call
rr::verifies! on the test’s own thread: traces from spawned threads or async
runtimes match no test and are dropped (the wrapper warns about them).
rr_node_test¶
# MODULE.bazel: bazel_dep(name = "aspect_rules_js", version = "3.2.2") # or newer;
# its default Node 22 toolchain is enough: no toolchain or npm setup is needed.
load("@aspect_rules_js//js:defs.bzl", "js_test")
rr_node_test(
name = "clocksync_test",
rule = js_test,
test = ":dist-test/tests/clocksync.test.js",
data = [":web_tests_js", ":dist_test_pkg_json"],
)
A node:test file with one JUnit case per test (node:test). The macro
creates <name>, a js_test whose entry point is a generated
<name>.rr_node_main.cjs: it runs the test file in a child node — with the
same node flags, environment and exit code — and writes the JUnit. A copy of
the reporter, <name>.rr_node_reporter.mjs, sits next to it (rules_js runs
entry points from the output tree). rules_requirements does not load rules_js
itself: pass the js_test rule as rule. Target names are yours, so swapping
a js_test for an rr_node_test changes no label, test_suite or CI command.
Attribute |
Default |
|
|---|---|---|
|
required |
The |
|
required |
The node:test file: a source file of this package, or a generated one (e.g. a |
|
|
Runtime data — the rest of the compiled sources, their |
|
|
Arguments for the test file, baked into the entry point ( |
|
|
Level for cases that do not declare one. |
|
Forwarded to the |
googletest needs no macro: a cc_test depending on
@rules_requirements//cc:gtest writes traced JUnit by itself, and so does a
plain-assert cc_test depending on @rules_requirements//cc:case
(rr_case.h).
Evidence and reports¶
rr_evidence¶
rr_evidence(
name = "evidence",
tests = [":controller_test", ":interlock_test", ":setpoint_test"],
)
Runs the tests inside a build action and collects their JUnit in a directory
laid out like bazel-testlogs (<name>/testlogs/<pkg>/<test>/test.xml, with
each test’s output in a test.log beside it), so every result keeps its test’s
label. Failing tests do not fail the build — they are evidence, and the report
shows them as FAILED. The output is cached like any other action: the tests run
again only when something they depend on changes.
Attribute |
Default |
|
|---|---|---|
|
required |
Test targets to run. |
|
|
Per-test timeout in seconds; a test that exceeds it gets |
|
|
Add |
|
|
Each test runs with its runfiles directory (<exe>.runfiles/<workspace>) as
working directory and an environment modelled on bazel test: TEST_SRCDIR,
RUNFILES_DIR, TEST_WORKSPACE, TEST_TARGET, XML_OUTPUT_FILE,
TEST_TMPDIR (also HOME and TMPDIR), TEST_UNDECLARED_OUTPUTS_DIR,
PATH and LANG, plus the test’s own env attribute. A test that writes no
JUnit gets one synthetic test case carrying its exit status, as under
bazel test; a test that exits non-zero although its report shows no failure
gets an extra exit-status error case. A test’s args attribute is not
available to other rules, which is why the macros in this module bake their
arguments into a generated main instead.
The tests are built in the exec configuration, because the action runs them
on the build machine. (Using the target configuration would reuse the binaries
bazel test builds, but with Bazel 7’s test-configuration trimming a non-test
rule that depends on tests conflicts with the tests’ own actions.) The cost is a
second build of the tests. Tests that need the network, devices, or anything else
the sandbox does not provide — hardware-in-the-loop suites, typically — do not
belong in rr_evidence: run them with bazel test and aggregate the real
bazel-testlogs with the CLI (see Integrating into an existing monorepo). The target provides
RrEvidenceInfo(testlogs).
rr_report¶
rr_report(
name = "report",
model = [":model"],
evidence = [":evidence", "evidence/panel_inspection.rr.yaml"],
srcs = glob(["src/**/*.py"]),
)
Renders the report as <name>.html, <name>.json and <name>.md, addressable
as :<name>.json and so on.
Attribute |
Default |
|
|---|---|---|
|
required |
|
|
|
|
|
|
Sources to scan for annotations: adds implementation links and |
|
|
Which outputs to build. |
|
|
Report title (default: the project name). |
|
|
Fail on model and attribution warnings too ( |
|
|
Current artifact identity; evidence recorded against another is stale. |
|
|
Also create |
|
|
The lane the evidence comes from ( |
|
|
A file listing the targets that lane runs, one label per line ( |
|
|
A quarantined test case (several ids, several claimants, one test code with several owners) fails the build ( |
|
|
The evidence comes from tests. |
The build fails if the model is invalid, if a test case is quarantined (unless
on_attribution_error = "warn"), and if an attribution issue is an error
(lock-owner-changed, a rule set to error, or with strict any warning).
An rr_model with a lock pins the sets. The target provides
RrReportInfo(json, lane, on_attribution_error, lock).
rr_sets_lock_test¶
rr_sets_lock_test(
name = "lock_test",
model = ":model", # rr_model(lock = "verification.rrlock")
evidence = [":evidence"],
)
Runs rr sets check on the lock the model pins (rr_model(lock), the one
rr_report reads): fails on a case the lock expects but the evidence does
not hold (missing-case), an owned case the lock does not list
(unlocked-member), an owner change and a stale entry. lock names the lock
only for a model that has none (model files, or an rr_model without
lock); analysis fails when it differs from the model’s. bazel run :lock_test.update rewrites that lock in the source tree from the same evidence
(rr sets lock --write; append -- --allow-removals to drop entries the
evidence no longer has, after checking the run was complete). For hermetic
projects whose evidence comes from rr_evidence; with real bazel-testlogs
run rr sets check / rr sets lock from the CLI. To start a lock, create an
empty lock file and run the .update target.
The thermostat example wires the three together the way a product should: an
rr_model with lock, an rr_report with the default
on_attribution_error = "fail" and check = True, and an
rr_sets_lock_test over the same evidence, so bazel test //... fails on a
shared claim (:model_test), a quarantined case (:report), a report that
breaks the partition (:report_check_test) and a set that changed without
its lock (:sets_lock_test):
EVIDENCE = [
":evidence",
"evidence/panel_inspection.rr.yaml",
]
# A quarantined test case (one naming two requirements, or claimed by two)
# fails this build; :report_check_test re-proves from report.json alone that
# no test case verifies two requirements.
rr_report(
name = "report",
srcs = SOURCES,
check = True,
evidence = EVIDENCE,
model = [":model"],
)
# The lock agrees with the evidence: no member missing, none unlocked, no
# owner changed (`rr sets check`). `bazel run //:sets_lock_test.update`
# rewrites requirements/verification.rrlock after an intended change.
rr_sets_lock_test(
name = "sets_lock_test",
evidence = EVIDENCE,
model = ":model",
)
rr_golden_test¶
rr_golden_test(
name = "report_golden_test",
src = ":report.json",
golden = "report.golden.json",
)
Compares a generated file with a checked-in golden and prints a unified diff
when they differ. bazel run :report_golden_test.update rewrites the golden
from the current output. Pinning report.json (or report.md) this way makes
every change to traceability — a requirement losing its evidence, a test
starting to fail, a risk’s status changing — show up in code review.
Generated mains¶
The helper tests (<model>_test, rr_annotations_test, rr_py_test,
rr_wrapped_test, rr_node_test, rr_golden_test) do not use the args attribute: Bazel
passes args only under bazel test / bazel run, so a test run by
rr_evidence would silently lose them. Instead each macro generates a small
<name>.rr_main.py with the arguments baked in (runfiles-relative paths,
resolved against the working directory at run time), and uses it as the test’s
main (rr_node_test: <name>.rr_node_main.cjs, its entry_point, with the
test file’s path baked in too). The tests therefore behave identically under bazel test, bazel run
and rr_evidence.
Compatibility¶
Tested with Bazel 7.7.1 and 8.8.1 using bzlmod. The module’s dependency floors
are rules_python 2.0.3, rules_cc 0.2.22, rules_rust 0.71.3 and googletest
1.17.0; newer versions in your workspace win. Python 3.9 or newer.
rr_node_test is tested with aspect_rules_js 3.2.2 (its default Node 22) and
its runner with Node 18 (the fallback), 20, 22 and 24; rules_js is not a dependency of the module.