Testing Guide¶

Test Scope in bf-dev¶

The harness itself includes smoke-level tests in tests/ focused on scripts and orchestration behavior.

Running Harness Tests¶

Run focused harness tests directly from tests/ when changing BFDev scripts. For example:

tests/service_quality_test.sh
mise exec -- uv run --dev pytest tests/code_index_test.py

Service-level Tests¶

Use the root Mise quality tasks for checked-out service repos:

mise run test core
mise run test web
mise run test manage
mise run test

The same selector model is available for linting and type checks:

mise run lint web
mise run type-check web

Selectors match repositories under bf/. Exact aliases such as core and web resolve to their matching service repos, broad selectors such as manage resolve to all checked-out matching repos, and no selector runs every detected service command.

When a service has its own mise.toml, the root task delegates to the service-local task through Mise monorepo task discovery. For example:

mise run //bf/manage-core:test
mise run //bf/manage-web:lint

Services without local Mise tasks can still be handled by fallback detection for common npm, uv, pipenv, and make test/lint commands. Container-only checks should still be run inside the relevant downstream service container.

Mutation testing¶

Mutation testing checks whether ordinary tests detect plausible mistakes in production code. The mutation tool generates altered variants; engineers do not maintain a separate class of mutation tests.

Use it selectively:

• Run an existing named scope when changed code overlaps that scope.
• Consider a bounded ad hoc run for important deterministic business, parsing or validation logic after an escaped defect, substantial refactor, property-based test addition or material uncertainty about test sensitivity.
• Skip generated code, trivial accessors, thin framework wiring and adapters whose behaviour is dominated by external systems or mocks unless a specific risk justifies the cost.

Do not use a repository-wide mutation run, a mutation percentage target or a combined quality score. Keep the scope, outcome, duration and unresolved gaps as separate evidence.

Running configured scopes¶

List services with mutation tasks, then inspect the selected service's scopes:

mise run mutation --list
mise run mutation schedule-creator --list-scopes

Run every named scope configured by one service or select one explicitly:

mise run mutation schedule-creator
mise run mutation schedule-creator --scope gtfs-time

Use an ad hoc Mutmut name or wildcard to investigate a prospective scope:

mise run mutation schedule-creator \
  --mutant '_helpers.time_helpers.x_parse_gtfs_time*'

The strict command passes only when every selected mutant is killed. --report-only returns exploratory results without making them delivery proof. Use --max-children to bound local concurrency.

Interpreting results¶

• killed: an ordinary test detected the altered behaviour.
• survived: decide whether the changed behaviour matters. If it does, strengthen an ordinary behavioural test. A survivor can also expose a production defect, as the missing test may describe intended behaviour that the implementation does not satisfy.
• no tests, not checked, timeout or tool error: the run is incomplete and cannot support delivery.
• Equivalent or irrelevant mutant: narrow or split the scope rather than add a meaningless assertion solely to improve a result.

Report the selected scope, killed and non-killed counts, duration, tests added, production defects found and any unresolved equivalent mutants or tool limits.

Creating a named scope¶

Start with a bounded ad hoc run. Promote it to a named scope only when:

• the production logic is important, deterministic and likely to contain meaningful boundary or branching mistakes;
• focused ordinary tests can exercise it without a broad or fragile runtime;
• the selected mutations are mostly meaningful rather than equivalent noise;
• the runtime is repeatable and fits the service's CI limit.

A complete scope addition includes:

1. The production-code selection and focused test selection in the service's mutation-tool configuration.
2. A stable named scope in the service runner.
3. Ordinary behavioural tests that kill every meaningful selected mutant.
4. A service-local Mise task and usage documentation.
5. A path-triggered GitLab job that retains machine-readable results and blocks dependent delivery when the job exists.

Each new scope must also update the GitLab change paths that trigger it. Use a separate CI job when independent path rules or runtime controls would avoid running unrelated scopes together.

Manage Web worker count¶

Manage Web limits Vitest to two workers by default to avoid exhausting memory on developer machines. A capable workstation can increase the limit for one run:

VITEST_MAX_WORKERS=4 mise run test web

For a persistent machine-specific override, create bf/manage-web/mise.local.toml with:

[env]
VITEST_MAX_WORKERS = "4"

This local file is ignored by Git. The Manage Web CI test jobs set VITEST_MAX_WORKERS=8 for their xlarge runners.

Validation Layers¶

Practical Checks¶

• Start target mode and ensure healthchecks pass.
• Validate critical UI/API paths for the active domain.
• Re-run after ./pull when dependencies move.