15 — Testing Standards

Status: Ratified Version: 1.0.0 Applies to: All code, tests/, CI gates, golden datasets

1. Testing Pyramid

          acceptance (E2E, few)
         integration (adapters, API, DB)
        unit (many, fast, isolated)
      component (per subsystem, in-process)
    ────────────────────────────────────
    golden (regression datasets, cross-layer)

2. Unit Tests

• One test module per production module (test_<module>.py).
• No network, no real DB, no subprocess, no AI provider (all via fakes).
• Domain tests use real objects with injected fakes; no mocks of domain internals.
• Property-based tests (hypothesis) for parsers, normalizers, canonicalizers.
• Table-driven where sensible.
• Naming: test_<behavior>_<condition>.

3. Integration Tests

• Real PostgreSQL/Redis/queue against test databases (CI services or docker-compose).
• Adapter tests run real tool binaries where available; otherwise skipped with marker @pytest.mark.tools (explicit skip rationale logged).
• API tests exercise routes against the real app factory with the same dependency overrides used in production composition.
• AI provider tests use a mock provider implementing the provider contract (no external calls); one smoke test per real provider gated by env key.

4. Regression Tests

• Every bug fix ships with a regression test that fails on the old behavior.
• Golden tests guard parser/normalizer output stability (data/golden/): inputs → expected canonical events; any schema/output change requires updating goldens deliberately (reviewed in PR).

5. Performance Tests

• tests/performance/ benchmarks via pytest-benchmark.
• CI gate: hot-path benchmarks regress ≤ 10%.
• Nightly load tests: ingest 10k/s, 1M-finding triage, 1,000 concurrent targets.
• Failure = artifact (flame graph, profile) attached for triage.

6. Security Tests

• Sandbox escape: fixtures attempt escapes; assert blocked + quarantined.
• Injection: tool args with ;, |, $(...), path traversal, template injection — assert neutralization.
• Secret leakage: run representative paths; assert no secrets in logs, errors, reports, or AI prompts (grep-style assertions on masked output).
• Scope enforcement: out-of-scope target rejected at planner, executor, and sandbox.
• RBAC: forbidden resource access returns 403; tenant isolation verified.
• Deserialization: malicious payloads rejected.
• Prompt injection: poisoned tool output does not alter decisions (UNSUPPORTED flagged).
• Supply chain: dependency scan + plugin signature verification tests.
• See 13 - Security Standards.md §15.

7. Golden Dataset

Located data/golden/. Structure:

data/golden/
├── tools/<tool-id>/
│   ├── input_<case>.txt|jsonl      # raw tool output fixture
│   └── expected_<case>.json         # expected canonical events
└── workflows/<workflow-id>/         # end-to-end pipeline goldens

Rules:

• Goldens are versioned; changes are deliberate and reviewed.
• Every parser change must update or add goldens.
• A golden mismatch fails CI unless the golden file itself is updated in the PR.

8. Acceptance Tests

• Full mission runs on sandboxed/vulnerable-by-design targets (e.g., DVWA, OWASP Juice Shop, Metasploitable in isolated docker network).
• Assert: mission completes, expected findings present with evidence, reports generated in required formats, no scope violation, audit log populated.
• Acceptance criteria per profile: see 24 - Quality Assurance.md §2.

9. Coverage Requirements

• Coverage enforced in CI (coverage + threshold fail).
• Exceptions (adapters without binaries) require explicit # pragma: no cover with justification, reviewed.

10. Test Doubles

• Fakes for ports (in-memory stores, fake queue, fake cache) — preferred.
• Stubs for canned tool outputs.
• Mock provider for AI contract testing.
• Spies for event-bus assertions.
• Mocks used sparingly and only at boundaries; no mocking of the code under test’s own modules.

11. CI Execution

12. Local Testing Commands

pytest                                   # default: unit+golden+component
pytest -m integration                    # integration (needs services)
pytest -m tools                          # real tool binaries
pytest -m security                       # security suite
pytest -m perf --benchmark-enable        # performance subset

13. References

• 04 - Coding Standards.md §14 (code review gates)
• 13 - Security Standards.md §15 (security tests)
• 14 - Performance Standards.md §9 (performance tests)
• 22 - Tool Integration Standard.md §Tests