17 — Error Handling Standards

Status: Ratified Version: 1.0.0 Applies to: All layers; exception taxonomy; retries; rollback; graceful degradation; recovery

1. Philosophy

HunterX treats failures as first-class, observable, and recoverable events. Errors are never silent, never swallowed, and never leave the system in an unknown state. The platform fails loud but degrades gracefully.

2. Exception Taxonomy

Root: HunterXError. Hierarchy:

HunterXError
├── ConfigurationError        (bad config/manifest/schema)
├── ScopeViolationError       (HX-SCOPE-*)        [non-retryable, critical]
├── ValidationError           (HX-VALID-*)        [non-retryable]
├── NotFoundError             (HX-NF-*)           [non-retryable]
├── AuthenticationError / AuthorizationError      [non-retryable]
├── TransientError            (HX-TRANS-*)        [retryable]
│   ├── NetworkError
│   ├── TimeoutError
│   ├── RateLimitedError
│   ├── ProviderUnavailableError
│   └── ToolCrashedError
├── PermanentError            (HX-PERM-*)         [non-retryable]
│   ├── ToolMissingError
│   ├── UnsupportedModeError
│   ├── ParseError
│   └── DataIntegrityError
└── CancelledError            (HX-CANCEL-*)       [no retry]

Rules:

Every exception carries: code (stable machine string), message (safe, no secrets), retryable flag, context (dict), and is logged with correlation_id.
Domain errors are defined in domain/exceptions; infra errors map to these.
Never raise raw third-party exceptions across layer boundaries; translate at adapter boundaries.

3. Error Codes

Format: HX-<AREA>-<NNN>. Areas: SCOPE, VALD, TRANS, PERM, AUTH, CFG, PLUGIN, TOOL, DB, AI, QUEUE, CANCEL.

Codes are registered in docs/bible/README.md error-code table (or linked register) and are stable across releases (new codes added; never reused).

Example:

Code	Meaning	Retryable
HX-SCOPE-001	Target outside engagement scope	no
HX-TRANS-001	Transient network failure	yes
HX-TRANS-002	Tool timed out	yes (bounded)
HX-PERM-001	Tool binary missing	no
HX-AI-001	AI output failed schema validation	yes (bounded)
HX-DB-001	Store unavailable	yes (bounded)
HX-PLUGIN-001	Plugin load failure	no

4. Failure Handling by Layer

Layer	Behavior
Domain	Raise typed domain errors; never I/O errors leak in
Application	Map use-case outcomes; do not leak stack traces to delivery
CLI	Print `code`, safe message, and remediation hint; exit code per table
API	JSON error envelope (`20` §Errors)
Worker/Engine	Step-level retry → degrade → pause → fail (per `10` §7)
Adapter	Translate tool failures to typed outcomes (`06` §13)

5. Retry Policy

Retries are always bounded (max attempts + backoff + jitter).
Only retryable=True errors retry.
Backoff: exponential by default (base=1s, factor=2, max=60s, jitter).
Retry budgets: per-step (workflow), per-mission, per-AI-call.
Retry counters and final outcome recorded in telemetry and audit.

Policy	Default
Transient network	3 attempts
Tool timeout	2 attempts
Rate limited	up to policy, honor `Retry-After`
AI schema failure	2 attempts then fallback
DB unavailable	3 attempts then degrade
Scope violation	never retried; abort + alert

6. Rollback

Transactional steps (DB writes, evidence storage): full rollback on failure — nothing half-persisted; events are idempotent.
External side effects (tool actions): cannot be universally rolled back. Document per tool in its knowledge file (07 §Errors) the cleanup/rollback steps. Where possible, the executor runs cleanup hooks.
Compensating actions: for multi-step operations with side effects, define compensating steps in the workflow (e.g., revoke a created account in a test environment).
Partial successes are preserved, never discarded; the audit trail records exactly what succeeded.

7. Graceful Degradation

Degradation ladder (applied in order, recorded in events/logs):

Fallback: use alternate provider/tool/strategy (AI → rule-based; provider A → B; tool X → Y with same capability).
Route around: mark tool unavailable; continue with peers; skip with reason.
Scope reduction: execute reduced-profile (fast over thorough).
Pause for operator: stop and request intervention (approval gate).
Fail the unit, preserve the mission: isolate failed step; mission may complete-with-warnings.

Degradation is observable: degradation.activated event with reason and level; metrics hx_degradation_activations_total.

8. Recovery & Resume

Workflows: resume from checkpoint (10 §6–7).
Missions: on crash/restart, MissionService reconciles state from TIDB; in-flight runs resume or are marked for operator review.
Queue: at-least-once delivery; consumers idempotent; DLQ captures poison messages for inspection.
Databases: transactional integrity + WAL point-in-time recovery (09 §11).

9. Operator-Facing Errors

Errors in CLI/API are: actionable (what happened), safe (no internals, no secrets), and referencable (error code).
Every error output includes a remediation hint or a docs link.
Machine-readable: API returns {error: {code, message, details?, retryable, correlation_id}}; CLI --output json emits the same envelope.

10. Testing Error Paths

Every retry policy has a unit test (attempt limits, backoff, jitter bounds).
Failure injection: integration tests inject provider/tool/DB failures and assert degradation behavior (15 §3).
Chaos-style tests (nightly): random timeouts, killed workers, provider outages → assert no data loss, graceful degrade, resume works.

11. References

10 - Workflow Engine.md §7 (workflow recovery) & §11 (error classes)
13 - Security Standards.md §7 (tool execution safety)
18 - Logging Standards.md (error event fields)
19 - CLI Standards.md / 20 - REST API Standards.md (surface errors)