02 — HunterX System Architecture

Status: Ratified Version: 1.0.0 Applies to: Core, all modules, plugins, adapters, SDKs, deployment

1. Architectural Principles

HunterX follows Clean Architecture with strict dependency inversion. The binding rules (enforced by lint gates in CI, see 04 - Coding Standards.md §7):

Dependency Rule (outward-bound-free): source code dependencies point inward only. The Domain layer knows nothing about Infrastructure, Frameworks, or Delivery.
Abstraction over implementation: the Core depends on interfaces (ports); implementations (adapters) are injected at composition root.
Loose coupling via events and messages: subsystems communicate through an Event Bus and Message Bus; no subsystem imports another subsystem’s internals.
AI isolation: AI providers are reachable only through the AI Provider Abstraction. Agents and skills never call LLMs directly.
Pluggability as a contract: every capability is a plugin behind an interface. The core never imports a concrete plugin.
Deterministic core, stochastic periphery: AI, randomness, and timing are pinned and recorded so plans and reports are reproducible.

2. System Context Diagram (C4 Level 1)

                        +---------------------------+
                        |  Engagement Operator      |
                        |  (Pentester / Analyst)    |
                        +-------------+-------------+
                                      |
                    +-----------------+-----------------+
                    |               HunterX             |
                    |   AI-Powered Security Operations  |
                    |           Platform                |
                    +--------+--------+--------+--------+
                             |        |        |
              +--------------+        |        +-----------------+
              |                       |                          |
      +-------+--------+     +--------+-------+         +--------+--------+
      | External Tools |     | AI Providers   |         | External Data   |
      | (nmap, nuclei, |     | (Local/Ollama, |         | (CVE, CWE,      |
      | sqlmap, burp,  |     | Cloud LLMs)    |         |  CAPEC, MITRE,  |
      |  ... 100+ )    |     +----------------+         |  EPSS, CT, DNS) |
      +----------------+                                +-----------------+

Operator interacts through CLI, REST API, or scheduled missions.
External Tools are executed through the Tool Execution Sandbox; output is parsed and normalized.
AI Providers receive structured prompts, never raw secrets or raw target data beyond the engagement scope.
External Data enriches findings (CVE/EPSS) and intelligence (CT logs, DNS).

3. Container Diagram (C4 Level 2)

+------------------------------------------------------------------------------------+
|                              HUNTERX PLATFORM                                      |
|                                                                                    |
|  +------------------+   +------------------+   +------------------+                |
|  |   CLI             |   |   REST API        |   |   Scheduler      |                |
|  |   (Typer/Click)   |   |   (FastAPI)       |   |   (APScheduler)  |                |
|  +--------+---------+   +--------+---------+   +--------+---------+                |
|           |                     |                     |                             |
|           +---------------------+---------------------+                             |
|                                 |                                                   |
|                        +--------v---------+                                        |
|                        |   ORCHESTRATION  |                                        |
|                        |   (Mission)      |                                        |
|                        +--------+---------+                                        |
|                                 |                                                   |
|                 +---------------+----------------+                                 |
|                 |               |                 |                                |
|        +--------v-----+  +-----v--------+  +-----v---------+                       |
|        | Planner      |  | Workflow     |  | Agent Fleet   |                       |
|        | (AI + rules) |  | Engine       |  | (Multi-Agent) |                       |
|        +--------+-----+  +-----+--------+  +-----+---------+                       |
|                 |               |                 |                                |
|                 +-------+-------+-----------------+                                |
|                         |                                                         |
|               +---------v---------+                                               |
|               |   REASONING/      |                                               |
|               |   DECISION CORE   |                                               |
|               +---------+---------+                                               |
|                         |                                                         |
|      +------------------+--------------------+                                    |
|      |                  |                    |                                   |
| +----v-----+     +------v------+     +-------v-------+                           |
| | Tool     |     | Plugin      |     | AI Provider   |                           |
| | Execution|     | Host        |     | Abstraction   |                           |
| | Sandbox  |     | (SDK)       |     | (Routing,     |                           |
| +----+-----+     +------+------+     |  Cache,       |                           |
|      |                  |            |  Retry,       |                           |
| +----v-----+     +------v------+     |  Circuit)     |                           |
| | Parsers  |     | Registry    |     +-------+-------+                           |
| +----+-----+     +-------------+             |                                    |
|      |                                       |                                    |
| +----v---------------------------------------v----+                               |
| |           DATA & INFRASTRUCTURE LAYER           |                               |
| |  SQL (Target Intel DB) | Graph (KG) | Object    |                               |
| |  Store (Evidence) | Cache (Redis) | Queue (Rabbit) | Event Bus | Log/Telemetry |
| +--------------------------------------------------+-----------------------------+

4. Layered Architecture (C4 Level 3 — component view)

+----------------------------------------------------------------------------------+
| DELIVERY LAYER                                                                   |
|   CLI Commands   REST API   Webhook Sink   Scheduler Jobs                        |
+----------------------------------------------------------------------------------+
| APPLICATION LAYER (use-cases / application services)                              |
|   MissionService   WorkflowService   ReportService   AuthService   AdminService  |
+----------------------------------------------------------------------------------+
| DOMAIN LAYER (pure Python, no framework imports)                                  |
|   Entities: Target, Asset, Finding, Evidence, Mission, Plan, Task, Report         |
|   Value Objects: Port, Service, Risk, Severity, Confidence, Payload               |
|   Domain Services: Planner, Correlator, Validator, RiskScorer, Deduplicator       |
|   Ports (interfaces): ToolPort, StorePort, AIPort, QueuePort, CachePort,          |
|                       EventPort, SecretPort, SandboxPort, LogPort, MetricPort     |
+----------------------------------------------------------------------------------+
| INFRASTRUCTURE LAYER                                                              |
|   Adapters: SQLAlchemy/PostgreSQL, Neo4j, S3/FS, Redis, RabbitMQ, EventBus,       |
|             Vault, DockerSandbox, OpenTelemetry, JSONLogger                       |
|   Tool Integration: ToolExecutor, Parsers, Normalizers, Adapters, Knowledge Files |
|   AI Integration: OpenAIAdapter, OllamaAdapter, local model adapter, caches       |
+----------------------------------------------------------------------------------+

Dependency direction: Delivery → Application → Domain. Infrastructure implements Domain ports and is injected upward at composition root. Nothing in the Domain layer imports fastapi, sqlalchemy, redis, requests, or any third-party library outside the allowed core set.

5. Subsystem Specifications

5.1 Core Engine

The Core Engine is the Domain layer’s heart. It owns:

The unified security schema entities (08 - Unified Security Schema.md).
Domain services: planner, correlator, validator, deduplicator, risk scorer.
The mission lifecycle state machine (draft → approved → queued → running → completed → archived).
The ports that every other layer depends upon.

Responsibilities:

Enforce scope invariants (a target outside scope is rejected at the domain edge).
Guarantee determinism of plan IDs and finding hashes.
Provide the single source of truth for Entity, ValueObject, and aggregate logic.

Non-responsibilities:

No I/O, no HTTP, no DB, no subprocess, no AI calls. All of these happen through ports.

5.2 AI Engine (Reasoning & Decision Core)

The AI Engine converts goals into validated decisions.

Sub-components:

Component	Responsibility
`GoalAdapter`	Converts agent/mission goals into structured AI tasks
`PromptManager`	Builds versioned, schema-tight prompts (no secrets, scoped context)
`ProviderRouter`	Selects provider/model per task class, cost, latency, policy
`AIMiddlewarePipeline`	Logging, masking, schema validation, augmentation
`OutputValidator`	Validates raw LLM output against JSON Schema; rejects malformed
`ConsensusEngine`	Aggregates N samples; strict (unanimous) or relaxed (majority)
`ConfidenceScorer`	Assigns confidence 0.0–1.0 from agreement + calibration
`AICache`	Exact + semantic cache keyed on prompt+model+params
`RetryHandler` / `CircuitBreaker`	Provider resilience

Rules:

Agents NEVER call AI providers directly; they emit goals only.
All prompts and completions are logged with masking; never with secrets.
Every AI decision returns an AIProvenance record (prompt hash, model, seed, temperature, version, latency) attached to the resulting domain object.
See 11 - AI Standards.md for behavior, scoring, and learning rules.

5.3 Workflow Engine

Executes Directed Acyclic Graphs (DAGs) of steps.

Step types: condition, parallel, loop, wait, agent, tool, skill, submission, transform, notify.

Execution model: async, concurrency-controlled, dependency-resolved.
Checkpoint/resume: every step boundary is a checkpoint; recovery resumes at the last successful checkpoint.
Retries: per-step policy (max attempts, backoff, retryable error classes).
Validation: output schema validation gates each step.
Cancellation: graceful (finish in-flight step) and forceful (kill).
See 10 - Workflow Engine.md.

5.4 Mission Engine

The Mission Engine is the top-level orchestrator of a complete assessment.

Instantiates a Mission from a Mission Profile (12 - Mission Profiles.md).
Binds profile → workflow template → plan → execution → validation → reporting.
Owns mission state, progress, phase transitions, and the mission timeline.
Emits mission-level events (mission.started, mission.phase_changed, mission.completed, mission.failed, mission.aborted).
Enforces the engagement contract: scope, legality flags, destructive-action flags.

5.5 Planner

The Planner decides what to do next.

Two modes:

Template planner — deterministic expansion of mission profile templates into a plan DAG (reproducible, plan_id = hash(profile, target_scope, params, version)).
AI planner — adaptive replanning during execution: given live findings, the planner proposes next actions, reorders phases, prunes exhausted branches, and requests operator approval for high-risk actions.

Output: a Plan (DAG of PlanSteps), each step referencing a workflow step, tool, skill, or agent with explicit preconditions.

5.6 Knowledge Base

The Knowledge Base is the read-side, versioned corpus of structured knowledge:

Tool Knowledge Files (07 - Tool Knowledge Base Specification.md) — CLI syntax, capabilities, profiles, error codes, performance, AI usage rules.
Mission Profiles (12 - Mission Profiles.md).
Playbooks / skills — reusable detection and validation procedures.
CVE / CWE / CAPEC / MITRE ATT&CK / EPSS reference data (mirrored, versioned).
Best practices and workflow-position rules per tool.

Consumed by the Planner (what to run), the AI Engine (grounding), and the Workflow Engine (step definitions).

5.7 Knowledge Graph

The Knowledge Graph stores entities and their relationships for a mission and across missions:

Nodes: Asset, Domain, Host, IP, Service, Finding, Evidence, CVE, MITRE Technique, Payload, Person (OSINT), Certificate.
Edges: resolves_to, hosts, listens_on, runs, exposes, affected_by, exploits, maps_to, derived_from, correlates_with.

Used for:

Attack-path reconstruction and lateral-movement visualization.
Correlation across findings (5.11 Correlation Engine).
AI grounding (the planner and correlator read subgraphs, never the full graph).
Reporting visualizations (graph exports).

5.8 Plugin System

The Plugin System is the extension backbone. See 05 - Plugin SDK Specification.md.

Registry: discovers, validates, versions, and catalogs plugins.
Host: isolated runtime (subprocess or process-isolated) with a stable ABI.
Lifecycle: register → validate → install → load → execute → update → unload → remove.
Permissions: capability-based sandboxing (filesystem, network, process, AI).
Signing: plugins are signature-verified; unsigned plugins run in restricted mode.
Dependency management: plugin manifest declares dependencies; resolver checks versions and conflicts.

5.9 Tool SDK & Adapter Layer

Two-tier integration contract:

Tool Adapter SDK (06 - Tool Adapter SDK.md): Python interfaces — BaseAdapter, ScannerAdapter, CrawlerAdapter, EnumeratorAdapter, AnalyzerAdapter, ReporterAdapter, ValidationAdapter.
Tool Knowledge File (07 - Tool Knowledge Base Specification.md): YAML metadata describing how to invoke, parse, and reason about a tool.

Every adapter implements: execute(), parse(), normalize(), validate(), healthcheck(), capabilities().

5.10 Parser Engine & Normalizer

The Parser Engine and Normalizer convert arbitrary tool output into the Unified Security Schema.

Parser Engine: pluggable parsers per tool+format (JSON, XML, CSV, HTML, plain text, regex streams). Streams large outputs; never loads megabytes into memory.
Normalizer: maps parsed structures onto canonical entities (Finding, Service, Technology, Certificate, …). Applies canonicalization (IP/CIDR, FQDN case, dedup keys) and enrichment hooks.

Pipeline: tool stdout/stderr/file → Parser → Normalizer → CanonicalEvent → Domain entity → Store.

5.11 Correlation Engine

The Correlation Engine derives insight from the store and graph:

Deduplication: identical findings collapsed to one canonical finding with occurrences.
Chaining: evidence links (e.g., subdomain → IP → service → CVE → payload → finding) form attack paths.
Aggregation: a vulnerability class across many hosts → one reportable issue with affected assets list.
Enrichment: joins CVE → CWE → CAPEC → MITRE → EPSS automatically.
Risk aggregation: per-asset, per-domain, and mission-level risk rollup using the risk model.

5.12 Reporting Engine

Generates all output artifacts. See 21 - Reporting Standards.md.

Renderers: JSON, Markdown, HTML, PDF, SARIF, (future: docx, csv).
Views: technical report, executive summary, evidence package, compliance mapping (PCI-DSS, ISO 27001, OWASP Top 10, MITRE), timeline.
Template system: versioned report templates; deterministic rendering.
Evidence packaging: bundles screenshots, raw responses, and logs into a checksummed archive.

5.13 CLI

The CLI is a first-class interface. See 19 - CLI Standards.md.

Command groups: mission, workflow, scan, report, tool, plugin, knowledge, config, secret, api, admin, doctor, completion.
Deterministic, machine-parseable output (--output json|csv|yaml|text).
Rich terminal UX with non-TTY fallback (no colors/ANSI when piped).

5.14 REST API

The REST API exposes the platform programmatically. See 20 - REST API Standards.md.

Framework: FastAPI; OpenAPI 3.1 schema; pydantic v2 models shared with the schema.
Auth: API keys, OAuth2/OIDC, mTLS, or local (RBAC enforced at the API boundary).
Async job model: long operations return 202 + job_id; status polled via /jobs/{id}.
Pagination, filtering, sorting, HATEOAS-lite envelope.

5.15 Target Intelligence Database (TIDB)

The Target Intelligence Database is the SQL core store. See 09 - Database Design.md.

Stores all normalized entities with full history/versioning (bitemporal).
Indexed for search (FTS) and graph adjacency.
Written by the Normalizer; read by Correlation, Reporting, and API.

5.16 Cache

Purpose: AI responses, DNS lookups, tool outputs (within retention), plan expansion, HTTP responses, WHOIS.
Backend: Redis in distributed mode; in-process TTL cache in single-node mode.
Keys: versioned, namespaced, TTL-aware; cache invalidation on schema or knowledge-version bump.
Cacheability contract: only idempotent, side-effect-free operations are cached; destructive or state-changing operations bypass cache.

5.17 Scheduler

Backend: APScheduler (or equivalent) for cron-like and interval missions.
Enqueues missions into the Queue; supports time windows, rate limits, and scope calendars (e.g., only within approved testing windows).
Produces the continuous mission type for scheduled delta scanning.

5.18 Queue

Purpose: decoupling between delivery/application and execution.
Backend: RabbitMQ/AMQP (distributed) or in-process async queue (single-node).
Messages: typed, schema-validated, durable, DLQ (dead-letter queue) on failure.
Guarantees: at-least-once with idempotent consumers.

5.19 Logging

Structured JSON logging to stdout and rotating file sinks. See 18 - Logging Standards.md.
Fields: ts, level, logger, correlation_id, span_id, actor, module, message, context, event_type.
Sensitive data is masked at the boundary; secrets never logged.

5.20 Configuration

Layered: defaults → file (YAML) → environment → CLI flags → remote (envs).
Schema-validated (hunterx.yaml), hot-reloadable, deterministic merge order.
See 04 - Coding Standards.md §Config and docs/configuration.md (project docs).

5.21 Secrets

Backend-agnostic secret store interface (SecretPort): local encrypted store, Vault, or cloud KMS.
Secrets never appear in: logs, reports, AI prompts, config dumps, or errors.
Rotation-aware; scoped per mission/tool with least-privilege delivery.
See 13 - Security Standards.md.

5.22 Telemetry

OpenTelemetry-based: traces (spans per workflow step, AI call, tool run), metrics (counters, histograms, gauges), and logs correlated by trace ID.
Exporters: OTLP, Prometheus, or local console in air-gapped mode.
See 18 - Logging Standards.md.

5.23 Event Bus

Typed pub/sub with wildcard subscriptions and historical replay.
Event categories: mission.*, workflow.*, task.*, tool.*, finding.*, plan.*, ai.*, plugin.*, system.*, audit.*.
Events are the primary mechanism for cross-subsystem integration (loose coupling).
Every event carries correlation_id, actor, ts, payload_schema_version.

6. Key Runtime Flows

6.1 Mission Lifecycle Flow

Operator submits target+profile
  → MissionService validates scope & legality
  → MissionEngine creates Mission (state: DRAFT)
  → Planner expands mission profile into Plan DAG (state: PLANNED)
  → Operator approval gate (if required by profile) (state: APPROVED)
  → WorkflowEngine executes plan steps (state: RUNNING)
        per step:
          tool run → parser → normalizer → store → correlate → update graph
          AI calls for planning/triage/correlation (through AI Engine)
  → Validation phase (proof-of-exploit / false-positive filter) (state: VALIDATED)
  → Correlation & risk aggregation (state: CORRELATED)
  → ReportingEngine generates artifacts (state: COMPLETED)
  → Archive with retention policy (state: ARCHIVED)

6.2 Tool Execution Flow

Workflow step requests tool execution
  → ToolExecutor resolves adapter + knowledge file + sandbox policy
  → Sandbox prepares: scope envelope, time limit, resource limits, env secrets
  → Adapter.execute() runs tool (process or remote worker)
  → ParserEngine parses output (streamed)
  → Normalizer maps to canonical entities
  → CanonicalEvents written to TIDB (transactional, versioned)
  → Events emitted (tool.completed, finding.created)
  → Correlation Engine triggered on new findings

6.3 AI Reasoning Flow

Goal emitted by agent/planner
  → GoalAdapter → structured AITask
  → PromptManager builds prompt (context from KG subgraph + knowledge base)
  → ProviderRouter selects provider/model
  → Middleware (masking, validation, logging)
  → Provider executes (with cache check first)
  → OutputValidator validates schema
  → ConsensusEngine aggregates samples
  → ConfidenceScorer scores confidence
  → Validated ReasoningResult + provenance returned to requester

7. Deployment Topologies

Topology	Use case	Components
Single node	Personal, small team	Everything in one process/container
Single node + workers	Team scanning	API/UI process + N worker processes
Distributed	Enterprise	API nodes, scheduler, worker fleet, shared Redis/Queue/DB/Graph
Air-gapped	Sensitive environments	All services local; AI local (Ollama); mirrors of external datasets

All topologies share the same codebase and configuration schema; only the deployment profile differs.

8. Architectural Decision Records (Summary)

ADR	Decision
ADR-001	Clean Architecture with ports & adapters; no framework imports in domain
ADR-002	PostgreSQL as primary TIDB; Neo4j for Knowledge Graph; object store for evidence
ADR-003	Async-first execution (asyncio) with process isolation for untrusted tool runs
ADR-004	AI only through provider abstraction; agents never call LLMs directly
ADR-005	YAML knowledge files as single source of tool truth
ADR-006	Event-driven integration via typed Event Bus + Message Bus
ADR-007	Plan IDs and finding hashes are deterministic (content-addressed)
ADR-008	Long operations are async jobs; API returns `202 + job_id`
ADR-009	Plugins execute in isolated sandboxes with capability-based permissions
ADR-010	Secrets isolated in vault-backed secret store; masked at all boundaries

9. Cross-Cutting Concerns

Security → 13 - Security Standards.md
Performance → 14 - Performance Standards.md
Observability → 18 - Logging Standards.md
Error handling → 17 - Error Handling Standards.md
Testing → 15 - Testing Standards.md

10. References

01 - Vision.md — mission, scope, philosophy
03 - Folder Structure.md — repository layout
05 - Plugin SDK Specification.md — plugin contracts
06 - Tool Adapter SDK.md — adapter contracts
09 - Database Design.md — TIDB design
10 - Workflow Engine.md — DAG execution
25 - Future Expansion.md — growth path