Architecture

System design

Scheduler polls customer tenants, Worker drains four queues, API serves this console. Messages carry row IDs, never payloads; Azure PostgreSQL holds all state, so any replica can process any stage and every stage is idempotent.

Auth to customer tenants is certificate-based through a single multi-tenant Entra app — no client secrets. A tenant that fails auth three times is auto-suspended so a poison tenant never blocks the polling loop for others. Console sign-in is Entra as well: MSSP staff (admin / analyst) authenticate against the MSSP tenant, invited customer users against their own — scoped to their customer's incidents only.

Tenant onboarding

A customer joins by admin consent, not credentials: the console generates a consent URL for their global admin plus an az command granting workspace RBAC. The landing page is passive — activation only happens after authenticated verify probes prove Graph, KQL, and ARM access.

Pipeline workflow

ingest → enrich → triage → writeback. Each stage does its work in a DB session, then enqueues the next stage. Failures abandon the message; Service Bus redelivers up to 5 times, then dead-letters (requeue from Monitoring).

The ingest material-change gate re-triages only when the incident meaningfully changed, breaking the triage → write-back → update feedback loop. Incidents the customer already resolved on their side are scored as outcomes and not re-triaged.

Triage

Two engines behind one interface, selected per tenant: single_shot sends one structured prompt; agentic runs a tool-using loop (KQL, enrichment lookups) and records every step. Both must argue competing hypotheses with likelihoods, not just a verdict.

Per-tenant context notes — the customer's environment quirks, expected admin behavior, known-benign patterns — are console-editable and injected into every triage prompt. Model, tokens, and cost are recorded per call; when a tenant's daily LLM budget is exhausted, incidents queue as parked and flagged waiting for review — never dropped — while ingestion continues.

Data model

SQLAlchemy 2.0 on Postgres. Incidents upsert on (tenant, graph incident id); enrichments upsert on (provider, indicator, tenant); write-back comments carry a fingerprint so retries detect their own prior comment.

Enrichment

Providers implement supports() + enrich(); the registry builds the active list from configured API keys. External TI results are cached globally, Microsoft-native and KQL results per tenant — a hard multi-tenancy invariant.

GeoIP runs as a KQL query (geo_info_from_ip_address) in whichever tenant's workspace triggered it, but since it is a pure function of the IP the result is cached globally. Private and reserved IPs skip external lookups entirely. Provider failures are stored as error-marked rows with a short TTL and triage proceeds with partial data. KQL enrichments are configuration: versioned YAML packs keyed by entity type, one Log Analytics query per pack entry.

Closing policy and feedback

AI triage may close an incident on its own only when the verdict is benign and confidence clears the console-configured threshold. Everything else waits for an analyst — and every human disposition is recorded as ground truth against the AI verdict.

verdict_outcomes is append-only: analyst confirms and overturns, plus closes the customer performed in their own portal, each land as an agree/disagree record against the triage verdict. The calibration panel on the operational health page reads these to show where confidence is over- or under-stated.

Observability

One trace per incident: the scheduler starts it, the traceparent rides every queue message, and worker stages continue it. LLM calls emit GenAI-convention spans with token and cost attributes; exports go to App Insights.