Verification

How Stackurity's claims are tested, written in answer to an external review that asked for proof rather than documentation. Every row names the tests that run in CI; the "Found" column lists real defects the tests exposed, each fixed in its own commit in 0.4.0.

Concern	How it is verified	Found and fixed
Background work survives crashes: feed refresh, inventory upload, notification delivery, periodic tasks and jobs	internal/crashtest (CI job crash) starts the real server, kills it with SIGKILL at each of these moments and restarts it. Asserts previous data is kept, work resumes or fails visibly, findings are never duplicated, and notifications arrive at least once with a delivery id to deduplicate. See reliability.md	3: jobs killed on every attempt were retried for ever; interrupted feed syncs showed no error; a crash could delay new findings by a day
Several replicas	internal/worker and internal/store tests race replicas for periodic tasks, deliveries and rollout releases	1: two rollout loops could exceed max_concurrent (Pro)
Vulnerability matching is correct per ecosystem	TestCorpus in internal/intel: 181 cases on 74 real OSV advisories and 6 real Amazon Linux updates (internal/intel/testdata/corpus, sources cited) across 18 ecosystems, through the whole sync and evaluation pipeline: affected, fixed, backports, epochs, pre-releases, last_affected, withdrawn advisories, unparseable versions	12, including Rocky Linux packages never matched, RHEL 10 matching nothing, Ubuntu FIPS channels misapplied, Drupal modules never matched
Missing data never looks healthy	TestCoverageInvariants: 16 scenarios (feed never synced, failed, stale, disabled; collection failed, partial, unsupported; stale asset; unparseable version; runtime outside a package manager; never evaluated) each checked through the store and every API that summarises coverage	Releases a feed does not cover (a new Debian or Alpine) and some unparseable versions were shown as clean
Risk priorities are sound	internal/risk/scenarios_test.go: 36 reviewed scenarios pinning priority, SOS rule, every factor and its evidence; property tests for monotonicity, missing evidence, explanations adding up and reproducibility. See risk.md	6, including a saved SOS rule silently inheriting conditions from the default rule, and a missing environment scoring like non-production
Policy changes are auditable and reproducible	Each finding records the policy version that produced it; policy changes are audited with the full policy before and after in the same transaction; tests re-score and compare	Audit was written outside the transaction and held only the new policy
Workspace isolation	TestEveryRouteIsClassified walks the router (149 routes in the paid edition) and fails on any unclassified route; TestIsolationSweep tries workspace A's objects with every kind of workspace B credential (each role's session, admin, collector and host-bound agent tokens), including guessed ids, exports, reports, search, audit and jobs. Revoked, expired and downgraded credentials are tested	4, including a workspace admin able to disable shared feeds for everyone, a workspace admin able to reset an instance admin's password, and cross-workspace references through foreign keys
The loop: how you know it is fixed, and ignoring with care (0.7)	internal/store/loop_test.go: regressed detection (the same finding back after a removal, a new version after a fix) and what is not a regression (an upgrade that stays vulnerable, in either order of the matcher's steps; an intelligence change); snooze until a fix ships returning on a free fix only, with Home's hidden counts still counting it; ignore expiry, limits, and scope everywhere including assets and findings found later; false positives kept rejected by the matcher; check-again lifecycle (request, agent pickup, report, evaluation) with each outcome (passed, partly fixed, still open, expired); "I applied this" resolving nothing. internal/api/loop_test.go: who may ignore, ask and decide per role, the inbox per role, audit entries, the agent poll and server-side repository scans. internal/notify: issues that came back are urgent once and summarised. The isolation sweep covers every new route and object	An ignore request "until a fix ships" for issues that already have a fix was accepted and silently applied to none
Project isolation	TestEveryRouteIsClassified also requires every route (162 in the paid edition) to declare its project scope, matching the router, which refuses unlisted routes to members with project access only. TestProjectIsolationSweep sends every route (2,720 requests in the paid edition) as 17 members of Project A and 6 organisation viewers with a role on it: each role by session, API token and with Project B chosen in the switcher, through a team, through a single-asset grant, after a revoked grant, through an archived project, and as an admin downgraded to project access; they must never see, count, change or delete Project B's objects (assets, findings, fix groups, components, incidents, search, exports, reports, audit, jobs, guessed ids), organisation routes must answer 403, and B's data is compared before and after. TestProjectAccessRules: organisation-wide roles see everything, the switcher narrows them, an asset shared by two projects shows in both, an incident spanning projects shows each member only their part, roles from grants act per asset, downgraded grants and leaving a team take effect at once, agent and collector tokens keep working. TestProjectScopeInDatabase checks every table with asset_id and raw reads and writes on a narrowed connection; TestProjectMCP runs every assistant tool with a member's token; TestProjectRoute checks notifications limited to a project, including issues that came back, assets connected outside it and overdue issues, and TestProjectRouteWithholdsOtherAssets the check every such message passes. The sweep seeds Beta's people, team and Jira link where Alpha can reach them (a grant on the shared database, an assignment, the incident spanning both), reads unfiltered lists and those objects, and also runs as organisation viewers with a role on Alpha; TestProjectPeopleAndLinks covers assignment, the directory, tracker links, narrowed incident filters and asset merges. The 0.7 routes (checks, ignore rules, the approval inbox) are in the sweep with Beta's check, ignore request and ignore rule, and TestProjectAccessRules checks that a project member cannot ignore on every asset, see or end a rule for every asset that covers none of their findings, or see or decide another project's requests. Removing the narrowing from findings alone makes 14 checks fail	Archived projects' names stayed readable by their former members; the unresolved identity-conflict count on the overview counted other projects' assets; members' lists took 0.2 to 0.3 seconds until the scope functions were evaluated once per query (see projects.md). An independent review then found, and these tests now cover: grants on a shared asset audited with the person's email under the asset; assignment to any user id revealing emails in the history; the full directory for organisation viewers with a project role; Jira links and stored priority and escalation of incidents spanning projects
Onboarding: invitations and a quiet first scan (0.9)	internal/api/onboarding_test.go: TestInvitationSecurity checks that only organisation admins invite, invalid emails, roles, projects and unknown fields are refused, the token is nowhere in the database (only its SHA-256), the list never shows a link, wrong and altered tokens get the same 410, a request cannot choose the role, organisation, project or email (400), weak passwords are refused without using the link, accepting creates the account with project access only and the invitation's role, a link works once, expired, withdrawn and replaced links stop working, a member is granted at once, and every step is audited. TestInvitationExistingAccount: an existing account must be signed in as itself (409 otherwise), API tokens cannot accept, an admin of another organisation joins with the invitation's role only and keeps everything elsewhere, and a member's organisation role and higher project grant are never lowered or raised. TestInvitationRateLimits: 10 link attempts a minute per address, 30 invitations an hour per organisation. TestInvitationEmail: the link goes through the email channel's SMTP server to the invitee only. TestOnboardingFlow, TestAssetsLandInProject: the flow's state and steps, and assets added through a project's token, enrollment code or SBOM upload landing in it. internal/notify/quiet_test.go: TestQuietFirstScanEveryPath connects an asset whose backlog is known exploited and 60 days past every deadline, with a webhook route and a Jira channel, and checks one connected message and nothing else: no urgent message, no Jira issue, no escalation candidate or record, no overdue, newly overdue or not-acknowledged counts, no "new issues" in the summary; a late part of the backlog stays quiet; a new issue afterwards alerts, creates a Jira issue and escalates. TestQuietBacklogOnlyForNewAssets, TestRegressedOnNewAssetIsQuiet. actions_test.go: Assign to me, Snooze and Open on every fix, Slack buttons inside the block limit, email links, and links opened in the route's project. The isolation sweeps cover the new routes (170 in the paid edition; 2,822 requests by project members) and invitations	Jira issues were created for a new asset's backlog (the quiet check ran after them); a new asset's backlog was escalated and reported overdue once its deadlines passed.
Why each finding matched (1.0)	The matcher stores the rule that produced each finding when it makes it (evidence.match: rule, deciding record, SEMVER range, distro backport, normalised version, ambiguity), so the explanation is the rule that really matched. TestCorpus checks every corpus finding names its rule and a record among its sources; internal/why tests pin the sentence for package advisories, OS packages, container images and end of life, every rule, missing information for each state, and findings stored before 1.0 (said so, never guessed); TestFindingWhyAndSecondCheck checks GET /findings/{id}, explain and the second check through the API
Independent second check (1.0)	internal/secondcheck re-derives findings from the raw stored record and the raw observed version with its own purl parser, release rules, comparators (dotted numbers, dpkg, rpm, apk) and range reading, importing nothing from the matcher. TestSecondCheckDisagrees pins deliberate disagreements: a corrupted range, events in the wrong order, a package identity mismatch, a record for another release, a distro backport the upstream range calls vulnerable, a different fixed version, a subscription mismatch. TestSecondCheckCouldNotCheck pins what it does not read (pre-releases, SUSE, FIPS builds, GIT-only ranges). TestCorpus runs it on every corpus finding: 156 findings, 96 agreed, 60 could not check, 0 disagreements. TestSecondCheckCatchesACorruptedRecord corrupts a stored record after a real match and sees the disagreement shown on the finding while its verification stays confirmed; TestSecondCheckDisagreementKeepsUrgentAlerts (notify) checks a known exploited P1 with SOS stays P1 and SOS and still raises the urgent alert. internal/store/secondcheck_test.go: a disagreement holds through re-evaluation of the same match and is dropped when the match changes, a stale result is discarded, a person's false positive stays rejected
Scan jobs within limits; their output untrusted (1.0)	TestCloneSizeLimit, TestImageScanSizeLimit (a clone or image growing past its disk limit is stopped, with its process group), TestWalkLimits, TestWalkStopsWithItsContext, TestOversizedLockfile, TestCoverageIsCapped (lockfile walks), TestIngestLimits (64 MiB on the wire, a gzip bomb, NUL bytes through the API), TestUntrustedOutputIsCleaned, TestUntrustedIdentityIsRefused, TestUnsafeLink (every batch), TestLongEvidenceIsClipped (SBOM), TestHostilePluginMetadata. See security-audit.md	One NUL byte in any collector's output failed the whole ingest; a lockfile tree with over 1,000 unsupported files was refused as a whole; one over-long SBOM supplier refused the document; a cancelled repository scan kept walking; killing git or Syft at their deadline left their child processes running
SSRF and network access	Tests for every outbound path (Git, webhooks, Slack, Teams, Jira, SMTP, registries, image scans, Kubernetes): loopback, private, link-local, CGNAT, IPv6 forms, metadata addresses, encoded IPv4, redirects and DNS rebinding	3: zoned IPv6 link-local addresses, encoded IPv4 in Git URLs, DNS rebinding through image scans
Malicious input	Fuzz targets (seeds run in every build, long runs in the fuzz workflow) for SBOMs, every lockfile parser, OSV records, plugin output and Amazon updateinfo; bounded memory and time asserted	3: output caps never applied; very long versions stalling evaluation; one bad entry voiding a whole scan
Secrets never leak	Log, error and audit capture tests for tokens, passwords, webhook URLs and credentials in Git URLs	Slack and Teams URLs in delivery errors, typed passwords in the audit log, credentials echoed from malformed URLs
Remediation lifecycle (Pro)	End-to-end tests with a simulated agent using the real client and fixer: happy path closes the finding only on fresh inventory; revocation before and during dispatch; approval expiry; duplicates (double approval, two replicas, repeated fetches and reports); agent unreachable; lost reports; failed health checks; window closing; policy disabled mid-rollout. Package-manager dry runs tested in real containers (internal/agentfix/testdata/pkgmanagers.sh) for apt, dnf, yum and zypper	7, including a lost report causing a second change on the same host, and in-flight changes never verified after a revoke
Releases, upgrades and recovery	release-checks (on every pull request and required before a release): installs each supported earlier release (the from: list in .github/workflows/release-checks.yml; 0.2.0, 0.3.0 and 0.3.1 for 0.4.0) with data, upgrades to the new build on the same database, restores a backup into a fresh PostgreSQL, and rolls back to the previous image; releases are signed (Sigstore), carry provenance and an SBOM, and take their notes from CHANGELOG.md	A restore into a new PostgreSQL left the isolation role missing (fixed in 0.2.0)

What tests cannot prove: that there is no telemetry and that inspected code is never executed are design properties; security-audit.md explains how the code keeps them and section 4 there maps every stated security control to its tests.

Evidence states

This is the one authoritative definition of a finding's verification. risk.md uses it for scoring, the API returns it on every finding, and the dashboard and the help article "How Stackurity knows" say the same in plain words.

State	Definition	Set by	What resolves it
confirmed	The installed version was read from an authoritative local source (package database, lockfile; inventory confidence confirmed), the asset is reporting, and the version is in the advisory's affected range under the ecosystem's version rules (or its release line has ended). No standing second-check disagreement.	The matcher	Nothing: act on it.
needs_validation	The match cannot be relied on yet because the installed version was inferred or has low or unknown confidence. Severity is withheld (unknown); the score is scaled by 0.7 and the finding is never SOS.	The matcher	why.missing_information says exactly what: a confirmed version, for example from the agent.
stale	The asset has stopped reporting (no report within the stale period, 48 hours by default). The finding was true when the asset was last seen. Scored × 0.8, never SOS.	The matcher, from the asset's collection status	A fresh report: Check again, or get the agent reporting.
rejected	A person marked the finding a false positive with an ignore rule. The matcher keeps it rejected while the rule stands, and keeps its own verdict in verification_before, restored on reopening.	A person	Reopen the finding or end the rule.

A second-check disagreement is not a state. It never changes verification, so it never lowers the score or priority, never removes SOS and never keeps a finding out of urgent alerts: the second check is deliberately simple, and a defect in it must fail toward alerting. The disagreement is shown instead: the finding page and lists say "Needs validation: the second check disagreed (what differs)", why.needs_review is true and why.missing_information asks a person to compare the version with the advisory record, the finding carries second_check: "disagreed" and second_check_reason, and Home counts open disagreements to review (second_check_disagreed in GET /findings/summary). This was the simpler of the two designs considered (the other changed verification and exempted disagreements from the risk factors).

What none of them means: a confirmed finding is a confirmed advisory match. It says the installed version is in the advisory's affected range. It is not a confirmed exploit and does not show the issue can be exploited on that asset (the code may be unused or unreachable). "Known exploited" means the issue is in CISA's catalogue of vulnerabilities exploited somewhere, not that this asset was attacked.

Why it matched

Every finding carries why (in GET /findings/{id} and GET /findings/{id}/explain, and in the explain_finding tool), built by internal/why from what the matcher stored at evaluation time. It is never a re-derivation:

• sentence: one sentence, for example "Installed 6.10.0 (from web/package-lock.json, seen 2 h ago) is in the affected range < 7.0.7 of GHSA-xxxx (osv:npm, refreshed 3 h ago), compared with SemVer rules; fixed in 7.0.7."
• A common core: type (package_advisory, os_package, container_image, end_of_life), installed (version, the version compared when it was normalised, where it was read, collector, confidence, when last observed), source (deciding record, ecosystem, feed, last successful refresh, stale), rule (kind, range, comparator, text, and why it was marked ambiguous), fixed_version, evidence_state with evidence_state_text, missing_information, second_check and second_check_text.
• Type-specific detail: package (ecosystem, name, direct, location), os_package (release, source package, backport, subscription), container_image (image and the inner type), end_of_life (product, cycle, end date, latest in cycle). New evidence types add their own block beside the core.

The stored part is compact (evidence.match, about 60 bytes per finding): rule (range, last_affected, no_fix_listed, listed_version; for end of life exact_cycle, version_prefix, package_name, windows_build), record, and only when they apply range_type (SEMVER), backport, normalised and ambiguous (some_records_unreadable, sources_disagree, version_normalised, semver_range). Observation times and feed refresh times are read when the explanation is built, so evidence does not change, and findings are not rewritten, on every report. Findings stored before 1.0 have no match until they are next evaluated; their explanation says the rule was not recorded.

Second check

High-impact findings are checked a second time by internal/secondcheck, a separate and deliberately simple code path:

• Which: open vulnerability and end-of-life findings that are P1, SOS, known exploited, regressed, needs_validation, or that the matcher marked ambiguous (evidence.match.ambiguous). A partial index (findings_second_check, migration 0034) keeps the selection cheap.
• How it differs from the matcher: it imports nothing from internal/intel, internal/osv or internal/version. It parses the package URL with its own parser, derives the ecosystem, release and acceptable package names (binary, source package, source RPM, the packages of a kernel, and the kernel source a Debian or Ubuntu signed kernel wrapper such as linux-signed-aws-5.15 or linux-signed-amd64 wraps; metapackages are not mapped) itself, reads the stored affected entries from advisory_packages for the finding's group (the ranges and listed versions as the feed published them; the package is in the row's own columns, and severities and other feed-specific fields are not kept there because nothing reads them from it), and compares versions with its own comparators: plain dotted numbers for language ecosystems (pre-releases and words refused), dpkg, rpm (epoch, tilde, caret) and the common Alpine forms. It reads ranges in the order the record lists them, as introduced..fixed spans, instead of sorting events. Paid channels are used only when no free record applies.
• Results: agreed; disagreed with each difference (affected, fixed_version, package, subscription, release_line, end_of_life: what the matcher says, what the second check finds, and a note such as "package identity mismatch" or a distro backport); or could_not_check with the reason (a pre-release, SUSE product channels, Ubuntu FIPS builds, a GIT-only range, no stored record, a Windows Azure Edition host or a Windows build outside its own table of update lines). Windows missing-update findings are re-derived from the host's build with its own reading of the update line. It reads less than the matcher on purpose: what it cannot read is never a disagreement.
• Effect: results are stored apart from detection, in second_checks, against a hash of the match they checked. A disagreement leaves the finding's verification, priority, SOS and alerts unchanged; it adds a second_check_disagreed event, is shown on the finding, in lists and on Home for review, and is logged with the raw inputs in second_check_disagreements and the server log, for new corpus cases. When the matcher's claim changes, the check is dropped and runs again; a result for an older claim is discarded. Each result also stores the version of the second check's own rules (independent.checker, store.SecondCheckRules); results made under older rules are redone, so fixing a rule clears the disagreements it caused.
• Cost: a job (second.check), never work inside the periodic scheduler, whose few slots a long check must not hold: every minute a periodic task asks, within 5 seconds, whether any finding of the workspace is waiting and queues one job for it (deduplicated); the job checks at most 500 findings, only findings without a current check (a check is redone after 7 days). Evaluation never waits for it, so an urgent alert can precede the check by about a minute. TestSecondCheckPerformance (STACKURITY_PERF_FINDINGS=130000 go test ./internal/secondcheck -run TestSecondCheckPerformance -v) seeds 130,000 open findings on 2,000 hosts (33,897 of them covered) with a stored advisory record for each issue, and on the development machine (PostgreSQL 17 in Docker, median of 5) measured: choosing a batch 126 ms, a whole run of 500 findings (read, checked and recorded) 0.93 s, the Coverage summary 36 ms, re-evaluating one host of 65 checked findings 65 ms. A first pass over 33,897 covered findings takes about 70 one-minute runs; after that only changed findings are checked.
• Where it shows: the finding page, Coverage (counts and the latest disagreements), GET /findings?second_check=, GET /second-checks, and the metrics stackurity_second_check_findings{result} and stackurity_second_check_disagreements_total.