Public transparency report — process

A quarterly aggregate report that BlackLake publishes at https://www.blacklake.systems/transparency. Anonymised, aggregated, verifiable. This doc is the engineering process for assembling the report — what numbers go in, where they come from, how they're anonymised, who reviews before publish.

Purpose#

Buyers and regulators want to know whether BlackLake actually does what it claims. The transparency report is a quarterly receipt: aggregate metrics on governed evaluations, deny rates, approval medians, audit events, retention, and cross-tenant isolation, plus the count of any incidents in the quarter and how they were resolved.

Same data the BlackLake team uses internally; we just publish the aggregates.

Cadence#

Quarterly — first Friday after quarter end. Q1 publishes April, Q2 July, Q3 October, Q4 January.
Annual roll-up alongside the Q4 report.
Mid-quarter updates only for material incidents (RTO breach, customer-data exposure) — those land within 30 days and are tagged on the page.

What goes in#

A quarter's report carries six sections:

1. Volume#

Metric	Source
Total governed evaluations	`count(*) FROM policy_evaluations WHERE evaluated_at IN <quarter>`
By decision (allow / deny / approval_required / default_deny)	`groupBy(outcome)`
By capture path (mcp / sdk / ci / shell / cloud_audit / existing_workflow_engine / depth / manual)	`groupBy(source)` from joined agents
Total approvals created → decided	`approvals` table per status
Total receipts issued	same as evaluations (every eval gets one)

2. Deny rate#

Aggregate deny + default_deny percentage of total evaluations.
Top 5 denial reason categories (BL-DIF-14 denial_reason): policy / budget / agent_suspended / binding_missing / default_deny.

3. Approval medians#

p50 / p90 / p99 time-to-decide.
Approval acceptance rate (approved / (approved + rejected)).
Override rate (break-glass approvals as % of total).

4. Audit retention + archive#

Audit rows archived in the quarter (BL-OPS-4 — count from audit_archive_runs).
Default retention (90 days unless org plan extends).
Customer-initiated deletions (org soft-deletes that completed past the 30-day grace).

5. Cross-tenant isolation#

Number of cross-tenant isolation tests passing (apps/api/src/tests/).
Any incidents where org A data appeared in an org B response (target: zero; if non-zero, link to incident postmortem).

6. Incidents#

Public count of customer-impacting incidents in the quarter.
For each: severity (S0 / S1 / S2), duration, RCA link, resolution.
S0 / S1 always reported individually with timeline + remediation.

Anonymisation rules#

The report is aggregate-only. Specifically:

No org names, no user emails, no key suffixes, no IP addresses.
No tool names that could fingerprint a customer (e.g. their internal acme.deploy_prod tool stays out of the public roll-up). Top tools shown by category (e.g. "deployment", "messaging") not by literal name.
Cells with < 5 customers are suppressed to prevent re-identification, replaced with "<5".
Median + percentile metrics are computed across the whole customer base, not per-customer.

The query that assembles each metric runs in a read-only role with suppression at the SQL layer; the published markdown never carries raw rows.

Source-of-truth scripts#

scripts/transparency-quarter.mjs (planned) — runs the queries, writes a draft markdown to site/content/transparency/<year>-Q<n>.md for review.
Fields in the script match the metric table above; if a metric is added to one and not the other, that's a bug.

Until the script lands, the queries live in the BL Operator Notion runbook at https://blacklake.notion.site/transparency-quarterly. The on-call SRE runs them manually first Friday after quarter end.

Review chain#

SRE / data lead runs the queries; confirms no cell shows < 5.
Eng lead reviews the draft markdown — confirms no leaked customer identifiers.
Legal / security reviews the incidents section.
CEO / signing officer approves before publish.

A report can't publish without all four sign-offs. Stored in the transparency-archive private repo as a co-signed PR for audit.

Customer-specific reports#

Customers on Production tier can request their own transparency extract — same shape, scoped to their workspace, signed by BlackLake. Process:

Customer email to transparency@blacklake.systems.
SRE runs the per-org query (no suppression, since customer is asking about their own data).
Output is signed (HMAC, key in Vault), delivered to the customer's primary contact.

Free / Team customers can run the queries themselves against their own workspace via GET /v1/insights/coverage, /v1/audit/events, and the related endpoints.

When this doc is wrong#

If the public report drifts from the metrics here, or the suppression rules failed, that's a process bug. Open an issue with the transparency label.