Before you let the AI close alerts on its own
You’re asking Which alert types are safe to hand over?
You get An autonomy map per alert type, with the evidence for each call and what would change it.
CyberSOC · know more
The detail behind the brief.
The research behind the numbers, how we handle your data, what regulators are asking, and answers to common questions.
Behind the numbers
Once it’s set up, an AI triage agent can close the alerts it judges harmless. Microsoft’s phishing triage agent, for example, resolves the emails it judges non-malicious, so analysts don’t spend time reviewing them.2
Only one in five security leaders leans on the vendor’s number.3 Re-checking everything is prudent, but it eats the time the AI was meant to save. And re-checking with the AI’s verdict on screen catches far fewer of its misses.
Why “it agrees with our analysts” isn’t enough
The proof most often offered for an AI analyst is how often it agrees with a human one. Here are two published figures.
99.7%
How often an AI agent agreed with human analysts in one vendor’s customer trials, across 138,475 alerts.4
67%
How often analysts agreed with an AI agent’s verdict in a 14-month study at a university SOC: 72 of 108 tickets. Of the seven outright clashes, only one was a clear AI error. The other six were judgment calls.5
Both can be true. Different AIs, different alerts and, above all, different people to agree with.
Analysts disagree with each other, and anyone can call the same alert differently on a different day. So “agreement with an analyst” is a ruler that moves.
We fix the ruler first. Where outcomes exist, we use them. Where they don’t, we use your team’s steady judgment.
That’s how your team decides on a good day. We call it the house view. Then we measure the AI against that.
A wrong “benign” can miss a deadline
Regulated firms run on reporting clocks:
- Six hours in India, under CERT-In7 and RBI8 rules.
- Four hours in the EU under DORA once a major incident is classified, and no later than 24 hours after you become aware of it.9
- One hour for firms regulated by MAS in Singapore.10
If the AI closes a real incident as harmless, nobody knows the clock should be running.
So for regulated teams we add one more number: reporting-clock exposure. For each alert type, it shows how often the AI clears alerts your team would have escalated as a possible reportable incident.
0h
2h
3h
5h
1 hourMAS · Singapore
4 hoursDORA · EU
6 hoursCERT-In and RBI · India
When you’ll want the evidence
Ten moments when you need to know how far to trust the AI. Find yours.
Decide
Before you switch an AI on
You’re asking How will we know if it helped?
You get A baseline of how your team decides today, so you can judge any AI from day one.
At provider renewal
You’re asking Is our provider’s AI good enough to renew?
You get If the AI runs in your tenant, a report on it. If it runs on the provider’s platform, the clause to ask for a per-alert decision export.
Prove
When the board, an auditor or your regulator asks
You’re asking How do we know the AI works?
You get An independent evidence pack for risk and audit teams: accuracy against outcomes, consistency, drift and override reasons, per alert type.
When your reporting clock is short
You’re asking Could the AI be hiding a reportable incident?
You get Reporting-clock exposure per alert type, against your regulator’s clock.
When a client asks how your AI performs (for MDR and MSSP providers)
You’re asking How do we prove our AI to each client?
You get An independent report for each client, delivered to the client unedited.
Run
When everyone re-checks every verdict
You’re asking Where can we stop?
You get The alert types where the evidence supports spot-checks instead of full review, and the ones where it doesn’t.
After a model update, a new detection rule or a new attack wave
You’re asking Did anything change?
You get A drift check against your frozen baseline that says what moved: the AI, your team or the alerts.
When analysts keep overruling the AI
You’re asking Who’s right, and what should the AI learn?
You get The reasons, sorted into six causes, and a teach list for the AI.
When an incident traces back to an alert the AI closed
You’re asking A one-off, or a pattern?
You get How often the AI makes that call, on which alert types, and how your team would have called it.
See which of these fit your SOC.
Book a 30-minute callBuilt for the people who sign for it
CISO
You’re accountable for AI you didn’t choose. Get evidence you can sign off on.
Six questions →Head of SOC
Know where to trust the AI, and where to keep your people.
When you’ll want the evidence →Model risk, compliance and audit
Independent, outcome-based testing of vendor AI, with monitoring records you can file.
What your regulator is starting to ask →MDR and MSSP providers
Show each client how your AI performs, in a report they receive unedited.
Provider licence →What we touch, and what we don’t
We do
- Read decision records, read-only.
- Run inside your cloud, in your region. Nothing leaves it except the reports you choose to share.
- Scramble (hash) analyst and user IDs with a key you hold.
- Report at team level only.
- Recommend where the AI may act alone. You decide.
We never
- Write to your systems, or change an alert, a rule or a setting.
- Read email bodies, attachments, files or packet captures.
- Score, rank or report on individual analysts. We judge agents, not analysts.
- Change anything in your SOC without your sign-off.
- Let a vendor edit our findings.
Leaving: Revoke our access and we stop reading.
Independent by design
An AI shouldn’t be graded only by the company that sells it.
- We don’t build, sell or resell SOC agents.
- No vendor can edit our findings.
- Our fee never depends on the result.
- When a provider pays for a report on its AI, its client gets the report unedited.
- Every finding says what it rests on, and what it doesn’t mean.
What your regulator is starting to ask
Today, the hard deadlines are for reporting incidents. On AI, most rules are new, in draft, or start later. But they all point the same way: you stay accountable for the AI you use, including AI from a vendor.
India
- Draft
- In force
- RBI’s draft guidance on model risk management (24 June 2026) covers third-party and AI models. It asks banks to validate them independently, “notwithstanding any validation, certification or assurance provided by the third-party provider”.11 It also asks for continuous monitoring of drift. It is still a draft.
- RBI’s cyber directions, issued on 31 July 2026 with immediate effect, require banks to report cyber incidents within six hours of detection.8 CERT-In’s clock is also six hours.7
Where Tacit helps: outcome-based evidence your own validation team can use and challenge; drift monitoring; reporting-clock exposure.
Singapore
- Final
- Applies 2027
- MAS’s Guidelines on AI Risk Management (final, 7 October 2026) keep firms responsible for risks from third-party AI.12 Governance expectations apply from 7 October 2027, and lifecycle controls from October 2028.
- MAS’s SAFR paper (first published July 2026, updated 9 October 2026; voluntary) sets out four outcomes for an AI agent’s action: Auto-Execute, Observe, Escalate and Deny.6
- Firms must notify MAS within one hour of discovering a relevant IT security incident.10
Where Tacit helps: measured evidence on third-party AI; an autonomy map that uses SAFR’s four outcomes; reporting-clock exposure.
European Union
- In force
- Supervisory
- DORA has applied since 17 January 2025. It covers ICT third-party risk. The first notice of a major incident is due within four hours of classifying it, and no later than 24 hours after becoming aware of it.9
- In July 2026 the ECB asked the banks it directly supervises for action plans on AI-enabled cyber threats, including AI-enabled defence, with plans due by 31 October 2026.13
Where Tacit helps: oversight records for the AI in your detection and response; monitoring of third-party AI.
Canada
- Applies 1 May 2027
- OSFI’s Guideline E-23 on model risk management takes effect on 1 May 2027. It covers AI models, and models from third-party vendors.14
Where Tacit helps: outcome analysis and ongoing monitoring of AI triage models.
Who checks the checker?
A judge has to be tested too. Before we judge your AI, we test our own engine on data where we planted the answers.
Our engine was first built for insurance underwriting desks. In a synthetic underwriting test (8 underwriters, 3,200 files), it found all 7 patterns we had planted, with 1 false alarm in 113 places where there was nothing to find.15
Our SOC test comes next: synthetic data from 10 analysts and about 30,000 alerts, with a sealed answer key. We’ll publish what it finds, including what it misses.
Agent Decision Audit · sample report (illustrative, synthetic data)
| Alert type | Accuracy | Missed threats | Consistency | Drift | Call |
|---|---|---|---|---|---|
| User-reported phishing Rests on outcomes | 97% 96–98 | 0.3% 0.1–0.6 | 98% 97–99 | Stable | Act alone |
| Unfamiliar sign-in Rests on outcomes and a blind second look | 92% 90–94 | 1.2% 0.8–1.8 | 95% 93–97 | Stable | Act, and log for review |
| Impossible travel Rests on a blind second look and team view | 86% 82–89 | 2.6% 1.7–3.8 | 90% 86–93 | The AI moved · week 4 | Recommend; a person decides |
| Privileged role change thin · 46 decisions Rests on team view | 81% 69–90 | 4.3% 1.4–11 | 84% 72–93 | Too few to tell | Off |
Why your people overruled it · 214 overrides
- The AI was wrong 28%
- It lacked context 26%
- Against how the team decides 12%
- The team changed approach 7%
- The AI was right after all 18%
- A fair judgment call 9%
Every rate comes with a range. Thin evidence is labelled as thin.
Common questions
No. Tacit sits beside them and reads where decisions are recorded. It never sits between your analysts and their tools.
No. We judge agents, not analysts. Analyst IDs are scrambled with a key only you hold, and every result about people is reported at team level.
To judge the AI, we need its decision records. Who holds them depends on your set-up:
- In-house SOC on Microsoft Defender and Sentinel, or Google Security Operations. You hold the records, in your own tenant. You grant read-only access. No other vendor needs to say yes. Our Google connector arrives in early 2027.
- A provider co-manages your SOC inside your tenant. You hold the records, so it works the same as in-house.
- An MDR provider works in your CrowdStrike console. Usually you hold the records, so we connect directly once our CrowdStrike connector is ready in 2027. We confirm what’s recorded in one short call.
- An MDR provider runs its AI on its own platform. Your provider holds the records, so we connect through them. Ask for a per-alert decision export (we’ll give you the clause), or ask them to commission a Tacit report. Either way, the report comes to you unedited.
Not sure which one you are? That’s the first thing we work out on the call.
Tacit is built to read from the systems where decisions land, and to judge any AI agent whose verdicts are recorded: a platform’s agent, your provider’s AI, a tool you bought, or one you built.
- Microsoft Defender XDR and Microsoft Sentinel: alerts, incidents, the AI’s classifications and audit history. First, with design partners from December 2026.
- Google Security Operations: investigations, and the AI’s verdict and confidence. Next, in early 2027.
- CrowdStrike Falcon: alerts and decisions. Planned for 2027.
- Splunk Enterprise Security: notable events and how they were closed. Planned.
- ServiceNow and Jira: outcomes and case history. First.
- Your provider’s export (CSV or API): per-alert decision records. With the first design partners.
Running something else? Tell us what you use.
Yes, through the provider. You have two options, and either way you get the report unedited:
- Ask for a per-alert export of the AI’s decisions. We’ll give you the clause for your renewal.
- Ask your provider to commission a Tacit report.
We read decision records, and only the fields you approve. We list them before we connect:
- the alert’s type, timing and approved fields;
- the AI’s verdict;
- your team’s decision, the action and the outcome.
User names, email addresses and IP addresses are hashed with your key. We don’t read email bodies, attachments, files or packet captures. Nothing leaves your cloud except the reports you choose to share.
Up to 90 days. For a firm answer on an alert type, we need a few hundred AI decisions on it. If you have fewer, we record forward for six weeks.
We suggest starting with user-reported phishing. It’s high-volume, and outcomes come back fast. Identity alerts come next.
It’s designed for exactly that, because it’s normal in a SOC. We use the outcomes that do exist, the blind second look and your team’s steady judgment. Each finding says which it rests on, and thin evidence is labelled as thin.
A vendor’s dashboard reports on its own AI, by its own measure. Tacit measures it independently, on your alerts, against what actually happened. It also shows why your people overrule it and where it can act alone.
Those test whether planted attacks get caught. They’re useful, and we use their results as outcomes. But they can’t tell you why your people overrule the AI, whether it’s drifting on real traffic, or where the evidence supports letting it act alone.
No. Those tools use another AI to grade an AI. Tacit checks your AI’s calls against what actually happened on your alerts, and against how your own team decides.
No tool can. Tacit gives you evidence your frameworks ask for: independent testing, monitoring, and records of human oversight. Your compliance team decides how to use it.
Yes. Under NDA, we share our method documentation, test results and the evidence behind every finding. Your validators can review it like any other third-party model.
We’ll complete your security questionnaire and share a data-flow diagram before we connect.
The Audit is a fixed fee. Shadow Judge is a subscription priced per agent. Provider licences are priced per client. Talk to us for a quote.
Sources
- Microsoft Learn: Security Copilot inclusion in Microsoft 365 E5 and E7
- Bono et al., Microsoft: Randomized Controlled Trials for Phishing Triage Agent, arXiv 2511.13860, Nov 2025. 167 security analysts; the errors were planted by the researchers.
- Prophet Security: State of AI in the SOC 2026, 3 Aug 2026. Survey of 250 security leaders.
- Help Net Security (sponsored): Stellar Cyber customer-trial study, 5 Aug 2026
- Banstola et al., University of South Florida and Cyber Florida: It is Not Yet Another Tool, arXiv 2609.06250, Sep 2026
- MAS SAFR paper, first published July 2026; v1.1, 9 Oct 2026
- CERT-In Directions under section 70B(6) of the IT Act, 28 Apr 2022.
- RBI Cybersecurity, Technology Risk, Resilience and Assurance Directions, 31 Jul 2026 (KPMG summary)
- Commission Delegated Regulation (EU) 2025/301, Article 5 (Advisera)
- MAS one-hour incident notification (Pinsent Masons summary)
- RBI draft Guidance on Regulatory Principles for Model Risk Management, 24 Jun 2026 (LegalEra summary)
- MAS Guidelines on AI Risk Management, 7 Oct 2026 (PwC Singapore summary)
- ECB supervisory letter on AI-enabled cyber threats, 7 Jul 2026 (KPMG Germany summary)
- OSFI Guideline E-23, Model Risk Management: final 11 Sep 2025, effective 1 May 2027 (Blakes summary)
- Tacit Labs internal test on synthetic underwriting data, 29 Sep 2026.
Give your AI analyst the interview it never had.
Book 30 minutes. Tell us your platform, your provider and which AI is switched on. On the call, we’ll tell you whether we can connect, what we’d measure first and what you’d get.
Prefer to write? Email hi@tacitlabs.ai.