Managing risk on systems that use AI: what changes, what doesn’t

The deadline moved. The exposure didn’t.

On 24 July 2026 the Official Journal published Regulation (EU) 2026/1744 — the “Digital Omnibus on AI” — amending the AI Act. It entered into force on 27 July. Its headline effect is a deferral: obligations for stand-alone high-risk systems under Annex III now apply from 2 December 2027, and for high-risk AI embedded in products covered by Annex I product-safety legislation, from 2 August 2028.[1][2] The stated reason is prosaic — national competent authorities were not designated in time and the harmonised standards that providers were meant to conform to are not finished.[2]

We have had a run of calls since then that all start the same way: so we can park the AI work until next year? Two reasons that is the wrong conclusion.

First, not everything moved. The transparency obligations still applied from 2 August 2026, and the prohibitions and AI-literacy provisions have been in force since February 2025. The deferral is targeted at the high-risk conformity regime, not at the Act as a whole.[1]

Second, and more to the point: regulatory deadlines were never the reason to assess these systems. Most organisations do not have an Annex III high-risk system at all. What they have is a retrieval-augmented assistant sitting on top of the intranet, a model summarising support tickets, a copilot with write access to a repository, and three business units that procured something with “AI” in the name without telling IT. None of those are regulated as high-risk. All of them are new attack surface, live in production, today.

The extra sixteen months are genuinely useful. They are useful as time to get your inventory, your assessment method and your governance in order before the conformity work lands on top. They are not a reason to stop.

The compliance clock stopped. The threat clock never does.

What does not change

There is a strong instinct, when AI enters the conversation, to reach for an entirely new framework. Resist it for a moment. The overwhelming majority of the risk in an AI-enabled system is ordinary system risk: authentication, authorisation, data handling, logging, supplier dependency, availability. ENISA’s Threat Landscape 2025, which analysed 4,875 incidents between July 2024 and June 2025, still describes an EU threat environment dominated by familiar mechanics — phishing as the dominant entry vector, ransomware as the dominant damage, and a sharp rise in hacktivist DDoS against public administrations.[3] Nobody’s model got jailbroken into a ransomware incident. Somebody clicked a link.

So the sound starting position is: an AI system is a system. Assess it with the method you already use. If that method is ISO/IEC 27005 or ITSRM, the structure holds without modification — identify the primary assets, identify the supporting assets, model which supports which, enumerate threats, score, treat.[4]

What changes is what goes in the boxes. Three things specifically: there is a new class of supporting asset, a new class of threat, and a new problem with assurance. Everything else is inherited.

The new assets nobody has on the inventory

Run a normal asset workshop on an AI-enabled system and the usual things get listed: servers, databases, network segments, the application, the team. Then somebody says “and the model” and the room pauses, because it is not obvious what kind of thing a model is.

For risk purposes, treat these as supporting assets in their own right, each with an owner, a source, and a trust level:

• The model itself — whether hosted by a third party behind an API, self-hosted open weights, or fine-tuned in-house. The distinction matters enormously for who controls the risk. An API-hosted model is an outsourced dependency with an inherited risk profile; open weights you host yourself are a supply-chain artefact you are now responsible for verifying.
• Training and fine-tuning data — both a primary asset (it may contain your confidential material) and an integrity-critical input. If someone can influence what goes into a fine-tune, they can influence what comes out of it forever.
• The system prompt and its configuration — frequently contains business logic, guardrail instructions, and sometimes credentials or internal URLs that were never meant to leave. It is a confidentiality asset that most teams treat as a config file.
• The retrieval corpus and the vector store — in a RAG system this is where your actual sensitive data lives. Access control on the vector store is frequently weaker than on the source documents it was built from, which quietly converts a well-governed document library into an ungoverned one.
• Tools, plugins and agent permissions — every function you let the model call is a privilege you have delegated to a component whose behaviour is probabilistic. This is the asset class that keeps us up at night, and it is almost never inventoried.
• Conversation logs — users paste things into chat interfaces that they would never put in a ticket. Assume the log is more sensitive than the system it serves, then check where it is stored and for how long.

If you do nothing else after reading this article, add those six lines to the supporting-asset list for every AI-enabled system you run. The gaps that exercise surfaces are usually enough to justify the rest of the work.

The new threats, and which ones are real

Two reference points are worth having open when you build the threat list.

The first is Article 15 of the AI Act itself, which requires high-risk systems to be resilient against attempts to manipulate the training data set (data poisoning), attacks on pre-trained components (model poisoning), inputs crafted to cause the model to err (adversarial examples or model evasion), confidentiality attacks, and model flaws.[5] That list is a decent AI threat taxonomy even if you are nowhere near being a high-risk provider — the legislator has effectively done the enumeration work for you.

The second is the OWASP Top 10 for LLM Applications, whose 2025 edition (v2.0) was published in November 2024 by the OWASP GenAI Security Project. It remains the most operationally useful catalogue for anything language-model shaped, running from LLM01 Prompt Injection through LLM10 Unbounded Consumption, with newer entries covering Excessive Agency, System Prompt Leakage, Vector and Embedding Weaknesses, Misinformation, and Supply Chain.[6]

Being honest about relative severity, in our experience across deployed systems:

• Indirect prompt injection is the one that will actually get you. Not the user typing “ignore your instructions” — that is the demo version. The real version is a model that reads a document, a web page, a calendar invite or a support ticket containing instructions written for it by someone else, and then acts on them. Any system where the model both ingests untrusted content and holds a privilege is exposed. Treat every piece of retrieved content as untrusted input, exactly as you would treat form data.
• Excessive agency is the multiplier. Prompt injection on a read-only summariser is embarrassing. Prompt injection on an agent with write access to your ticketing system, your repository or your mailbox is an incident. The control is unglamorous and effective: least privilege on tool access, human confirmation on anything irreversible or outbound, and a hard separation between the model’s ability to propose an action and the system’s ability to execute it.
• Supply chain is the underrated one. Pre-trained models pulled from public hubs, datasets of uncertain provenance, plugin ecosystems with no review process. Traditional software supply-chain hygiene does not cover model artefacts, and most organisations have no policy on where a model may come from.[6]
• Data leakage through retrieval is the quiet one. Not a model attack at all — an access-control failure. The classic pattern is a corpus indexed once with an administrator’s permissions, then queried by everyone, with the result that the model cheerfully surfaces the salary review to whoever asks nicely. Enforce authorisation at query time, per user, or accept that everything in the index is readable by every user of the system.
• Confabulation matters more in some systems than in any threat model. A model that states something false and plausible, in a workflow where a human acts on it without verification, is a business risk with no attacker required. NIST’s Generative AI Profile lists confabulation and information integrity among its twelve risk categories for exactly this reason.[7]

There is also an assurance problem that no control list solves. A conventional system is deterministic: you test it, it passes, it keeps passing. A model is not. The same input can produce different outputs, and a vendor updating a hosted model can change your system’s behaviour without touching your code. This is why AI risk work leans on evaluation and monitoring rather than one-off testing, and why ISO/IEC 23894 emphasises continuous, dynamic risk assessment over point-in-time evaluation.[8] Practically: build a small regression suite of prompts that must keep behaving, run it on a schedule, and log enough to reconstruct what the system said when someone complains.

A worked example

Abstraction is cheap, so here is a small, real-shaped case: an internal assistant over the HR and policy document library, available to all staff, answering questions in natural language. Roughly 900 users. No customer data. Nobody would call it high-risk under Annex III.

Supporting assets: the hosted model API; the vector store; the ingestion pipeline; the document library it draws from; the system prompt; the chat front end; conversation logs; the identity provider; and the two engineers who can deploy changes.

The risks that mattered, once the modelling was done, were not the exotic ones:

• The corpus was indexed with a service account holding access to all HR folders, while the front end authenticated users but did not filter retrieval by their entitlements. Any employee could ask a well-phrased question about a colleague’s disciplinary file. This is not an AI risk; it is an authorisation risk that the AI made trivially exploitable at scale, which is the recurring theme of this entire discipline.
• Conversation logs, containing staff describing personal grievances in detail, were retained indefinitely in a store with broader access than the HR system itself.
• Policy documents were ingested from a library where any staff member could upload. A crafted document could place instructions in the retrieval path — indirect prompt injection with no external attacker required.
• Nobody had defined what happens when the model is confidently wrong about a policy and an employee acts on it. No disclaimer, no citation of source documents, no route to challenge an answer.

Treatments, in the order they were prioritised: per-user authorisation filtering at query time; log retention cut to 30 days with restricted access; ingestion restricted to an approved-publisher workflow; mandatory source citation on every answer so users can verify; and a short regression suite of policy questions run weekly against the hosted model to catch behavioural drift after vendor updates.

Total effort, a little over three weeks. Note that four of the five treatments are conventional security engineering. That proportion is typical. The value of a structured assessment is not that it finds exotic AI attacks — it is that it finds the ordinary failures that AI deployment has quietly made much more consequential.

The governance layer, briefly

Per-system assessment is necessary and not sufficient. Above it sits the question of who decides what the organisation may deploy at all. Three published references are worth knowing, and you do not need all three:

• ISO/IEC 42001:2023, published in December 2023, is the first management-system standard for AI. If you already run an ISO 27001 ISMS, this is the recognisable shape: policy, roles, impact assessment, lifecycle management, supplier oversight, continual improvement. It is certifiable, which is increasingly what enterprise procurement asks about.[9]
• ISO/IEC 23894:2023 is guidance rather than requirements, designed to be used alongside ISO 31000. It is the cheapest way to bolt AI-specific considerations onto a risk process you already run, and does not require a certification programme.[8]
• NIST AI RMF 1.0, released January 2023, organises AI risk work into four functions — GOVERN, MAP, MEASURE, MANAGE — where GOVERN runs across everything and the other three apply per system and per lifecycle stage. Its Generative AI Profile (NIST AI 600-1, July 2024) adds twelve GenAI-specific risk categories and over 200 candidate actions, and is the single most practical free document in this space.[10][7]

Our usual advice to a mid-market organisation: start with the NIST GenAI Profile because it is free and actionable, run the per-system assessments with your existing methodology, and only reach for ISO 42001 certification when a customer or a tender actually asks for it. Buying a management system before you have assessed a single system is how organisations end up with excellent documentation about systems they do not understand.

Whatever framework you pick, three governance decisions do most of the work: an inventory of AI systems that is actually maintained; a rule about which classes of data may be sent to which classes of model; and a named person who can say no.

The control most organisations skip

Almost every AI incident we have looked at involved a person doing something reasonable. Pasting a contract into a public chatbot to get a summary before a deadline. Giving an agent broader permissions than it needed because narrower ones broke the workflow. Acting on a confidently wrong answer because it was formatted well and arrived quickly.

Traditional security awareness training does not cover any of this. The phishing simulation programme you already run does nothing for a staff member deciding whether the tool they just discovered is safe to feed client data into. If you are deploying AI internally — and if you have not deployed it, your staff have, whether or not you know — then awareness content needs a new module covering what may be pasted where, why a fluent answer is not a verified one, and how to report an AI output that was wrong in a way that mattered.

There is also a regulatory nudge here: the AI Act’s AI-literacy obligation has applied since February 2025 and asks providers and deployers to ensure a sufficient level of AI literacy among staff dealing with these systems.[1] Most organisations have not done anything about it, largely because it carries no dramatic penalty. It is still the cheapest control in this article.

Where to start on Monday

• Inventory first. One page listing every AI-enabled system in use, including the ones procured by business units. You will find more than you expect, and the inventory is the prerequisite for everything else.
• Triage by privilege, not by hype. Rank by what the system can read and what it can do. A model with write access and access to sensitive data outranks a flashier one that only summarises public text.
• Assess the top two properly. Full asset-based assessment, using the six AI supporting-asset classes above and a threat list drawn from AI Act Article 15 and the OWASP LLM catalogue.
• Fix the authorisation problems first. They will be the majority of your findings, and they are the ones with a known engineering answer.
• Write the acceptable-use rule. Two pages, in plain language, covering what data may go to which tools. Publish it before you need it.
• Diarise a re-assessment. Hosted models change under you. Six months is a reasonable cadence for anything with real privilege.

How CL2R Advisory can help

Three of the things we do map directly onto the work above.

CISO as a Service. If nobody in your organisation currently owns the question “may we deploy this?”, that is the gap to close first. We provide senior security leadership on a fractional basis — setting the AI governance rules, chairing the decisions, owning the inventory, and representing the security position to your board and your customers, without the cost of a full-time hire.

IT Security Plans. Our ITSP service runs a full ITSRM v1.2 risk assessment on a named system and delivers a plan with treatment measures, owners, deadlines and computed residual risk. The methodology handles AI-enabled systems well precisely because it is asset-based: the model, the vector store and the tool permissions become supporting assets like any other, and the arithmetic works the same way. Fixed price, delivered in weeks. See ITSP as a Service.

Security awareness. We build and run awareness programmes, including AI-specific modules covering safe use of assistants, data-handling boundaries, and how to recognise and report a bad output. It is the control that costs the least and gets skipped the most.

If you want a straight answer on where your AI exposure actually sits, a thirty-minute call is usually enough to tell. Get in touch or book a slot in the calendar.

Sources

• [1] Regulation (EU) 2024/1689 (Artificial Intelligence Act), including Article 4 (AI literacy) and the application timeline. EUR-Lex. Implementation timeline: artificialintelligenceact.eu.
• [2] Regulation (EU) 2026/1744 of 8 July 2026 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 as regards the simplification of the implementation of harmonised rules on artificial intelligence (“Digital Omnibus on AI”) — published in the Official Journal on 24 July 2026, in force 27 July 2026; Annex III high-risk obligations deferred to 2 December 2027 and Annex I to 2 August 2028. Analysis: Hunton, Gibson Dunn.
• [3] ENISA, ENISA Threat Landscape 2025, published 1 October 2025; 4,875 incidents analysed over the period 1 July 2024 to 30 June 2025. enisa.europa.eu.
• [4] ISO/IEC 27005:2022, Guidance on managing information security risks (fourth edition). iso.org. On ITSRM², see our companion article on ITSRM risk assessment and the ITSP.
• [5] AI Act, Article 15 — Accuracy, robustness and cybersecurity, covering data poisoning, model poisoning, adversarial examples / model evasion, confidentiality attacks and model flaws. artificialintelligenceact.eu.
• [6] OWASP GenAI Security Project, OWASP Top 10 for LLM Applications 2025 (v2.0), published 18 November 2024. genai.owasp.org.
• [7] NIST, AI 600-1: Artificial Intelligence Risk Management Framework — Generative Artificial Intelligence Profile, released 26 July 2024; twelve GenAI risk categories and over 200 suggested actions. nvlpubs.nist.gov (PDF).
• [8] ISO/IEC 23894:2023, Information technology — Artificial intelligence — Guidance on risk management, published February 2023, intended for use with ISO 31000:2018. iso.org.
• [9] ISO/IEC 42001:2023, Information technology — Artificial intelligence — Management system, published December 2023. iso.org.
• [10] NIST, AI 100-1: Artificial Intelligence Risk Management Framework (AI RMF 1.0), released 26 January 2023; core functions GOVERN, MAP, MEASURE, MANAGE. nist.gov.