The relevant passage of the Online Safety Act 2023 does not say that platforms must remove illegal material. It says that a regulated service must carry out a suitable and sufficient assessment of the risk of illegal content appearing on it, and must then use proportionate systems and processes to prevent users encountering the most serious categories of it, and to take it down swiftly where it appears.
Every operative word there is about a system. Not a post, not an image, not an account. The Act is a piece of process regulation wearing the costume of content regulation, and almost every argument about it goes wrong at that point.
On Monday, Ofcom wrote to X. The subject was the use of the Grok tool on that service to produce sexual images of identifiable real people, including children, and the question the regulator put was how material of that kind had come to be distributed at the scale it was. What follows is not a verdict on that question, which is the regulator’s to answer. It is an account of the machinery the letter set in motion, and of why this particular case is awkwardly shaped for it.
What a duty of this kind actually obliges
A service in scope has to do three separable things. It has to assess the risk that illegal content appears on it, and write that assessment down. It has to put in place measures proportionate to the risk it has identified. And it has to keep both current, because a risk assessment describing a service that no longer exists is not a suitable and sufficient assessment of anything.
What the duty does not require is perfection. A platform that has assessed a risk honestly, adopted measures proportionate to it and acted quickly on reports has discharged its obligation even if illegal material still appears, because no realistic system stops everything. Conversely a platform on which relatively little appears may still be in breach if it never seriously assessed the risk in the first place.
This is why the regulator’s opening move is a letter rather than a takedown order. The question is not primarily what is on the service. It is what the service knew, when it knew it, and what it did about it. That evidence exists mostly inside the company.
How the machinery escalates
An information notice is the first hard instrument. It is legally binding, and it is not a request: failure to comply properly is itself a breach, and the Act attaches criminal liability to named senior managers who fail to ensure a response. The regulator has already used the smaller end of this. In one earlier case a website operator was fined a million pounds over inadequate age checks and a further fifty thousand for failing to answer the information requests about them, which is a useful illustration that the two failures are separately punishable.
If the answers do not satisfy, a formal investigation follows, then a provisional decision, then representations, then a confirmation decision. Only at the end of that does the penalty power arrive, and it is large: up to eighteen million pounds or ten percent of qualifying worldwide revenue, whichever is greater. For a service of any size the percentage is the operative figure.
Beyond fines sit the business disruption measures, which are the Act’s genuine deterrent and its least used. On application to a court, the regulator can require payment providers and advertisers to withdraw from a service, or require internet providers to block it in the United Kingdom. That is an instrument for a service that has decided not to comply at all, and using it against a major platform would be a first.
The distance from Monday’s letter to any of that is measured in months at best.
Why this case sits badly in the frame
The Act was drafted against a model of how harmful material moves: somebody makes it somewhere else, uploads it, it spreads, and the platform’s job is to find it and stop it spreading. Every duty in the illegal content regime assumes that shape. The platform is a conduit with obligations about what flows through it.
A generation tool built into the service collapses that model. The material is not arriving from outside and being distributed; it is being produced inside the service, on demand, by the service’s own feature, at whatever rate users ask for it. The conduit is also the factory.
The Act is not silent on this. Content produced by a user through a tool the service provides is still content generated by a user on that service, and it falls inside the regime. But the duties were designed around a corpus you can sample, filter and measure, and a generator does not have a corpus. It has a capability. Assessing the risk of a capability is a different exercise from assessing the risk of a library, and the codes of practice that tell services what proportionate measures look like were written for the library.
Two regulators, two legal orders
The Irish regulator, Coimisiun na Mean, has said it is engaging with the European Commission rather than acting directly, and the reason is structural rather than reticent. Under the European regime, the largest platforms are supervised by the Commission itself, not by the national regulator of the country where the company keeps its European headquarters. Ireland regulates a great many services. The very biggest are not among them.
So a single feature on a single service is now in front of two systems that do not share a threshold, a timetable or a definition. The United Kingdom is asking whether a service assessed and mitigated a risk. The European question is framed around systemic risk and the adequacy of the measures taken against it. Both can proceed at once, neither is bound by the other, and a company can in principle satisfy one and not the other.
What to watch next
Three things will tell you whether this is enforcement or correspondence.
The first is whether the letter becomes a formal investigation, and how fast. That is the step where the regulator commits publicly and puts a clock on the process.
The second is whether the case is framed around the tool or around distribution. A case about failing to take images down once reported is a conventional one and the existing codes fit it. A case about deploying a generation capability without adequately assessing what it would be used for is a much larger claim, and it would be the first serious test of whether process regulation reaches design decisions.
The third sits in Parliament rather than at the regulator. The government has said it intends to legislate in the Crime and Policing Bill to criminalize the tools themselves. If that passes, the question stops being whether a platform responded adequately to what its tool produced, and becomes whether the tool may exist. That is a cleaner question, and a much harder one to answer at the borders of a jurisdiction that ends at the coast.
