You cannot ban your way out of shadow AI

A medical writer has a Friday deadline and a therapy-area summary to produce. The source material she needs — clinical study reports, prior submissions, the approved brand language for that market — sits somewhere in a SharePoint estate that has been growing for over a decade. Finding the right three documents takes most of a morning, and that assumes she already knows they exist.

Pasting the two documents she does have into a public chat tool takes four seconds.

That is the shadow AI problem in a global pharmaceutical organisation, and it is worth being honest at the outset that it is not really an AI problem.

The policy was never the bottleneck

This organisation had the policy. Every organisation in this position has the policy. It said, in the usual words, that confidential scientific and clinical material must not be entered into external tools. People did it anyway — not because they had not read it, but because the compliant alternative did not exist and the deadline did.

Shadow AI is a latency problem before it is a compliance problem. When the sanctioned path is slower than the unsanctioned one, a policy is a statement of preference rather than a control. You can escalate the wording, and teams under deadline will still choose the four-second option. The only durable fix is to make the sanctioned path the fast one.

There was a second problem underneath, and it was the one that actually frightened the compliance office: they could not see any of it. Not which tools were in use, not what was being pasted into them, not whether the resulting text respected the scientific and regulatory guidelines it was supposed to. There were no logs, nothing reproducible, no chain of evidence. Asked by a regulator to demonstrate how a piece of AI-assisted copy had been produced, the honest answer would have been that nobody knew.

And because everyone was improvising — different tools, different prompts, different reference documents — the same therapy area was being described differently in different markets. That is a commercial problem and a regulatory one at the same time.

Retrieval first. The AI part was never the hard part

Before any of the generative work mattered, the corpus had to become searchable in a way it had never been. A SharePoint estate of that age is not a library; it is sediment. Research, clinical and marketing material accumulated in folder structures that encoded a decade of reorganisations, with the useful content frequently locked inside tables in documents that were never written to be parsed.

So the first component was an ingestion pipeline running on GPU-backed container apps: process the source documents, pull the tables and structured content out intact, and transform them into retrieval-ready material aligned to the organisation’s own taxonomy rather than to whatever the folder happened to be called. Unglamorous, and it is the part that decides whether everything downstream works.

The part that took longest: where the permission check lives

Here is the question that shaped the whole architecture. A user asks something. The system searches a corpus that contains material that user is not cleared to read. Where, exactly, does the permission check happen?

There are three answers, and two of them are wrong in ways that are not obvious until you have built them.

The first wrong answer is to filter the output: let retrieval see everything, generate an answer, then strip out whatever the user should not have seen. This fails on inspection. By the time you are editing the answer, the restricted document has already been read into the prompt and has already shaped what the model said. Removing a citation does not un-read the source. You have built a system that leaks quietly and reports that it did not.

The second wrong answer is to build a separate index per user, so that retrieval can only ever return material that person can open. This is genuinely correct, and it is unaffordable at this scale. Every joiner, every internal move, every routine permission change means re-indexing. You would spend the organisation’s entire compute budget keeping the indexes honest.

The answer that works is to make the user’s permissions part of the query itself. Their group memberships travel with the request, and retrieval filters against them at query time, so material they cannot already open is never a candidate for retrieval in the first place. One index. No leak surface, because the restricted content never enters the pipeline rather than being removed from it later. The existing SharePoint folder structure stays the source of truth for who can see what, which also means the security model is one the organisation already understood and already audited.

One thing this raises that is worth flagging, because it is a product decision and not a technical one: if a user asks a question whose answer sits entirely in a document they cannot open, what should the system say? Answering from whatever partial material is visible produces a confident, wrong answer. Saying “there is material here you cannot access” is more honest and leaks the existence of the document. There is no clean answer. What matters is that the choice gets made deliberately, written down, and applied the same way every time — because whichever way it goes somebody will eventually ask why the system behaved as it did, and “nobody decided” is the worst available answer.

Why a coordinator agent rather than one retriever

The requests coming into this system were not all the same shape. Some needed retrieval from the internal corpus. Some legitimately needed current external information — a recently published paper, a regulatory update — which meant web access, which meant a path out of the tenant. Some were asking for content to be updated rather than read.

Those have different permission models, different freshness requirements and very different risk profiles, so they were built as specialised agents behind a coordinator, orchestrated with LangGraph. The coordinator routes each request and, more importantly, is the point where the guardrails sit: nothing returns to the user without passing the check.

The consistency problem got solved in the same place. Standardised system prompts hold scientific accuracy, brand-aligned tone and regulatory messaging steady across teams and therapy areas, so the same molecule is described the same way in two markets — because it is the same system saying it, not fourteen people with fourteen prompts.

And everything is logged: every retrieval, every agent action, every external reference, every generation, attributed to a user and timestamped. That is what turns “we think it is fine” into something a compliance team can actually reconstruct.

The objection worth taking seriously

There is a good argument, made often and made well, that most multi-agent systems are over-engineered — that one strong retriever and a capable model will beat an agent graph on latency, on cost and above all on debuggability, and that the graph mostly adds failure modes and places for a request to get stuck. I think that is right more often than the market currently admits, and if you are building internal search over one repository with one permission model, you should not be reaching for an agent framework.

Where this case genuinely differs is the heterogeneity. Internal retrieval and gated external access are not the same operation with a different parameter; they cross different trust boundaries and need different approvals. Once one of your paths leaves the tenant and the others must not, the routing decision becomes a governance control rather than an optimisation — and it needs to be an explicit, inspectable step, because a regulator will eventually ask which path a given answer took. That is the argument for the coordinator. It is not that it is faster. It is that it leaves a decision trail.

About those numbers

An earlier version of this page led with three figures: zero confidential documents leaving the tenant, one hundred per cent of AI actions captured for audit, and project-scoped access control honoured at retrieval time. All three are true. None of them is a result.

They are properties of the architecture. They were true the day the first component was deployed, because the system runs inside the tenant and logs by construction — a well-built system cannot fail to have them, and a buyer reading them should discount them accordingly. We would rather say that than have you work it out yourself.

The numbers that would mean something are ones no architecture is entitled to claim on its own behalf. If you are evaluating something like this, measure three. How long it takes someone to locate the right source document, before and after — that is the gap the entire design exists to close, and if it does not move, nothing else you measure matters. Whether external-tool use actually falls, which requires that you were measuring it beforehand, and almost nobody is. And what your compliance function can evidence to a regulator afterwards that it could not evidence before — the only one of the three that is a governance outcome rather than a productivity one.

We would rather hand you the measurements to take than a figure from somebody else’s estate, which tells you nothing about yours.

What is still awkward

Permissions inherited from a decade-old SharePoint estate are permissions that encode a decade of decisions, some of which nobody remembers making. A retrieval system that faithfully honours them will faithfully honour the mistakes too, and the system surfaces those far more visibly than manual search ever did — which is useful, and is also work somebody now has to do.

The gated web-access path remains a policy question rather than a technical one. The control is built; deciding who gets it, for what, is not an engineering decision and should not be presented as one.

And the whole premise rests on being faster than the four-second option. That holds only as long as retrieval quality holds, which means the corpus needs curating, and curation is a standing commitment rather than a project phase. The failure mode here is always the same shape: the system works, the index quietly goes stale, retrieval quality drifts, and a few quarters later somebody has a deadline and a public chat tab open again. Deciding who owns the corpus after go-live is a more consequential appointment than it sounds.

Idea Infotech builds retrieval and agent systems that run inside the customer’s own environment, where the data cannot leave and the audit trail is a requirement rather than a feature. See the other engagements.