Let me start with a question back at you.
Do you know where the documents your employees are uploading into AI tools are going right now?
Not in theory. Not what the terms of service say. Where they’re actually going, which systems touch them, which services process them, where they’re stored, and who carries responsibility if something goes wrong.
If the honest answer is no, you’re not alone. And that’s the risk nobody is talking about seriously enough.
Risk one: the data privacy chain nobody can see
Here’s what happens when someone uploads a document into an AI-powered tool, even a simple, well-intentioned one.
The responsibility for what happens to that document doesn’t sit in one place. It’s distributed across a chain: the person who uploaded it, the content of the document itself, the product they uploaded it into, the underlying language model that processes it, the sub-services used to extract and retrieve information, and the hosting provider (including where their servers are physically located and what data residency rules apply).
Each link in that chain carries risk. Each link involves decisions made by different organisations, governed by different agreements, subject to different regulations. And most people uploading documents into AI tools have no idea that chain exists, let alone how long it is.
If people knew how complicated a simple thing was, they would probably be a little more scared. I am.
Risk two: hallucinations you aren’t checking for
The second risk is one that astonishes me: not because it exists, but because so few businesses seem worried about it.
Hallucinations.
AI systems produce wrong answers. Not occasionally, not rarely: regularly, and with complete confidence. They state incorrect facts, fabricate sources, misrepresent data, and construct plausible-sounding nonsense that looks identical to accurate output. The system doesn’t know it’s wrong. It has no mechanism for knowing. It produces the most statistically likely next token, not the most accurate one.
Most businesses using AI tools right now are not systematically checking outputs for accuracy. They’re reading them, finding them plausible, and moving on. Which means wrong answers are entering workflows, decisions, documents, and communications, dressed in the same confident language as correct ones.
Why a confident answer is not a correct one
The most important thing anyone has ever said to me about AI came from one of my lead developers.
We were working on a product together. The interface produced an answer we expected. I said, “Great. It’s working.”
He stopped me. “You don’t know it’s working,” he said. “You don’t know how it arrived at that answer. It just looks right because that’s what you think is right.”
That sentence changed how I use AI permanently.
The output confirming your expectation is not evidence that the output is correct. It’s evidence that you wanted it to be correct. The AI has no way of distinguishing between the two; and neither, in that moment, do you.
I literally don’t believe anything AI tells me these days without checking it. Not because AI isn’t useful. It’s extraordinarily useful. But because the confidence of the output has no relationship to its accuracy. A wrong answer and a right answer look exactly the same.
Risk three: no AI policy while everyone already uses it
The third risk sits underneath both of these. And it’s the one most businesses aren’t even asking about yet.
Most organisations have no AI policy. Employees are already using ChatGPT. They’re already uploading documents, summarising meetings, drafting communications, answering customer queries. And nobody has asked where any of it goes, what gets retained, or what happens when something goes wrong.
This isn’t a criticism of the employees. They’re using the tools available to them to do their jobs better. The gap is organisational. The business hasn’t caught up with what’s already happening inside it.
The risk isn’t that AI is coming. It’s already there. The question is whether it’s being used with any awareness of what it involves, or whether it’s running on enthusiasm and assumption, which is where most organisations currently are.
Using AI without the risks outweighing the benefits
None of this means don’t use AI. I use it every day and it has made me significantly better at my work.
But I use it with a specific disposition: scepticism first, verification always, and a clear understanding that the output is a starting point for my thinking, not a replacement for it. That disposition takes discipline to maintain, especially when AI is producing things that look exactly right.
The businesses that will get real, sustained value from AI are the ones that build that discipline in deliberately. Not as a policy document that nobody reads, but as an actual operating practice: a culture of checking, questioning, and never assuming that what looks right is right.
That’s harder than buying a tool. It’s also the only thing that makes the tool actually safe to use.
Garth Shoebridge has been building with AI and thinking carefully about what it does and doesn’t do reliably. If you’re trying to work out how to bring AI into your business without the risks outweighing the benefits, start with a conversation.