Because AI Says So: What Happens When A Chatbot Becomes The Office Expert?

At some point, “I asked AI” stopped meaning “I wanted to see what it would say” and started meaning “I have evidence that you’re wrong.” And if you’ve ever had someone come back to you with a ChatGPT screenshot after you’ve already explained something, you’ll know exactly how quickly that can get irritating.

The problem isn’t that AI gave them another answer – sometimes getting another perspective is useful – the problem is that the chatbot usually wasn’t given the information that made the original answer make sense in the first place. A professional might know what platform a website is built on, what security it needs, where it’s hosted or what features are involved, while the chatbot has been given a five-word question and somehow ended up sounding like the final authority.

 

AI Doesn’t Know The Context You Forgot To Give It

 

It usually starts with someone asking a very broad question, getting a very neat answer and forgetting that the chatbot has no idea about all the things they didn’t put into the question.

Denys Romanov, founder of ReNewator, has seen just how much context matters from building AI agents himself. His voice AI agent Polina is now being used in three restaurants in Spain, where it has handled 7,344 calls and created 926 bookings independently. But getting an AI system to actually work for a business takes more than plugging in a language model. It needs the right business information, access to the relevant systems and clear boundaries around what it can and can’t do.

That applies to a lot more ordinary uses of AI too; as Denys explains, “The same applies to everyday professional AI use. If you give AI incomplete context, it may confidently suggest something that sounds right but doesn’t fit the actual situation.”

Ivan Vislavskiy, CEO of Comrade Digital Marketing Agency, has seen the same thing from the other side of the conversation. “Increasingly, a customer will object to our proposal on the grounds that ‘ChatGPT says we should do X’.  The issue here, is that they used the AI to ask an overly broad question that ignored their situation, market, competitors, specific facts, and instead received a generic answer that is sounding impressive but has nothing to do with their reality.”

 

 

When “AI Says…” Becomes The Evidence

 

Raul Menoyo, founder of Citora, has had clients arrive with ChatGPT screenshots showing that their company wasn’t mentioned or that another company was supposedly the obvious choice. Instead of simply accepting the answer, his team went back and checked what AI had actually said. Six out of six companies that had supposedly “not appeared” showed up when they asked the exact same question again on the same day.

“That screenshot is the least reliable thing in the room,” Raul says.

Neycho Tepavicharov, co-founder of Flashcloud, has seen something similar when using AI for competitor research. The pricing figures it produced looked specific enough to be believable, but several were wrong when checked against the companies’ own websites. They weren’t so obviously wrong that you’d immediately question them either, which is arguably what makes this kind of mistake more annoying.

His team’s solution is quite simple: AI can help gather information and get a draft moving, but it doesn’t get to decide whether that information is actually correct. Anything factual gets checked against the primary source before it’s used.

Which is probably a good rule for anyone who has ever been tempted to send a screenshot to someone and say, “See?”

 

The “Yes-Model” Problem

 

There is another little problem with using AI to settle an argument: sometimes you can influence the answer simply by arguing with the chatbot first.

Dr Mark Khater, Head of the Centre for Strategy and Performance at Cambridge University, calls this the “yes-model”. We have spent years talking about the dangers of yes-men who tell people what they want to hear. Now, according to Mark, we have a version of that problem with AI.

“Tell a model its answer is wrong and it will apologise, accept your correction and give you a new answer that backs up what you just told it. So when someone brings an AI answer to challenge professional advice, the first question is what they told the model before it answered.”

So if someone has already decided they are right, asking AI isn’t exactly going to challenge that belief. Depending on how the conversation goes, it’ll probably just help them build a better-looking argument for it.

 

So, What Does Using AI Properly Actually Mean?

 

Using AI well probably has less to do with how much you use it and more to do with knowing when to stop and question what it’s given you. If the answer depends on information the chatbot wasn’t given, or something that could easily be checked elsewhere, it probably needs a bit more work. You don’t need to write a 14-paragraph prompt every time you ask an AI something, but you do need to give it enough information to understand what you’re actually asking.

Arjun Jaggi, an applied AI researcher, sums up one part of this quite nicely: “fluency and accuracy are completely different things and a well written wrong answer is still a wrong answer.” A polished response can make it easy to forget that the quality of the writing and the quality of the information are two different things.

Neycho’s approach of checking factual information against primary sources is one practical way of dealing with that, while Denys’ experience shows why giving AI the right information matters in the first place. Mark uses AI to transcribe and tidy his thinking, but keeps people responsible for the actual judgement. Livia Bernandini, creator of Intuitive Space, describes an LLM as “a member of a curated team, not the head of it”, putting the emphasis back on AI as part of the process rather than the person making the final call.

 

Maybe Don’t Fire The Office Expert Just Yet

 

There is nothing wrong with asking AI for another opinion – the problem only usually starts when that second opinion is treated as though it has automatically settled the conversation, without anyone stopping to check where the answer came from or what information it was based on.

Because the professional you’re arguing with might have context that the chatbot doesn’t. They know the specific circumstances behind their advice, while the AI is only working with whatever information you’ve given it.

 

Our Experts:

 

• Dr Mark Khater: Head of the Centre for Strategy & Performance at Cambridge University and CEO of AQM Squared
• Raul Menoyo: Founder of Citora
• Denys Romanov: Founder of ReNewator
• Livia Bernandini: CEO of Intuitive Space
• Arjun Jaggi: Applied AI Researcher
• Ivan Vislavskiy: CEO and Co-founder of Comrade Digital Marketing Agency
• Neycho Tepavicharov: Co-founder of Flashcloud

 

 

Dr Mark Khater – Head of Centre for Strategy & Performance at Cambridge University, CEO of AQM Squared

 

 

“Tell a model its answer is wrong and it will apologise, accept your correction and give you a new answer that backs up what you just told it. So when someone brings an AI answer to challenge professional advice, the first question is what they told the model before it answered.”

“Treat an AI answer as a first draft from a fast, well-read colleague with no context about your situation. Ask what it was told, and what it could not know.”

“Before you use it to challenge someone who has done the job for twenty years, ask what they know that the model does not.”

Raul Menoyo, Founder of Citora

 

 

 

“Clients bring me a screenshot from ChatGPT and treat it as a verdict. Usually it says their company does not appear anywhere, or that a competitor is the obvious choice, and the whole meeting starts from there.”

“In my work that screenshot is the least reliable thing in the room. We checked it properly and six out of six companies that “did not appear” showed up when we asked the same question again. Same prompt, same day, different answer.”

 

Denys Romanov, Founder of ReNewator

 

 

 

“The same applies to everyday professional AI use. If you give AI incomplete context, it may confidently suggest something that sounds right but doesn’t fit the actual situation.”

“AI literacy means learning to question the output, check the context and understand why the answer makes sense before using it professionally.”

 

Livia Bernandini, CEO of Intuitive Space

 

 

 

“I experienced this in workshops I run with senior leaders from global organisations. One senior member once asserted he was right “because Claude said so”. Seriously.”

“In my view, an LLM must be treated as a member of a curated team, not the head of it.”

“As we lean into assisted intelligence, so we need to assist our judgements. Making sure we train our intuition to sense when something is slightly off or not good enough, and operating these tools with the right human colleagues rather than in an intimate one-to-one relationship, is where the real competitive advantage now sits.”

 

Arjun Jaggi, Applied AI Researcher

 

 

 

“Here’s what I’ve noticed in my own work. People trust AI the way they used to trust Google, and they assume that the answer is right because it showed up fast and sounded authoritative.”

“That’s the literacy gap nobody is really talking about. It’s not about whether you use AI or how often, it’s about whether you understand that fluency and accuracy are completely different things and that a well written wrong answer is still a wrong answer.”

“The thing I keep coming back to is that intelligence is cheap now but evidence is not and the people getting real value from AI are the ones who treat every output as a starting point rather than a final answer and know exactly what to verify before they act on it.”

 

Ivan Vislavskiy, CEO and Co-founder of Comrade Digital Marketing Agency

 

 

 

“Increasingly, a customer will object to our proposal on the grounds that ‘ChatGPT says we should do X’. The issue here, however, is that they used the AI to ask an overly broad question that ignored their situation, market, competitors, specific facts, and instead received a generic answer that is sounding impressive but has nothing to do with their reality.”

“As for my workflow, I have learned to use AI as a source of inspiration rather than a verdict, its responses being a helpful idea generator but not a replacement of human judgment based on facts.”

“My recommendation: before trusting an AI answer, think whether it had specific context or not. If it didn’t, it’s just guessing even though it sounds confident.”

 

Neycho Tepavicharov, Co-founder of Flashcloud

 

 

“AI answers sound equally confident whether they’re right or wrong, and nothing in the answer tells you which. We learned this researching competitor pricing for a comparison article. The figures looked authoritative and specific, and several were wrong when we checked each company’s own pricing page. Not wildly wrong, which we’d have caught. Wrong in the way that survives a quick read.”

“Now we have a rule: AI does the gathering and drafting, but it’s never the source. Anything factual gets checked at the primary source before it goes out.”

“For anyone using AI to question an expert, I’d ask one thing first: can the AI tell you where its answer came from? If not, you’re weighing a professional’s judgement against a guess that sounds sure of itself.”