Signing a contract with an AI consultancy without a technical background is not the problem. The problem is signing without asking the right questions.
This article is not meant to teach you how to evaluate language models or agent architectures. It is meant to help you know what to ask before committing budget — and which answers should concern you.
The impressive-demo trap
Most AI proposals are won or lost in the demo. The vendor shows an agent answering questions, generating a report, or processing invoices. It looks solid. The technical team nods.
The problem is that a demo answers none of the questions that matter to the business: how much does it cost to run? What happens when it fails? Who maintains it six months from now?
An experienced CFO doesn't evaluate the demo. They evaluate what comes after the demo.
Question 1: What is the baseline metric today?
Before discussing improvements, any serious proposal must establish the starting point. If the consultancy proposes to "reduce monthly financial close time," the first question is: how long does that process take today, measured in person-hours per cycle?
If they don't have that figure — or if the proposal doesn't mention it — there is a problem. Not because the data is hard to obtain, but because its absence indicates the vendor didn't do the work of understanding the business before proposing a solution.
Without a baseline metric, any promise of improvement is decorative.
Question 2: What defines success at week 6 and at month 3?
AI projects that fail don't fail at the end. They fail silently during the first few months, when no one is measuring anything and everyone assumes things are "moving forward."
Ask for intermediate milestones with specific success criteria. Not "the agent will be running," but "the agent will process X type of requests with an error rate below Y%, measured in this way."
If the vendor cannot commit to measurable intermediate milestones, the project has no real structure. It has intent.
Question 3: Who operates this when you're no longer involved?
This is the question that makes vendors who sell dependency most uncomfortable.
There are two business models in the market. The first: the consultancy builds, delivers, and disappears. The client is left with a system no one on their team knows how to operate or modify. The second: the consultancy builds alongside the team, transfers the knowledge, and remains available as a safety net — not as a bottleneck.
The first model is more common. The second is more expensive upfront and less expensive over time.
Ask the vendor: if we end the contract tomorrow, can my team operate and modify what you built? An honest answer to that question is worth more than any architecture slide.
Question 4: What is the monthly operating cost once in production?
AI proposals typically present implementation costs clearly and leave operating costs in the fine print. Language models carry per-use costs. Agents that process volume generate variable costs. Integrations with external systems may require additional licenses.
Ask for a monthly operating cost estimate across three scenarios: low usage, expected usage, and high usage. If the vendor cannot provide this, or if the variance between scenarios is wide without explanation, that is an unquantified financial risk.
A concrete example: a distribution company with 40 people in administration deployed an agent for purchase order management. The implementation cost was EUR 18,000. The estimated monthly operating cost was EUR 400. In practice, as processed volume scaled, it reached EUR 1,100 per month. It wasn't a serious problem — but it wasn't in the contract. That kind of surprise erodes confidence in the project.
Question 5: What happens if the team doesn't adopt the solution?
The failure rate in automation projects is not driven primarily by technical problems. It is driven by teams that don't change how they work.
An agent no one uses generates no ROI. An automated workflow the team avoids because "it's faster to do it manually" is a cost with no return.
Ask the vendor how they manage adoption. Is there a change management plan? Who works with the operational team during implementation? How is actual usage measured against expected usage?
If the answer is "the client handles that," the vendor is offloading the project's primary risk.
Question 6: Do you have experience with companies our size?
Consultancies that work with large corporations apply methodologies designed for dedicated teams of 10, six-figure budgets, and 18-month timelines. That doesn't work in an 80-person company where the CFO also manages treasury and the COO also handles client relationships.
Ask for references from clients with a structure similar to yours — not in revenue size, but in operational complexity, team size, and capacity to absorb change.
A useful working hypothesis: in companies of between 30 and 150 people, AI projects that generate positive ROI in under 90 days tend to target processes with high repetitive volume, low variability, and a clear owner. If the proposal aims at something more complex as a first step, the risk of seeing no results in the first quarter is high.
Question 7: How is the system governed once in production?
Governance is not a technical term. It is the answer to: who checks that the agent is still performing correctly? Who detects when it starts making errors? Who decides when to update it?
Without governance, AI systems degrade over time. Data changes, processes change, underlying models are updated. An agent that worked well in January may be generating errors by July if no one is monitoring it.
Ask the vendor to explain their post-implementation governance model. If they don't have one, the project ends the day it is delivered.
What these questions reveal
A vendor who answers these seven questions clearly, with data, and without evasion has substance behind the proposal. A vendor who deflects, generalizes, or makes promises without committing to metrics is selling intent, not results.
You don't need to understand how a language model works to make a sound decision. You need to know what to ask — and which answers not to accept.
If you want to test these questions against your specific situation, OuroAI offers a free diagnostic with no prior call required. Share your case through the form and we'll respond with an analysis tailored to your context.
