The problem: AI projects that vanish without a trace
You approved an AI project three months ago. The vendor promised to reduce monthly close time. Today, six weeks in, the team is still working in Excel. No one knows whether the agent is working, what it actually costs to maintain, or whether it's worth continuing.
This happens because most mid-size companies approve AI projects without clear metrics. No baseline. No target. No way to know whether anything is working.
As a CFO or COO, your responsibility differs from the CTO's. You don't care about the architecture. You care about: how much does it cost? how much does it save? when do we break even?
Why generic metrics don't work
"Improve efficiency" is not a metric. Neither is "reduce errors." These are wishes, not numbers.
AI projects fail because:
-
There is no clear baseline. You don't know how many hours the monthly close takes today. You don't know how many errors occur. Without a baseline, any improvement is invisible.
-
Targets are aspirational, not realistic. "Reduce the close from 5 days to 1 day" sounds good in a presentation. In practice, it's unachievable. When it fails, the project gets cancelled.
-
Measurement happens at the end, not during. You wait 6 months to discover the agent isn't working. You've lost 6 months and budget.
-
No one owns the metric. The vendor says it's working. The team says it isn't. You don't know who to believe.
The 3-metric framework that works
Before approving any AI project, require these three metrics:
1. Baseline: Where are we today?
Measure the current state with precision. No approximations.
Concrete example:
- Monthly close: 40 hours of manual work (5 people × 8 hours)
- Errors detected at close: 8–12 per month (0.2–0.3% of transactions)
- Current cost: 40 hours × $50/hour = $2,000 per month in manual work
Without this number, you have no way to know whether you improved.
2. Realistic 6-month target
Define what "success" means in concrete numbers. Not aspirational percentages.
Rule: The target must be achievable with 80% confidence. If it isn't, it's a wish, not an objective.
Concrete example:
- Monthly close: 25 hours (37% reduction)
- Errors: 2–3 per month (70% reduction)
- Cost: $1,250 per month
Why these numbers? Because they are realistic. They don't promise magic. They promise measurable improvement.
3. Cost per unit of improvement
This is where many projects fail. The vendor says "we saved 15 hours per month." But what does it cost to maintain the agent? What does team training cost? When do you recover the investment?
