Why do so many AI pilots fail to show a return?

Usually because no baseline was captured. Without a measurement of how long the task took, how much it cost or how often it failed before AI was introduced, any claimed improvement is an opinion. The second most common cause is choosing a metric the business does not care about — a pilot that improves a number nobody was managing will never survive a budget review, however well it performed.

What costs do people forget when calculating AI ROI?

The licence fee is usually the smallest line. The real costs are integration and data preparation, the time your team spends learning and adjusting, ongoing verification of output, the internal owner who maintains it, and usage-based charges that scale with success. Budget for the total cost of ownership over twelve months rather than the monthly subscription price.

When should you kill an AI pilot?

When it misses the success criteria you agreed before starting, or when the reason it is failing is a data or process problem the tool cannot fix. Set those criteria and a review date up front. Stopping a pilot that did not work is a good outcome, not a failure — the expensive mistake is quietly funding something indefinitely because nobody agreed what success looked like.

KTH Tech/SIGNAL/How to Measure ROI on AI (South Africa)

How to measure the ROI on AI in your business

By K. Thebe · AI & value · 8 min read

Ask a South African business what their AI investment returned and you will usually get an anecdote: it feels faster, people like it, the reports look better. That is not an answer a board accepts twice. The problem is almost never that AI produced nothing — it is that nobody set up the measurement to detect it. Here is a framework for proving whether AI is paying, built for businesses that have to justify the spend rather than admire it.

If you cannot state what the task cost before you deployed AI, you will never be able to prove what it saved. Take the baseline first — it takes an afternoon.

1.Baseline first, always

The single most valuable hour in an AI project happens before the project. Pick the process you intend to improve and measure it as it stands: how long it takes, how many times a week it happens, who does it and what they cost, the current error or failure rate, and the current outcome the business cares about — response time, conversion, collection days, downtime, whatever applies.

This is not sophisticated work. A spreadsheet, a fortnight of observation and honest numbers are enough. But without it every subsequent claim is unfalsifiable, and unfalsifiable claims lose to budget pressure every time.

2.Pick a number the business already manages

Tie the pilot to a metric that appears in a management meeting already. Time to respond to an enquiry. Debtor days. Unplanned downtime hours. Cost per support ticket. Stock write-offs. Conversion rate.

Avoid inventing an AI-specific metric such as "queries handled" or "documents processed". Those measure activity, not value, and they are exactly the numbers that look impressive in month two and get the programme cancelled in month nine. If improving your chosen number would genuinely make someone's job easier, it is the right metric.

3.The four kinds of value — count them separately

Cost reduction. Hours saved, converted at a real cost, or a cost line that actually falls. Be strict: hours saved only become money if they are reallocated or not paid for.
Revenue gain. More enquiries answered, faster quotes, better conversion, fewer lost sales from stock-outs. Usually the largest category and the most under-claimed, because it is harder to attribute.
Risk reduction. Fewer errors, fewer compliance failures, earlier fraud detection, less downtime. Value it as the expected cost of the events you avoided — imprecise, but not zero.
Capacity. Work you can now take on without hiring. For a growing South African SME this is frequently the real benefit, and it is invisible unless you name it deliberately.

Counting them separately stops the two failure modes: claiming a soft benefit as hard cash, and ignoring genuine value because it did not show up as a cost saving.

4.Count the costs vendors leave out

Subscription price is the visible tip. The full cost includes integration with your existing systems, data preparation — frequently the largest line and the one nobody budgets — learning time while the team adjusts, verification of output as an ongoing overhead, an internal owner who maintains prompts, rules and exceptions, and usage-based charges that grow precisely as the tool succeeds.

Two South African specifics deserve a line. Currency: many AI tools are priced in dollars, so your cost moves with the rand regardless of usage. And connectivity and power: a cloud tool your team cannot reach during an outage has a resilience cost that belongs in the model.

5.Run it as an experiment, not a rollout

Give the pilot a defined scope, a defined period — six to twelve weeks is usually right — a named owner, and success criteria written down before it starts. Where possible run a comparison: one team or one segment using the tool, another continuing as before. Even a rough comparison is far more persuasive than a before-and-after in a business where five other things changed.

Write the kill criteria at the same time. A pilot without a stopping rule becomes a subscription nobody can defend and nobody wants to be the person to cancel.

6.Watch the second-order effects

Some effects will not appear in your primary metric. Positive ones: staff retention when a hated task disappears, faster onboarding because knowledge is written down, better data quality because a system now enforces structure. Negative ones matter more — customer complaints, rework caused by unverified output, staff quietly abandoning the tool, and skill erosion when people stop learning to do the underlying task.

Ask the team directly at the halfway point. "Has this actually helped you, and what has it made worse?" tends to produce information no dashboard contains.

7.Decide: scale, adjust, or stop

At the review date there are only three honest outcomes. Scale — it met the criteria; extend it and re-baseline for the next stage. Adjust — the value is visible but the scope is wrong; narrow it to where it worked rather than broadening in hope. Stop — it missed the criteria, or the underlying problem is data or process and no tool will fix it.

Stopping is a success. A business that runs four disciplined pilots and stops two has learned more, and wasted less, than one that runs a single undefined programme for two years. In South Africa, where capital is expensive and margins are tight, that discipline is worth more than enthusiasm.

8.The one-page format

Keep the whole case on a single page: the process, the baseline numbers, the metric, the four value categories with amounts, the full cost including internal time, the pilot period and owner, the success and kill criteria, and the review date. If it does not fit on one page, the pilot is too broad to measure — which is itself the most useful finding you can get before spending anything.

Before you approve any AI spend:

The current process is measured: time, frequency, cost, error rate

The success metric is one the business already manages

Cost, revenue, risk and capacity value are counted separately

Hours saved are only claimed as money if they are actually reallocated

Integration and data preparation costs are budgeted, not assumed away

Internal time — learning, verification, ownership — is in the cost model

Usage-based charges have been modelled at success-level volume

Currency exposure on dollar-priced tools is acknowledged

The pilot has a scope, a period, a named owner and a review date

Success criteria and kill criteria were written before it started

A comparison group or segment exists where practical

The team is asked at halfway what it has made worse

The whole case fits on one page

The bottom line

AI returns are provable, but only if you set up the proof before you start. Baseline the process, pick a metric the business already cares about, count value and cost honestly across all four categories, and agree in advance what would make you stop. Businesses that do this scale the things that work and cancel the things that do not — which, over a couple of years, is the entire difference between an AI capability and an AI expense.

Need a business case you can defend?

KTH-Tech runs AI readiness assessments and scoped pilots for South African businesses — with baselines, measurable criteria and an honest recommendation at the end.

Get an honest assessment →

General guidance, not financial advice. Investment decisions depend on your circumstances — validate with a qualified financial professional.