Designing AI Pilots That Can Scale: Our Approach

When adopting a new AI tool, organisations often fall into an all-or-nothing trap: commit everything at once and take on unnecessary risk, or avoid risk altogether. But starting small with a pilot offers a practical third option — a low-stakes experiment to see whether the tool delivers on its promise.

Unfortunately, most AI pilots never make it past that stage. A recent Deloitte report found that nearly 3 in 4 executives in health nonprofits say generative AI has already reduced inefficiencies, yet only 1% have scaled it across their organisations. Some barriers they point to are a lack of standardised frameworks, funding limits and low confidence. Funding is often the hardest to tackle because it’s a catch-22: you need funding to prove a pilot works, but you need proof it works to get the funding.

But that’s precisely why we think pilots matter. A pilot can tackle each of these barriers: it helps build the standardised framework, makes the case for impact and cost that unlocks funding, and builds confidence in the tool along the way.

For that to work, a pilot first has to communicate its impact in a way funders and other stakeholders understand. We believe that’s where pilots often stall.

So, here is how we’re solving this problem. We start with the client and finish with a pilot impact report that shows them what changed and what to do next.

Starting with client pain points

Before building anything, we start by understanding a client’s pain points, through a discovery call and questionnaire that scope where AI agents can help within their existing processes. A UN agency told us their biggest struggle was conducting real-time research on humanitarian aid donations as information kept updating. A conservation nonprofit’s bottleneck was mapping donors in their database who aligned with their mission. Their answers give us insight into where agents can help, and that becomes the starting point for building the pilot.

What the pilot team tracks

To make sure we and the client are on the same page, we first document how agents and staff will work together on their desired process. Once the process is agreed, we capture it in our implementation plan: agent roles, human review points, and the KPIs that tell us whether the pilot is succeeding.

For the UN agency’s research workflow, a KPI could be the hallucination rate: how often the agent’s sources are incorrect when a staff member checks them. For the conservation nonprofit’s donor mapping, it could be the task completion rate: how many donors the agent maps per week.

Once the KPIs are locked in, we build the workflow and demo it for the client. Once they approve the outputs, the live pilot begins and we start collecting data against the KPIs.

These KPIs are how the pilot team judges the agent, but they are numbers only the pilot team really understands. So, what about the clients we serve? What do they really want to know?

What matters to the client

While KPIs established during pilot development are useful in understanding agent behaviour, what clients ultimately care about is how much staff time gets saved — and how that compares to doing the work manually. For example, a staff member working alongside an agent can produce in 1 hour what would take 8 hours manually, resulting in an 8x increase in task efficiency. Our implementation plan records the client’s own estimate of how long each task takes manually, which we then compare against staff time with the agent.

But that time saving doesn’t always mean money saved. In our experience, hours saved often get reallocated to work that was likely being skipped before, like higher-order planning and strategic decision-making. That time doesn’t reduce headcount or spend, so it rarely shows up as a cost saving. As a result, the stronger case for mission-driven organisations is usually capacity gained, not cost cut.

Translating metrics for funders

The people who decide whether a pilot gets funded — a nonprofit board or executive — rarely ask for metrics by name. A task completion rate or a hallucination rate means little to someone who’s never heard the term, let alone knows it exists. And even explained, those metrics don’t answer the question funders are ultimately asking: what changed for their teams because of this?

Our job becomes translating all of it — technical benchmarks and efficiency numbers alike — into language they can act on. For example, an 8x efficiency gain becomes “this agent gives your organisation’s team back nearly a full workday.” And a 75% drop in AI agent hallucinations becomes “your team spends an hour a week checking the agent’s work instead of a full afternoon.”

At the end of the pilot, we present a pilot impact report that translates metrics for the relevant stakeholder and shows where the pilot did well and where it didn’t. It can be the case for funding to scale, or the basis for deciding not to go ahead. Either way, the decision rests on understandable data. In our view, this translation work is often what separates pilots that scale from the ones that stall.

So here is an open question to you as we learn more: what do you think a funder needs to see to move from pilot to scale?