This week, explained

Prebuilt AI agents are arriving by the thousand. Run a fit-gap before you adopt one

By Prachit Parikh, business analyst · · 15 min read

On 10 November, IBM and Microsoft announced a joint platform called Imagine Enterprise. According to IBM's release, it's "an agentic platform designed to help enterprises build, run, and govern AI agents that help people execute industry-specific processes," and it starts with "more than 6,000 AI agents and assets." The first industry frameworks are financial services, insurance and retail.

Six thousand is a catalog, not a project. And catalogs change the question a business analyst gets asked. For the last two years the question was "should we build an AI agent for this?" From here, more and more often, it'll be "there's already an agent for this, can we just use it?"

That second question has a good answer, and BAs already know most of it. It's a fit-gap analysis, the same one we've run on packaged ERP and CRM software for decades. What's new is what can go wrong in the gaps. A gap in a packaged screen means somebody works around it. A gap in a packaged agent that issues refunds means money leaves the building, and maybe nobody notices until month-end.

So this post does three things. It explains what was actually announced, without adding to it. It shows where a classic fit-gap needs new parts for agents. And it walks through a filled example on a fictional retail client, using a free matrix I've put in my BA Toolkit this week.

Prefer to watch first? The question before this one is whether a process should be an agent at all. That's my five-question test:

Video: Should This Be an AI Agent? A 5-Question Test for BAs (7 minutes). This post picks up where it ends: you've decided an agent fits, and a vendor has one ready.

What was announced, and what wasn't

I read both the IBM newsroom release and Microsoft's copy of it. Here's what they say, as close to the text as I can keep it.

And here's what the release doesn't say. There's no price. There's no named evaluation or testing feature; "observability" is the closest word. MarketScale's write-up a few days later noted the missing price too, and described the named customer, the investment firm SEI, as "still in the exploring phase." That's a fair reading of the release, which says SEI is "exploring emerging and agentic technologies" with IBM Consulting.

None of that's a criticism. It's an announcement, and announcements don't list test plans. But it tells you where the BA work sits: the platform gives you governance hooks and a catalog. Proving that a given agent does your process correctly is still your job.

It wasn't the only agent news that week. The AI Agents Directory brief for 11 November lists, among others, a seed round for Plurai, which builds simulation infrastructure for testing agents, and a $50 million round for Bay, an AI agent security company. Money is flowing into agent testing and agent security at the same time as catalogs are growing. That's a decent signal of where the industry thinks the risk is.

Why "can we just use it?" is a fit-gap question

Gap analysis, in Wikipedia's words, is "determining, documenting and improving the difference between business requirements and current capabilities." In packaged-software projects, BAs run it as a fit-gap: list the business process step by step, check what the product does at each step, and mark each one as a fit, a configuration, a customization or a gap.

A prebuilt agent is packaged software with a process baked in. Somebody at the vendor decided how a return gets checked, when a claim gets escalated, what a confirmation email says. That decision is in the product whether or not anyone wrote it down for you.

There's a line in Rayan Alghunaim's November post on IIBA's blog, The Requirement No One Asked For, that fits this exactly: "Stakeholders describe needs from inside the process they use today." When you put a prebuilt agent in front of them, their needs suddenly get described from inside the vendor's process instead. The fit-gap is how you notice the switch.

What's different about fit-gap for agents

Fit-gap works well on packaged systems. Three things change when the package is an agent, and they're why I don't think the old template is enough.

1. The product acts, so the cost of a gap depends on undo

In an ERP fit-gap, a gap is mostly an inconvenience. The screen doesn't have the field you need, so a user keeps a spreadsheet on the side. Annoying, recoverable.

An agent takes actions. It issues the refund, sends the email, updates the record. So every step needs one extra column: can this be undone? A wrong draft that a person reviews costs a minute. A wrong refund costs the refund. In my matrix, any step that can't be undone, that the agent would do alone, and that you haven't tested on your own cases is a Gap by default. Not a Configure. Not "probably fine." A Gap.

2. "Fit" from a demo isn't evidence

Packaged-software demos have always been flattering. Agent demos are worse, because the same agent can behave differently on a second run, and the demo shows the run that went well.

The fix is boring and it works: a golden set of your own past cases, with the decision your own people say was right, run against the agent during evaluation, before the contract. If the vendor can't give you a sandbox with realistic data, that's a finding in itself.

3. Some requirements can't live inside the agent

Classic fit-gap asks "does the product do X?" For agents there's a second question: "where is X enforced?" A rule like "never refund more than $200 without approval" can be written in the agent's instructions or enforced by the payment system. Those aren't the same. An instruction is something the agent is asked to follow. A limit in the payment API is something it can't get past. I'd only count the second one as met.

I first heard this put clearly in a reply on one of my Substack Notes, from the account AI Automation Lab: "never" rules only hold when they're enforced outside the prompt, through schema validation and API permission boundaries. It's become a gate in my matrix.

The method: gates first, then fit-gap per step

The matrix I put in the toolkit this week, the prebuilt AI agent fit-gap matrix (a free 6-page PDF), works in two passes.

Pass 1: six gates, pass or fail

Gates aren't scored. If one fails, the answer is "don't adopt yet," however good the step scores look.

# Gate Passes when
G1 One named owner A person on your side owns the agent's behavior and can stop it
G2 Never-rules enforced outside the prompt Every "must never" rule is enforced by permissions, limits or validation
G3 Off switch with names Who can stop it, where work in flight goes, who restarts it, tested in the trial
G4 End state you can check Every action leaves a record in your own system that you can query
G5 Your cases, before the contract You can run your own golden set during evaluation
G6 Data stays where policy says Data, logs and prompts stay where your policy allows, confirmed in writing

If G1, G3 and G2 sound familiar, they're the AI agent job description template turned into a buying checklist. A prebuilt agent still needs a job description. The vendor just wrote the first draft.

Pass 2: each process step gets a fit level

Level Meaning
Fit Does the step as you run it today, proven on your cases
Configure Does it after settings, rules or prompts you control
Extend Needs a custom tool, integration or code
Change our process The agent's way is fine and yours is the odd one out
Gap Can't do it, or can't do it safely

And the decision table, first match wins:

# If... Classify as
1 The step can't be undone, the agent would do it alone, and you have no evidence from your own cases Gap
2 The agent would break a never-rule on this step, or the rule isn't enforced outside the prompt Gap
3 It does the step correctly on your golden cases with no changes Fit
4 It does it correctly after changes you control Configure
5 It needs a new tool, an integration or code Extend
6 Your step differs, the agent's way meets the business rule, and changing costs less than extending Change our process
7 None of the above Gap

Rules 1 and 2 come first on purpose. They stop a good-looking demo from turning a risky step into a Fit.

Puzzle pieces arranged in the shape of a question mark
A prebuilt agent is a box of pieces cut to someone else's picture. Fit-gap is checking each piece against yours before you glue anything. Photo: "Question mark made of puzzle pieces" by Horia Varlan, CC BY 2.0, via Wikimedia Commons.

A worked example: Northstar's returns agent

Here's the filled example from the template, on a fictional client. Northstar Systems is an online retailer. A vendor (I'll call it Vendor R) offers a prebuilt returns and refunds agent. The head of customer operations wants to know if Northstar can switch it on before the holiday returns peak.

The team builds a golden set of 120 past return requests, anonymized, each with the decision the refunds lead says was right. They run it in Vendor R's sandbox.

The gates

Five pass. One fails.

G2 fails because Northstar's rule "never refund above $200 without approval" exists only in the agent's instructions. In the sandbox, the agent did route every over-$200 refund to a person. But the payment API would have accepted a $250 refund from the agent if it had sent one. Vendor R says an API-level limit is on the roadmap.

That single gate decides the outcome. The agent behaved well on this golden set. Northstar still can't rely on an instruction for a rule whose failure costs real money.

The steps

Step What the agent did on Northstar's cases Undo? Level
S1 Find the order Right order in 118 of 120; missed 2 guest orders Yes Configure
S2 Check the 30-day window Counted from ship date; Northstar counts from delivery Yes Configure
S3 Check the item is returnable Correct on every final-sale case Yes Fit
S4 Create the return label Calls a carrier Northstar doesn't use Yes Extend
S5 Auto-refund under $50 One wrong amount: a partial return with a promo code No Gap
S6 Route refunds over $200 to a person Routed all of them, but the limit lives only in the prompt No Gap
S7 Confirmation email Sends its own template Partly Change our process
S8 Flag repeat returners for fraud Not supported Yes Gap

A few rows are worth talking through, because they're where a quick evaluation would go wrong.

S2 is the kind of gap a demo never shows. Ship date versus delivery date is a one-line policy difference. On most returns it doesn't matter. On a return made on day 31 after shipping but day 28 after delivery, the agent rejects a valid return and a customer calls support angry. It's a Configure, and it becomes an acceptance criterion with a test on day 30 and day 31.

S5 failed on one case out of the set, and that's enough. The agent got almost every small refund right. The one it missed was a partial return with a promo code applied, which is exactly the case that's hard for people too. The refund can't be taken back once it's issued. Under rule 1 of the decision table it's a Gap until the team either moves that case type to a person or gets enough clean runs on it to meet a pass mark the refunds lead sets in advance. (How many runs that takes is its own question. I wrote it up this week: zero failures in 20 runs isn't a pass.)

S7 is the row I'd argue about. The agent sends its own confirmation email. Northstar has its own template. Extending the agent to use Northstar's template is a build. Switching to the vendor's template, after a brand review, is a decision. That's "Change our process," and it's often the right call for steps that are reversible and low-stakes.

The decision

The overall rule is first-match too: any failed gate means "don't adopt yet." So that's the answer, with a specific condition attached. Once Vendor R enforces the $200 limit at the payment API, and Northstar sees a $250 refund attempt refused in the sandbox, the matrix gets re-run. S5 then needs its own fix before the agent can act alone.

Notice what the decision isn't. It isn't "the agent is bad." It handled most steps well. It's "we know exactly which three things stand between us and switching it on, and who owns each one." That's the output a sponsor can act on.

A jigsaw puzzle illustration with one piece set apart
Three pieces still don't fit: the payment-API limit, the promo-code refunds and the fraud flag. Image: "Jigsaw Puzzle.svg" by Don F Becker, CC BY-SA 3.0, via Wikimedia Commons.

My take: change your process for reversible steps, never for the ones you can't undo

Here's where I expect some disagreement.

The packaged-software world has a strong tradition of "adopt the standard." Fit-to-standard workshops start from the vendor's process and treat your differences as things to justify. For ERP, there's a good argument for that: custom code is expensive to maintain, and your special process often isn't that special.

For prebuilt agents, I think that tradition is right for half the steps and dangerous for the other half. My rule:

Put simply: be flexible about how the work is done, and stubborn about what the work is allowed to do.

What I'd ask the vendor before the demo

Send these in writing before anyone books a demo. The answers tell you more than the demo will.

  1. Can we run our own anonymized cases in a sandbox before we sign? How many, and with what data?
  2. For each "the agent must never..." rule we give you, where is it enforced? Instructions, tool permissions, or the downstream system's API?
  3. Show us how we stop the agent. What happens to work in flight? Who can restart it?
  4. Where does every action the agent takes get recorded, in a system we can query ourselves?
  5. When you ship a new version of the agent, what changes, how are we told, and can we re-run our golden set before it reaches production?
  6. Which steps of this process did you design for a specific industry regulation? Which did you design because they were simplest to build?

Question 6 is the one vendors find awkward, and it's the most useful. It separates the parts of the vendor's process that carry real knowledge from the parts that are just one way of doing it.

What this method doesn't cover

What I'd do on Monday

If someone on your team has already asked "can we just use the vendor's agent?", this is the week's work.

  1. Write the process down as your team runs it, one step per line, before you look at the vendor's version. If you look first, you'll start describing your process in their words.
  2. Mark the irreversible steps. Money, external messages, records other systems trust. Those are where the gates and rule 1 bite.
  3. Build a golden set of 50 to 150 past cases with the right decision written by the person who owns it. Include the hard ones on purpose.
  4. Send the six vendor questions in writing, before the demo.
  5. Run the matrix with the template. Every Configure and Extend row becomes a requirement. Every "Change our process" row needs the process owner's signature.
  6. Write the acceptance criteria for the Configure rows straight away. My acceptance criteria template has the shape.
  7. Report the decision with conditions, not a yes or no: "don't adopt yet, because G2; here's what would change it, and who owns it."

If the evaluation is for a different kind of AI tool, a copilot or an analytics assistant rather than an agent that acts, the lighter version is in my post on evaluating an AI tool before you buy it. And if the agent will be built rather than bought, start one step earlier with writing requirements an AI agent can build.

Questions for you

If you've run a fit-gap on an agent already, I'd like to hear what your matrix had that mine doesn't. Leave a comment or find me on any of the profiles below.

References

About me

I'm Prachit Parikh, a business analyst. I turn what stakeholders ask for into requirements a team can test, check the data with SQL, and see changes through UAT. I keep free templates, including this week's fit-gap matrix for prebuilt AI agents, in my BA Toolkit on GitHub, and I make short animated explainers on YouTube. Besides this site, I write on Substack, Medium, Hashnode, Dev.to, Blogger and WordPress, and I answer questions on Quora. You can also find me on LinkedIn, Bluesky, Kaggle, Credly, Crunchbase and Gravatar. My case studies are on my portfolio page, and everything is collected on my profile hub.

← More articles by Prachit Parikh