Evaluating AI-Enabled Software: What Businesses Should Verify Before Deployment#

AI Is a System, Not Just a Model#

When people discuss an AI product, they often focus on the model.

But a production AI system can contain many additional components:

User
 ↓
Application
 ↓
Prompt / Instruction Layer
 ↓
Context / Retrieval
 ↓
AI Model
 ↓
Tools / APIs
 ↓
Business Systems
 ↓
User / Automated Action

There may also be:

• databases;
• vector stores;
• authentication;
• logging;
• moderation;
• evaluation systems;
• external APIs;
• human approval;
• monitoring.

The model is only one component.

Therefore, evaluating an AI application requires examining the whole system around the model.

Why Traditional Software Review Is Not Enough#

Conventional software often aims for deterministic behavior.

Given the same input and state, the expected output may be precisely defined.

AI systems can introduce additional variability.

Outputs may depend on:

• model version;
• prompt;
• context;
• retrieved information;
• tool results;
• temperature or other configuration;
• provider behavior.

This does not make AI impossible to evaluate.

It means the evaluation criteria need to reflect the system's actual behavior.

Start With the Intended Use#

Before evaluating an AI system, define what it is supposed to do.

For example:

"The system helps customer-support agents find relevant information."

is very different from:

"The system automatically approves customer refunds."

The consequences of an incorrect output are different.

Questions include:

• Who uses the system?
• What decision does it influence?
• Can the output directly trigger an action?
• What happens when the output is wrong?
• Is human review required?
• What is the acceptable failure rate?
• Which errors are unacceptable?

Model and Provider Dependency#

Document:

• model provider;
• model name/version where applicable;
• API dependency;
• context limits;
• pricing;
• availability assumptions;
• fallback strategy.

Important questions include:

• Can the provider change the model?
• Can the model behavior change without application-code changes?
• What happens if the provider is unavailable?
• How difficult would migration be?
• Does the system depend on provider-specific functionality?

Vendor dependency is not automatically bad.

It should be understood.

Data Flow#

AI systems can move data across several boundaries.

For example:

Customer
   ↓
Application
   ↓
Database
   ↓
Retrieval System
   ↓
AI Provider
   ↓
Application

Review:

• what data enters the model;
• whether sensitive information is included;
• where processing occurs;
• what is retained;
• who can access it;
• what external services receive it;
• how data is deleted.

Technical findings should be separated from legal conclusions.

Privacy and regulatory requirements may require specialist legal review.

Prompt and Instruction Security#

Prompts can contain important system behavior.

Review:

• system instructions;
• developer instructions;
• user-controlled input;
• retrieved content;
• tool descriptions;
• permission boundaries.

One important concern is prompt injection.

A system that retrieves untrusted text must not automatically treat that text as trusted instructions.

Tool Permissions#

An AI system becomes more consequential when it can perform actions.

Examples:

• sending emails;
• creating records;
• deleting data;
• making purchases;
• modifying infrastructure;
• executing code;
• changing permissions.

The key question becomes:

What can the AI cause the system to do?

A useful principle is least privilege.

The AI should receive only the permissions necessary for its intended task.

High-impact actions may also require human approval.

Evaluation#

A system should have a clear definition of acceptable behavior.

Evaluation can use:

• representative test cases;
• historical examples;
• adversarial cases;
• edge cases;
• failure cases;
• domain-specific datasets.

Metrics depend on the application.

Possible measurements include:

• accuracy;
• precision;
• recall;
• task completion;
• groundedness;
• hallucination rate;
• refusal behavior;
• latency;
• cost.

Not every AI system needs every metric.

The important thing is that evaluation should reflect the real purpose of the system.

Grounding and Retrieval#

Many AI applications use external information rather than relying only on model knowledge.

A retrieval-based architecture might look like:

User Question
     ↓
Retrieval
     ↓
Relevant Documents
     ↓
Context
     ↓
Model
     ↓
Answer

Review:

• how documents are selected;
• whether access controls are preserved;
• whether irrelevant data can enter the context;
• whether sources are cited where appropriate;
• how stale information is handled;
• what happens when retrieval finds nothing.

A technically impressive retrieval system can still expose data if authorization boundaries are ignored.

Human Oversight#

Human oversight should be designed around consequences.

Low-risk actions may be automated.

Higher-risk actions may require:

AI Recommendation
       ↓
Human Review
       ↓
Approval / Rejection
       ↓
Action

This is especially relevant when the AI affects:

• money;
• legal decisions;
• employment;
• safety;
• security;
• access;
• sensitive personal information.

The appropriate control depends on the application and applicable requirements.

Failure Handling#

AI systems should have defined behavior when the model:

• times out;
• returns malformed output;
• produces low-confidence information;
• refuses a valid request;
• gives an incorrect answer;
• hallucinates;
• exceeds cost limits;
• becomes unavailable.

Possible responses include:

• retry;
• fallback;
• human escalation;
• safe refusal;
• cached response;
• conventional workflow.

The system should not silently assume that every model response is correct.

AI Observability#

AI systems need visibility into more than server errors.

Depending on the application, useful observability may include:

• model version;
• prompt/version identifier;
• latency;
• token usage;
• cost;
• tool calls;
• retrieval results;
• failure categories;
• evaluation outcomes.

Logging must also respect privacy and data-minimization requirements.

Cost Verification#

AI systems can create variable costs.

Costs may depend on:

• input volume;
• output volume;
• model;
• context size;
• retrieval;
• tool calls;
• retries.

A system should therefore have reasonable cost expectations.

Questions include:

• What is the expected cost per task?
• What happens during unusual usage?
• Are retries bounded?
• Can users trigger expensive operations repeatedly?
• Is there a budget or rate limit?

Model Changes#

An AI provider can change models.

Even if the application's code remains unchanged, behavior may change.

Therefore, important systems should consider:

• model version tracking;
• regression evaluation;
• prompt versioning;
• evaluation datasets;
• rollout procedures;
• fallback models;
• change monitoring.

Security and AI#

AI-specific security concerns can include:

• prompt injection;
• data leakage;
• excessive tool permissions;
• insecure output handling;
• indirect prompt injection;
• model supply-chain dependency;
• untrusted retrieval content.

These concerns should be evaluated alongside conventional application security.

AI security does not replace ordinary security engineering.

What a Practical AI Evaluation Should Produce#

A useful evaluation can contain:

What "AI-Ready" Should Not Mean#

Businesses sometimes use phrases such as:

"AI-powered."

or:

"AI-ready."

These phrases do not explain whether the technology is appropriate.

A more useful evaluation describes actual properties:

• what the AI does;
• what data it uses;
• which model it depends on;
• what permissions it has;
• how its outputs are evaluated;
• how errors are handled;
• how changes are monitored.

Specific technical evidence is more useful than broad AI terminology.

The Neubofy Approach#

AI evaluation fits naturally into Neubofy's broader technical verification and assurance model.

A simplified flow is:

Business Requirement
        ↓
AI Use-Case Definition
        ↓
System Architecture
        ↓
Model + Data + Tools
        ↓
Evaluation
        ↓
Security Review
        ↓
Operational Verification
        ↓
Deployment
        ↓
Ongoing AI Assurance

The exact depth should depend on the consequences of the AI system.

A recommendation assistant and an autonomous system should not receive identical evaluation processes.

The Core Principle#

AI should not be trusted because it is impressive.

It should be trusted to the extent that the system around it provides appropriate evidence, controls, evaluation, and accountability for its intended use.

That means asking:

What can it do?

What can it access?

What happens when it is wrong?

How do we know?

Who remains responsible?

Those questions are more useful than simply asking which model the system uses.

Actionable Checklist#

• Define the AI system's intended purpose.
• Identify the consequences of incorrect outputs.
• Document model and provider dependencies.
• Map data entering and leaving the AI system.
• Review sensitive-data handling.
• Review prompts and instruction boundaries.
• Identify tool permissions.
• Apply least privilege.
• Define representative evaluation cases.
• Include failure and adversarial cases.
• Define human-approval requirements where appropriate.
• Test fallback and failure behavior.
• Monitor meaningful operational signals.
• Track model and prompt changes.
• Review variable AI costs.
• Re-evaluate the system after significant model, data, or architecture changes.