Evaluating AI Applications: From Impressive Demos to Measurable Results

Subtitle: A practical engineering approach to testing, benchmarking, and improving AI-powered software.

An AI application can look impressive during a demonstration. It may answer questions fluently, summarize a document in seconds, or generate a useful piece of code from a short instruction.

But a successful demonstration does not necessarily mean the application is ready for real users.

What happens when the input changes slightly? Does the application still produce a useful answer? Can it distinguish reliable information from an unsupported assumption? Does a new prompt or model version improve the result—or introduce new problems?

These questions lead to one of the most important aspects of AI engineering: evaluation.

Traditional software testing often checks whether a system produces an expected result for a given input. AI applications require additional methods because their outputs may vary, their quality can be subjective, and an answer can be well written without being factually correct.

The challenge is not simply to determine whether the AI works. It is to measure how well it works, where it fails, and whether changes make it better.

1. Why ordinary testing is not enough

Consider a conventional function that calculates the total price of an order.

If the inputs and business rules are defined, we can write tests that verify the exact expected result.

Now consider an AI feature that answers technical questions using a collection of engineering documents.

A response might be correct but expressed differently from the reference answer. Another response might contain the right terminology but incorrectly interpret an important specification. A third might answer confidently even though the documents do not contain enough information.

A simple exact-text comparison cannot reliably distinguish these cases.

AI evaluation must therefore examine several dimensions:

• Correctness: Is the answer factually accurate?

• Relevance: Does it address the actual question?

• Grounding: Are claims supported by the available evidence?

• Completeness: Are important details missing?

• Consistency: Does the application behave acceptably across similar cases?

• Safety: Does it respect the application's restrictions and permissions?

• Operational performance: Is it fast and affordable enough for its intended use?

Not every application needs every metric. The evaluation criteria should reflect the purpose and risk of the feature.

2. Start with a clearly defined success criterion

Before choosing an AI model or measuring its output, define what success means for the user.

For example, suppose an application extracts information from technical documents.

A vague objective would be:

“Use AI to analyze technical documents accurately.”

A more useful objective would specify that the application must extract named fields, preserve units, identify the source of each important value, flag missing information, and avoid inventing values when the document is incomplete.

The second objective is easier to test because it translates a broad ambition into observable behaviour.

For each AI feature, ask:

1. What task is the application expected to perform?

2. What evidence is available to support its answer?

3. Which errors are acceptable, and which are critical?

4. What should happen when information is missing?

5. What result would justify deploying the feature?

A success criterion should be defined before testing begins. Otherwise, it is easy to select whichever examples make the system look good.

3. Build a representative evaluation dataset

One of the most useful steps is to create a collection of realistic inputs with carefully reviewed expected outcomes.

This is often called a golden dataset or evaluation dataset.

For a technical question-answering system, it might contain:

• Common questions with clear answers.

• Questions requiring information from multiple documents.

• Questions with incomplete supporting evidence.

• Questions containing ambiguous terminology.

• Documents with conflicting statements.

• Questions that fall outside the system's intended scope.

• Inputs designed to expose security or instruction-following weaknesses.

Each test case should include the input, the expected behaviour, the relevant evidence, and the criteria used to judge the result.

For example, a test case might ask for a component's operating temperature when the source document does not specify it.

The correct behaviour may be to state that the information is unavailable, rather than generate a plausible temperature.

That is an important distinction: a reliable AI system must know when it does not have enough evidence to answer.

The dataset should represent actual usage patterns as closely as possible. A small collection of carefully selected examples is a useful starting point, but broader testing is needed before making strong claims about production quality.

4. Use different evaluation methods for different tasks

There is no single metric that can measure the quality of every AI application.

The evaluation method should match the task.

AI task	Useful evaluation approach
Classification	Precision, recall, F1 score and confusion matrix
Information extraction	Field-level accuracy and exact-match checks where appropriate
Question answering	Factual correctness, relevance, evidence support and completeness
Document summarization	Coverage of key facts, factual consistency and omission checks
Code generation	Compilation, unit tests, integration tests and security checks
AI agents using tools	Task completion, correct tool selection, permission compliance and side-effect checks
Content generation	Factual review, relevance, completeness, style and editorial review

For example, a classification system that detects critical defects may need high recall because missing a genuine defect is costly. A system that automatically rejects customer submissions may need particularly strong precision to avoid rejecting valid submissions.

The right trade-off depends on the consequences of each type of error.

A single overall score can hide these differences. It is often better to track several metrics and establish minimum thresholds for the most important ones.

5. Evaluate AI agents at the workflow level

For an AI agent, evaluating the final answer alone is not enough.

An agent may need to retrieve information, call an API, interpret the result, make a decision, and perform an action. It can fail at any of these stages.

Consider a software release preparation workflow.

The system must identify changes since the previous release, generate a summary, apply a template, and prepare the result for review.

A useful evaluation should check:

• Were all relevant changes identified?

• Were the changes described accurately?

• Were important security or compatibility changes omitted?

• Was the required template followed?

• Were unsupported claims introduced?

• Did the system respect the approval process?

• Were retries handled without duplicating actions?

This is more meaningful than asking whether the generated release notes sound professional.

It is also useful to separate task quality from workflow reliability. A good summary does not compensate for an application that occasionally skips a mandatory approval step.

6. Automate repeatable evaluations

Once a baseline dataset exists, evaluations can become part of the software development process.

A practical evaluation pipeline can follow these steps:

1. Select a version of the evaluation dataset.

2. Run the current model, prompt, and application configuration against it.

3. Record the generated outputs.

4. Apply deterministic checks wherever possible.

5. Evaluate qualitative criteria using human review or carefully validated evaluation methods.

6. Compare the results against the existing baseline.

7. Investigate regressions before deploying the change.

This makes it possible to compare different prompts, models, retrieval strategies, and application logic under similar conditions.

For example, a new prompt may improve answer completeness while increasing unsupported claims. Without a repeatable evaluation process, that trade-off may go unnoticed.

AI-based evaluators can help assess large volumes of responses, but they should not automatically be treated as ground truth. They can make mistakes too. Their judgments should be validated against human-reviewed examples, particularly for high-impact decisions.

7. Treat every model or prompt change as a potential regression

In conventional software development, regression tests help ensure that a new change does not break previously working functionality.

The same principle applies to AI applications.

A prompt change that improves one type of question may make another type worse. A newer model may produce better summaries but handle structured output less reliably. A retrieval change may improve relevance while introducing irrelevant documents.

Therefore, before replacing a working configuration, compare the new version with the previous baseline.

Record the relevant configuration, including model version, prompt version, retrieval settings, evaluation dataset version, and test results. Where possible, preserve representative failures so they can become permanent regression tests.

This creates a feedback loop in which each discovered failure improves future testing.

8. Measure the operational cost of quality

Accuracy is important, but production systems also have operational constraints.

A model may generate excellent responses but take too long for an interactive application. Another may be fast but require frequent human corrections. A third may be accurate but too expensive at the expected usage volume.

Useful operational metrics include:

• Average and high-percentile response latency.

• Token consumption and cost per request.

• Cost per successfully completed task.

• Retry and timeout rates.

• Human review and correction rates.

• Failure rates across different input categories.

Consider two approaches to the same task.

Approach A has a lower cost per request but frequently requires manual correction. Approach B costs slightly more per request but produces results that are consistently usable.

The second approach may be more economical when the full workflow is considered.

This is why cost per successful outcome can be more informative than the cost of a single model call.

9. Continue evaluation after deployment

Pre-release testing cannot anticipate every real-world input.

After deployment, monitor performance using appropriate privacy and data-governance controls. Review failed tasks, recurring corrections, unexpected inputs, and changes in usage patterns.

If the application serves different user groups or handles different categories of documents, evaluate performance across those categories instead of relying only on an overall average.

A gradual rollout or controlled comparison can help reveal problems before a change reaches every user.

Production monitoring should also include conventional operational signals such as API errors, database failures, queue delays, and integration outages. Not every AI application failure originates in the model itself.

The aim is to establish a continuous cycle:

Evaluate → Deploy carefully → Monitor → Learn from failures → Improve → Evaluate again.

Conclusion: Make quality measurable

AI engineering becomes more disciplined when quality is defined, measured, and reviewed rather than judged by a handful of impressive examples.

Start with clear success criteria. Build a representative evaluation dataset. Select metrics that match the task. Test the entire workflow, not just the final response. Automate repeatable evaluations, investigate regressions, and monitor real-world behaviour.

No single score can prove that an AI application is reliable. But a well-designed evaluation process makes weaknesses easier to detect and improvements easier to verify.

The key question is not simply whether the AI produced a good answer today.

It is whether we have evidence that the application performs well across the situations that matter—and whether we can detect when that performance changes.

That is the difference between experimenting with AI and engineering an AI system that can be trusted.

When AI Gets It Wrong: Designing Systems That Fail Safely

Why reliable AI applications need validation, recovery mechanisms, and clear boundaries—not just intelligent models.

Artificial intelligence is becoming part of everyday software engineering. We are using AI to analyze documents, answer technical questions, generate code, summarize information, and automate business processes.

The capabilities are impressive. But there is an important engineering question that deserves equal attention:

What happens when the AI gives the wrong answer?

In traditional software, we generally expect the same input and conditions to produce predictable results. A database constraint, a validation rule, or a calculation can be tested against an expected outcome.

AI systems introduce another dimension. A model may generate a plausible but incorrect answer, misunderstand an instruction, omit an important detail, or produce a response that looks convincing despite containing an error.

This does not mean AI is unsuitable for production. It means we must design AI-powered applications with failure in mind.

1. A convincing answer is not necessarily a correct answer

Consider an AI-powered application that analyses a software repository and prepares release notes.

The system examines commits, identifies changes, and generates a summary for the release team.

Now imagine that a commit contains a small but important security fix. The AI overlooks it and produces release notes that look professional and complete.

The problem is not the writing quality. The problem is that an important change has been missed.

A similar issue can occur in customer support, document processing, technical research, financial reporting, and many other applications.

The output may be grammatically correct, logically structured, and still be wrong.

Therefore, we should evaluate AI output based on correctness and business impact, not merely presentation quality.

2. Separate AI reasoning from deterministic business rules

One of the most useful architectural principles is to give AI and conventional software different responsibilities.

AI is well suited to tasks such as:

• Summarising large volumes of text.

• Classifying unstructured information.

• Extracting candidate insights.

• Drafting explanations and recommendations.

• Identifying patterns that deserve further investigation.

Conventional application logic should handle tasks such as:

• Authentication and authorisation.

• Database constraints and data integrity.

• Mandatory field validation.

• Permission checks.

• Financial calculations.

• Approval requirements.

• Publishing conditions and transaction boundaries.

For example, an AI model can recommend that a document is ready for publication. The application should still verify that the required approval has been recorded before publishing it.

The model should not be able to bypass that rule simply because it generates a response saying that approval is complete.

This separation makes the overall application easier to test, maintain, and secure.

3. Validate the output before acting on it

AI-generated responses should be treated as proposed data, not automatically trusted instructions.

Suppose an AI component is expected to return a release summary with a title, a list of changes, known issues, and a release identifier.

Instead of accepting arbitrary text, the application can require a defined structure.

For example:

{
  "title": "Release Summary",
  "changes": [
    {
      "description": "Improved login validation",
      "category": "security",
      "source_commit": "abc123"
    }
  ],
  "known_issues": []
}

The application can then validate the response against a schema and apply additional business checks.

Questions worth asking include:

• Are all mandatory fields present?

• Are the values in the expected format?

• Does each reported change correspond to an actual source?

• Are any required items missing?

• Does the proposed action comply with business rules?

Structured output and schema validation reduce formatting errors, but they do not prove that the underlying claims are true. Source verification remains necessary.

4. Design for recovery, not just success

A production AI workflow should have an explicit strategy for handling failures.

Consider a document-processing workflow:

1. Receive a document.

2. Extract relevant information.

3. Ask the AI to classify and summarise it.

4. Validate the generated result.

5. Store the result.

6. Send it for review or continue processing.

What should happen if the model times out at step three? What if it returns invalid data? What if the database operation succeeds but the next step fails?

These are ordinary software engineering questions, and they deserve ordinary engineering discipline.

Useful mechanisms include:

Timeouts: Prevent a slow model response from blocking a workflow indefinitely.

Bounded retries: Retry transient failures a limited number of times, with suitable delays. Repeatedly retrying a fundamentally invalid response will not necessarily fix it.

Idempotency: Ensure that retrying an operation does not create duplicate records, send duplicate messages, or publish the same content twice.

Dead-letter handling: Preserve failed jobs for investigation and controlled reprocessing.

Fallback paths: Route uncertain results to manual review or use a simpler, deterministic alternative where appropriate.

Audit logs: Record the input references, model and prompt versions, validation results, decisions, and failures needed for troubleshooting.

The objective is not to eliminate every failure. It is to ensure that failures remain visible, controlled, and recoverable.

5. Use human approval where the consequences justify it

Not every AI output needs human approval. Requiring someone to review every low-risk classification or internal summary can remove much of the efficiency that automation offers.

However, human review becomes important when an incorrect result could have a significant business impact.

For example:

• Publishing a public statement.

• Approving a sensitive business transaction.

• Making a consequential decision about a person.

• Deploying a production software release.

• Changing access permissions or security settings.

A useful approach is risk-based approval.

Low-risk operations can proceed automatically after validation. Higher-risk operations can require explicit approval, additional evidence, or a second verification step.

The important distinction is that human approval must be a real application-level control. It should not depend solely on the AI remembering to ask permission.

6. Measure failure, not just response time

Many AI prototypes are evaluated using a few successful examples. Production systems need a more systematic approach.

In addition to response time and cost, consider tracking:

• Schema validation failure rate: How often does the model return an unusable response?

• Grounding accuracy: How often are claims supported by the available source material?

• Task completion rate: How often does the entire workflow finish correctly?

• Retry and timeout rate: How frequently do requests fail or need recovery?

• Human correction rate: How often must reviewers modify the result?

• Duplicate action rate: Are retries causing repeated side effects?

• Cost per successful task: What does a genuinely completed task cost, rather than a single model call?

These metrics should be evaluated against representative test cases, including difficult and unexpected inputs.

A system that responds quickly but produces unreliable results may be more expensive in the long run because of rework, operational incidents, and loss of trust.

7. Test the boundaries, not only the happy path

Before deploying an AI feature, test situations where the input is incomplete, contradictory, misleading, or outside the expected scope.

For a document-analysis application, tests might include:

• A document with missing sections.

• Two documents containing conflicting information.

• An unsupported file format.

• A prompt containing instructions that conflict with the application's rules.

• A model response that is valid JSON but contains incorrect claims.

• A temporary model-provider outage.

• A repeated job that attempts to perform the same action twice.

This is where conventional testing and AI evaluation complement each other.

Unit tests can verify deterministic rules. Integration tests can verify system behaviour. Representative evaluation datasets can measure model quality. Adversarial tests can explore how the application behaves under hostile or unexpected conditions.

No single test suite guarantees correctness, but together these practices provide much stronger protection than testing only successful examples.

8. Treat AI as one component in a larger engineering system

An AI application is more than a model and a prompt.

It includes input handling, context retrieval, permissions, business rules, data storage, external integrations, monitoring, and recovery mechanisms.

A useful design principle is:

Let AI propose. Let the application verify. Let controlled workflows execute.

For example, an AI model may recommend a change to a software release. The application can verify the recommendation against repository data, apply release policies, request approval where required, and record the final decision.

This architecture does not eliminate AI errors. It prevents many individual errors from automatically becoming business actions.

Conclusion

The future of AI in software engineering will not be determined solely by which model produces the most impressive response.

It will also depend on how well we build the systems around those models.

Reliable AI applications need clear boundaries, verifiable outputs, controlled permissions, recovery mechanisms, meaningful monitoring, and appropriate human oversight.

These are not obstacles to innovation. They are the engineering practices that allow innovation to move from a demonstration into dependable business software.

When designing an AI-powered application, it is worth asking two questions:

How will the system behave when the AI is right? And, more importantly, what will happen when the AI is wrong?

The answer to the second question is often what separates an interesting prototype from a production-ready system.

Building AI Agents in the Real World: Lessons Beyond the Demo

Why reliable AI automation requires more than a powerful language model.

Introduction

Artificial Intelligence has opened up exciting possibilities in software engineering. We can now use AI to generate code, analyze documents, summarize information, and assist with complex technical tasks.

AI agents take these capabilities a step further by connecting AI models to software applications, APIs, databases, and automated workflows.

However, there is an important difference between demonstrating what AI can do and engineering a system that can perform a business task reliably.

As I explore the practical application of AI agents and workflow automation, one lesson stands out:

The real engineering challenge is not just making AI generate the right answer. It is designing the complete system that turns that answer into a reliable outcome.

1. An AI Model Is Only One Part of the Solution

Consider a simple example: preparing technical release notes for a software application.

A language model can summarize code changes and generate a well-structured document. But to produce useful release notes, the system must first identify the correct repository, branch, previous release, and set of changes.

It must also understand the expected format and distinguish significant enhancements from minor internal modifications.

Even a well-written summary can be misleading if the underlying information is incomplete.

This illustrates an important principle: AI quality depends not only on the model, but also on the quality of the data, the surrounding application logic, and the validation process.

2. From Manual Workflows to Intelligent Automation

Software development involves many repetitive activities beyond writing code.

Preparing release documentation, reviewing changes, organizing testing activities, communicating updates, and coordinating approvals can consume considerable time.

These activities provide practical opportunities for AI-assisted automation.

Imagine a workflow that:

1. Retrieves changes from a configured GitHub repository.

2. Identifies enhancements, bug fixes, and other relevant modifications.

3. Generates release notes using an approved template.

4. Checks the draft against the available change information.

5. Sends the document to an authorized reviewer.

6. Distributes the approved release notes and records the result.

AI can help interpret changes and prepare the document, while conventional software logic manages repository selection, release boundaries, permissions, approval status, and execution records.

The result is a more structured approach to automation than simply asking an AI model to write a document.

3. The Importance of Workflow Orchestration

An AI-powered application often depends on multiple components working together.

The language model may be responsible for generating or analyzing information. APIs provide access to external systems. The application manages business rules, and the workflow engine coordinates the sequence of operations.

Each component has a distinct responsibility.

For example, a scheduled release workflow must determine when to run, which project to process, what information to retrieve, and what to do if a required service is unavailable.

A failure at any stage can interrupt the overall process.

This is why workflow orchestration deserves as much attention as the AI model itself.

The system must handle errors, retry appropriate operations, prevent unintended duplicate actions, and record enough information to investigate failures.

A successful AI response does not automatically mean that the complete workflow has succeeded.

4. Infrastructure Is Part of AI Engineering

Another practical consideration is the environment in which an AI workflow runs.

Automation platforms and AI applications depend on compatible software versions, adequate resources, secure credentials, reliable scheduling, and access to external services.

A workflow that works on a developer's machine may encounter problems when deployed to shared hosting or a production server.

Compatibility issues, resource restrictions, expired credentials, and API failures can all affect reliability.

These are conventional engineering challenges, but they become especially important when automated workflows are expected to operate regularly.

The lesson is straightforward: deployment architecture must be considered from the beginning rather than treated as an afterthought.

The choice between a hosted automation service, self-hosted workflow engine, or custom application should depend on the operational requirements, available infrastructure, maintenance effort, and long-term product strategy.

5. Five Engineering Principles for Reliable AI Agents

Principle 1: Define the Business Process First

Before selecting an AI model, define the task, inputs, expected outputs, business rules, and failure conditions.

A clearly defined workflow is easier to implement, test, and improve.

Principle 2: Provide the Right Context

An AI agent must receive relevant and current information.

For release documentation, this includes the correct code changes, release metadata, project configuration, and documentation standards.

Incomplete context can lead to incomplete or inaccurate output.

Principle 3: Keep Critical Business Rules in Application Code

Operations such as checking permissions, determining release boundaries, enforcing approval requirements, and recording execution status should be governed by explicit software logic.

AI can assist with interpretation and content generation, but critical business rules should not depend solely on a language model's judgment.

Principle 4: Include Human Oversight

Not every operation needs manual intervention, but high-impact actions require appropriate controls.

Release documentation may need technical review. Production deployments may require additional testing and authorization.

The degree of autonomy should reflect the risk associated with the operation.

Principle 5: Measure Outcomes

The success of an AI agent should be evaluated using practical measures such as output accuracy, workflow completion rates, time saved, error frequency, and operating costs.

Generating a document is only one milestone. Delivering a correct, approved, and usable result is the real objective.

6. Why Traditional Software Engineering Still Matters

There is a tendency to view AI agents as a replacement for conventional application development.

I see a different opportunity.

Traditional software engineering provides predictable execution, data integrity, access control, integrations, and operational monitoring. AI adds capabilities for language understanding, flexible analysis, and content generation.

These capabilities complement one another.

For example, an AI system may summarize code changes, but application logic should determine which repository and release are being processed. AI may prepare a report, but the application should enforce who can approve and distribute it.

The strongest systems combine AI where flexibility is useful with deterministic programming where precision and control are essential.

7. Moving From Experiments to Dependable Applications

Building a proof of concept is a useful starting point, but production readiness requires additional work.

A dependable AI application must be tested with realistic inputs, unexpected conditions, incomplete information, and external service failures.

Its security permissions must be reviewed. Its outputs must be validated. Its operating costs and performance must be measured.

It should also provide clear execution records so that developers and administrators can understand what happened when a task fails.

These requirements do not diminish the value of AI. They are what make AI useful in real business environments.

Conclusion

AI agents create exciting possibilities for software development and business automation. They can reduce repetitive work, help process complex information, and coordinate activities across multiple applications.

But the value of an AI agent cannot be measured by the quality of its generated response alone.

It depends on the entire engineering system: data quality, application architecture, workflow orchestration, security, validation, and recovery.

The real measure of an AI agent is not what it can generate in a demonstration, but what the complete system can reliably accomplish in practice.

For software engineers and technology leaders, this is where the most interesting work begins: combining the flexibility of AI with the discipline of sound software engineering to solve real problems.

Context Engineering: The Missing Piece in Building Smarter AI Agents

Why the future of Artificial Intelligence depends not only on smarter models, but also on giving them the right information at the right time.

Introduction

Artificial Intelligence is evolving rapidly. Large Language Models (LLMs) can generate code, summarize documents, answer complex questions, and assist with business decisions. AI agents extend these capabilities by using tools, accessing information, and executing multi-step workflows.

Yet, despite these advances, AI systems can still make mistakes, misunderstand requirements, or produce responses that are technically convincing but factually incorrect.

Why does this happen?

Sometimes, the problem is not the intelligence of the model. The problem is the information available to it when it makes a decision.

Imagine asking a software engineer to fix a bug without providing the source code, error logs, application architecture, or business requirements. Even an experienced engineer would struggle.

AI systems face a similar challenge.

This brings us to an important concept in modern AI engineering: Context Engineering.

1. What Is Context Engineering?

Context engineering is the process of selecting, organizing, and supplying the information an AI model needs to perform a particular task effectively.

This information may include:

• System instructions and business rules.

• Relevant documents and knowledge bases.

• User preferences and conversation history.

• Database records and current application data.

• Results returned by external tools and APIs.

• Previous actions, decisions, and workflow status.

The objective is not to provide as much information as possible. It is to provide the most relevant and reliable information for the task.

A well-designed AI system must determine what information is needed, where to obtain it, how to validate it, and when to make it available to the model.

This is particularly important for AI agents that operate across multiple applications and business processes.

2. Prompt Engineering Versus Context Engineering

Prompt engineering and context engineering are related, but they address different aspects of AI application design.

Prompt engineering focuses primarily on instructions: explaining the task, defining the expected output, and setting constraints.

Context engineering takes a broader view. It considers the complete information environment in which the model operates.

For example, consider an AI agent responsible for preparing software release notes.

A prompt might say:

"Prepare release notes using the approved company template."

That instruction is useful, but it is not sufficient.

The agent also needs the relevant GitHub commits, the previous release version, the current release number, the company's formatting standards, and any applicable exclusions.

Without this information, the agent may produce generic notes or overlook important changes.

The prompt explains what to do. Context engineering helps provide the information needed to do it correctly.

3. Why More Information Does Not Always Mean Better AI

One might assume that giving an AI model access to every available document would improve its performance.

In practice, excessive information can create new problems.

Irrelevant documents may distract the model. Outdated records may conflict with current information. Long conversation histories may contain obsolete instructions. Unverified data may lead to misleading conclusions.

More information can also increase processing costs and response times.

Effective context engineering therefore requires selection and prioritization.

For every task, the system should identify the information that is relevant, current, authoritative, and permitted for use.

For example, a customer support agent should retrieve the applicable product documentation and the correct customer's records rather than indiscriminately loading the entire company's database.

The quality of context matters more than its volume.

4. Five Building Blocks of Effective Context Engineering

A. Instructions and Business Rules

An AI agent must understand its responsibilities, constraints, expected output, and permitted actions.

Instructions should clearly define what the agent can do and what requires human approval.

B. Retrieval of Relevant Information

AI agents often need access to information beyond what was included in their original instructions.

Retrieval-Augmented Generation (RAG) is one approach. It retrieves relevant information from documents or knowledge sources and supplies it to the model when needed.

A technical support agent, for example, can retrieve the latest product manual before answering a question.

Retrieval quality is critical: irrelevant or outdated information can reduce answer quality rather than improve it.

C. Memory and Historical Context

Some workflows require information from previous interactions.

An AI agent might need to remember a project's configuration, a user's preferences, previous decisions, or the outcome of an earlier task.

However, storing every interaction indefinitely is not necessarily useful.

A good memory design distinguishes between durable facts, temporary conversation context, and historical execution records. It should also respect privacy, access controls, retention policies, and data ownership.

D. Live Data and Tool Results

Business information changes continuously.

An AI agent preparing a release report needs current repository information. A scheduling agent needs the latest schedule. A reporting agent needs up-to-date records.

Connecting an AI system to reliable APIs and databases allows it to use current information instead of relying entirely on its training data.

The system should also validate the returned data and handle unavailable services gracefully.

E. Evaluation and Feedback

Context engineering should be treated as an ongoing engineering discipline.

Developers should evaluate whether the selected information actually improves results, whether important facts are missing, and whether the model uses the supplied context correctly.

Feedback, testing, monitoring, and controlled updates help improve the system over time.

5. A Practical Example: Building a Smarter AI Release Agent

Consider a software company that releases updates for multiple products.

A release agent must prepare accurate release notes every week. Simply connecting an LLM to GitHub is not enough.

The agent needs a carefully assembled context.

Step 1 — Identify the release: Determine the project, repository, branch, previous release, and target release.

Step 2 — Retrieve changes: Collect relevant commits and change information from the correct comparison range.

Step 3 — Apply business rules: Exclude irrelevant changes and follow the company's release-note template.

Step 4 — Assemble context: Combine the commit information, project details, previous release notes, and applicable instructions.

Step 5 — Generate and validate: Prepare the release notes and check them against the available source information.

Step 6 — Obtain approval: Submit the draft to an authorized reviewer before distribution.

Step 7 — Record the result: Store the final notes, approval status, and execution history.

Notice that the language model is only one component of this process.

The surrounding application determines which project data is retrieved, how the workflow progresses, what the model is allowed to do, and how success is verified.

This is the difference between experimenting with AI and engineering a dependable AI-powered application.

6. Context Engineering in AI Content Creation

Context engineering is equally valuable for content publishing.

Suppose an AI agent must prepare a daily technical article for an engineering audience.

Without appropriate context, the agent might repeatedly cover the same subject, use an unsuitable writing style, or publish content that does not match the author's interests.

A better system can provide:

• The author's preferred writing style.

• Previously published articles and topics.

• The intended audience.

• The selected subject and supporting research.

• Relevant facts and source references.

• Platform-specific formatting requirements.

• Approval and publishing instructions.

The agent can then prepare a long-form blog article and a separate LinkedIn post from the same approved subject.

The blog can contain detailed explanations and examples, while the LinkedIn version can use shorter paragraphs, engaging openings, and a discussion question.

The key is that both outputs are generated from a shared understanding of the subject, author, and publishing requirements.

7. Security and Governance Must Be Part of the Design

Context engineering also introduces important security responsibilities.

An AI system should not receive unrestricted access to every document, database, or conversation simply because the information might be useful.

Enterprise applications must enforce access permissions before retrieving information. Sensitive data should be protected, and instructions or content originating from untrusted sources must not automatically be treated as authoritative.

Organizations should also maintain appropriate logs, validate important outputs, and prevent retrieved content from overriding system-level security policies.

For multi-user AI platforms, context must be isolated according to the user's permissions and organizational boundaries.

These controls become especially important when an agent can act on the information it retrieves.

8. The Future of AI Engineering

As AI models become more capable, the surrounding application architecture will remain essential.

Successful AI systems will need more than a good model and a well-written prompt. They will require reliable data access, effective retrieval, useful memory, workflow orchestration, evaluation, monitoring, and governance.

This is an important opportunity for software engineers.

Traditional software engineering provides the foundation for predictable operations, permissions, integrations, and data management. AI provides flexible language understanding and generation.

Context engineering connects these capabilities by ensuring that the AI system has the information needed for its assigned task.

Conclusion

The development of AI agents is moving beyond simply choosing a powerful language model.

We must also understand the information that model receives, how that information is selected, whether it is current and trustworthy, and how it influences the resulting actions.

Context engineering provides a practical way to address these challenges.

Whether we are building software release agents, technical research assistants, customer support systems, or automated content publishing platforms, the same principle applies:

Better AI outcomes begin with better context.

The future of AI engineering will not be determined by model intelligence alone. It will also depend on how effectively we design the systems, data, memory, and workflows surrounding those models.

Further reading:

• Microsoft — The Economics of Agent Optimization: Context Engineering

• Databricks — Memory Scaling for AI Agents

• Splunk — Context Engineering for AI Agents in Production

AI Agents and Human Intelligence: Why the Future Is Collaboration, Not Replacement

Artificial Intelligence is changing how we work. But the greatest opportunity may not be replacing human intelligence—it may be combining human judgment with machine capabilities.

Introduction

Artificial Intelligence has entered a new phase.

Generative AI can write articles, generate code, analyze information, and answer complex questions. Agentic AI extends these capabilities by enabling systems to interact with software applications, use tools, and execute multi-step workflows.

Naturally, this raises an important question:

Will AI eventually replace human intelligence, or will humans and AI become more powerful by working together?

My view is that the greatest long-term opportunity lies in collaboration.

AI can help us process information faster, automate repetitive tasks, and explore possible solutions. Humans bring experience, creativity, contextual understanding, ethical judgment, and responsibility.

Combining these strengths can transform how businesses operate and how software is developed.

1. Understanding the Strengths of Humans and AI

To understand the future of AI, we must recognize that human intelligence and artificial intelligence have different strengths.

AI systems can process large amounts of information, identify patterns, generate alternatives, and perform repetitive digital tasks at impressive speed.

Humans bring a different set of capabilities:

• Understanding business goals and organizational context.

• Applying experience and practical judgment.

• Recognizing social and ethical implications.

• Making decisions when information is incomplete.

• Taking responsibility for important outcomes.

AI is not infallible, and humans are not error-free either. The opportunity lies in designing workflows where each contributes appropriately and where important outputs are verified.

2. From AI Assistant to AI Collaborator

The first wave of Generative AI encouraged people to interact with intelligent systems through prompts.

A user asks a question, receives an answer, and decides what to do next.

AI agents introduce a different model.

An agent can be designed to break a task into steps, use approved tools, gather information, generate outputs, and report the results.

Consider a software engineer working on a new product release.

Instead of manually reviewing every commit to prepare release notes, the engineer could use an AI agent to analyze repository changes and create a first draft.

The engineer then reviews the draft, corrects omissions, confirms the technical details, and approves the final version.

The agent handles much of the repetitive work. The engineer retains responsibility for the quality and correctness of the release communication.

This is collaboration in practice.

3. The Impact on Software Engineering

Software engineering is an excellent example of how human-AI collaboration can improve productivity.

AI agents can assist with:

Requirements analysis: Organizing requirements and identifying potential gaps.

Software development: Generating code suggestions and implementing well-defined tasks.

Testing: Proposing test cases and helping analyze failures.

Documentation: Preparing technical documentation and explaining code changes.

Release management: Summarizing commits, preparing release notes, and coordinating approval workflows.

However, these capabilities do not eliminate the need for engineering expertise.

Generated code must be reviewed. Tests must be meaningful. Security must be evaluated. Architectural decisions must account for maintainability, performance, and business requirements.

Recent research into human-agent collaboration in software development also highlights an important distinction: faster generation does not automatically guarantee that a proposed code change will be accepted or integrated successfully.

The lesson is clear: productivity depends on how effectively humans and AI work together, not simply on how much code AI can produce.

4. A Practical Example: An Intelligent Software Release Workflow

Imagine a software company managing several applications and publishing releases every week.

The conventional process may involve developers, release coordinators, testers, and project managers performing repetitive coordination activities.

An AI-assisted release workflow could operate as follows:

Step 1 — Analyze: Retrieve changes from the configured GitHub repository.

Step 2 — Summarize: Identify important enhancements, bug fixes, and other changes.

Step 3 — Generate: Prepare release notes using the company's approved template.

Step 4 — Validate: Check the generated information against the available commits and release records.

Step 5 — Review: Send the draft to an authorized reviewer.

Step 6 — Complete: After approval, distribute the release notes and perform any separately authorized follow-up actions.

The AI agent reduces repetitive effort, while the human team provides technical judgment and authorization.

This approach can make release processes more consistent without removing accountability.

5. Human Oversight Is a Design Principle, Not a Weakness

Some people believe that an AI agent is truly intelligent only when it can operate without human intervention.

I believe this is the wrong way to measure success.

The appropriate degree of autonomy depends on the task.

A low-risk activity, such as categorizing documents, may be automated almost entirely after suitable testing.

A high-impact activity, such as deploying a critical production change or sending a sensitive communication, may require explicit approval.

A well-designed AI system should therefore distinguish between tasks it can execute independently and tasks that require authorization.

This requires secure permissions, clear business rules, activity logs, output verification, and reliable failure handling.

Human oversight does not prevent automation. It helps make automation dependable.

6. The Rise of Human-AI Teams

In the future, a single professional may work with several specialized AI agents.

A software engineer might use one agent to analyze requirements, another to generate test cases, and another to prepare documentation.

A business leader might use AI agents to organize reports, analyze operational information, and prepare decision-support summaries.

A technical content creator might use agents to research topics, draft articles, and prepare content for different publishing channels.

However, coordinating multiple agents requires more than connecting several AI models. Their responsibilities, data access, communication, and outputs must be managed carefully.

The human role increasingly becomes one of defining objectives, establishing constraints, evaluating results, and directing the overall process.

This creates opportunities for professionals who understand both their domain and the practical application of AI.

7. What Businesses Should Do Today

Organizations do not need to wait for a completely autonomous AI future.

They can begin with small, well-defined workflows.

First, identify repetitive tasks that consume significant employee time.

Second, select a workflow where the expected benefits can be measured.

Third, introduce AI with appropriate access controls and human approval.

Fourth, evaluate the results using clear metrics such as time saved, error rates, output quality, and cost.

Finally, expand automation only when the workflow demonstrates reliable performance.

This gradual approach allows organizations to gain practical experience while managing risks.

Conclusion

Artificial Intelligence is changing the relationship between humans and technology.

The question is no longer simply whether AI can perform a particular task. We must also ask how AI can help people perform their work more effectively, responsibly, and creatively.

The strongest applications will combine machine capabilities with human experience, sound software engineering, and appropriate governance.

For software professionals, entrepreneurs, and business leaders, this represents an opportunity to rethink how work gets done.

The future is not necessarily humans versus AI. It is humans working intelligently with AI to achieve outcomes neither could achieve as effectively alone.

The real competitive advantage will belong to those who learn how to build, manage, and work alongside intelligent systems.

---

Further reading:

• Google Research — Towards AI as a Collaborative Partner

• World Economic Forum — Using AI Agents in Organizations