Your Agent Does Not Need More Knowledge: Separate Facts, Judgment, and Action

Your Agent Does Not Need More Knowledge: Separate Facts, Judgment, and Action

An enterprise agent has accumulated thousands of documents.

It can answer broad product questions. Yet when a user asks it to complete a real task—diagnose an equipment fault, export the right report, create a service ticket, or handle an interrupted voice request—it still fails.

The usual response is to add more knowledge.

More manuals. More FAQs. More chunks. More prompt instructions. More agents.

That often makes the system harder to retrieve from, harder to test, and harder to maintain without fixing the workflow.

The underlying problem is architectural:

Facts, judgment, and action have been placed in the same container.

A knowledge base can provide evidence. It should not be expected to own every contextual decision, collect every missing field, call every system, manage every state transition, and enforce every operational boundary.

Forward Deployed Engineers encounter this failure because they work where the agent meets real users, real data, and real systems. Their job is not to maximize the amount of content available to the model. It is to determine what the agent must know, decide, do, verify, and escalate for a bounded business outcome.

This guide turns three anonymized field cases into a reusable architecture and delivery method.

The knowledge-volume trap

A growing knowledge base can coexist with flat production performance for five reasons.

1. Documents are organized for authors, not users

Enterprise content is commonly grouped by product line, department, document chapter, or publishing owner.

Users do not arrive with that structure in mind. They arrive with a task:

• find a function;
• export a specific dataset;
• diagnose an abnormal state;
• change a configuration;
• create a ticket;
• or recover from an exception.

Retrieval can be technically correct while still returning material that is too broad, duplicated, or detached from the next action.

2. Stable facts and contextual judgment are mixed

“This button opens the export panel” is a stable fact.

“The user probably means the daily exception report rather than the summary report” is a contextual judgment.

When both are written as undifferentiated knowledge, the agent has no clear contract for deciding which statement is evidence and which is a rule for interpreting the current request.

3. The workflow is hidden inside prose

A manual may describe five steps, but the real task may require:

• checking the user’s permission;
• asking for a missing date range;
• calling a system;
• waiting for a response;
• validating the result;
• and escalating when the result is incomplete.

Retrieval does not automatically turn prose into reliable state management.

4. Exceptions live in people

Experienced operators know that a particular alarm often indicates a replaceable part, that two similar requests use different system paths, or that a caller has not finished speaking.

Those distinctions may never appear in the product manual. Adding more of the same documentation cannot recover tacit judgment that was never captured.

5. Offline answers are confused with production outcomes

An agent may answer a test question correctly while users still abandon the conversation, transfer to a human, correct the ticket, or repeat the request.

The system looks knowledgeable but does not complete the job.

The four-layer enterprise agent architecture

The source cases suggest a three-part separation: knowledge, judgment, and action. Production delivery benefits from adding a fourth layer: control.

This is not a rule that every fact belongs in vector search or every judgment belongs in a prompt. The correct implementation depends on stability, risk, latency, explainability, and ownership.

The essential principle is separation of responsibility.

Facts: what is true?

The facts layer should answer questions such as:

• What does this function do?
• Which fields are required?
• Which product version supports it?
• What is the documented diagnostic step?
• Which policy or manual is authoritative?

Useful knowledge units are usually aligned to a user task, scoped to one responsibility, and accompanied by metadata such as product, version, role, validity period, and source.

Judgment: what does this situation mean?

The judgment layer should determine:

• which task the user is trying to complete;
• whether the expression is complete;
• which evidence is relevant;
• whether the current case matches a known exception;
• whether confidence is sufficient;
• and whether to proceed, clarify, abstain, or escalate.

Judgment can use a model, rules, or both. High-risk decisions may require explicit policy and human review rather than unconstrained model reasoning.

Action: what must happen next?

The action layer should:

• collect missing information;
• call the correct system;
• transform and validate inputs;
• create or update a record;
• wait for an asynchronous result;
• handle retries and timeouts;
• and route unresolved cases to a person.

An agent that can explain a workflow but cannot perform or reliably coordinate it remains an information assistant.

Control: how do we keep it safe and measurable?

The control layer supplies:

• authentication and least-privilege authorization;
• input and output schemas;
• approval gates;
• idempotency for write operations;
• timeouts, retries, and fallbacks;
• trace and error capture;
• versioning and rollback;
• evaluation datasets;
• and release thresholds.

OpenAI’s current agent documentation distinguishes knowledge search, custom loops, tools, handoffs, guardrails, approvals, state, and tracing as different application concerns. Microsoft’s architecture guidance similarly recommends using the lowest complexity that reliably meets the task and warns that additional agents introduce latency, coordination cost, and new failure modes.

Case 1: equipment support needed a decision tree, not more chat logs

The situation

An equipment manufacturer wanted to improve abnormal-condition support and after-sales diagnosis.

The team had large volumes of historical conversations, but the records showed only what a support agent asked. They did not reveal why the question was asked, which evidence changed the diagnosis, or how the final decision was made.

Traditional intent recognition and slot filling were also a poor match for complex fault isolation.

The team disagreed about the most useful evidence. The FDE believed screenshots of equipment state were important; the customer expected users to send videos. Discussion alone did not resolve the question.

The FDE intervention

The FDE changed the discovery method:

1. bring domain experts into a short field co-design session;
2. decompose frequent faults into a diagnostic tree;
3. define the evidence, steps, possible causes, and resolution for each branch;
4. test real screenshots and other modalities at the scene;
5. rewrite the knowledge structure around judgment fields rather than conversation transcripts.

The resulting unit was no longer “a previous answer.” It became:

user symptom
→ required evidence
→ diagnostic step
→ possible cause
→ recommended resolution
→ escalation condition

The presentation reports that this deployment improved the independent handling rate by about 30 percentage points and reduced the early human-transfer rate by nearly half. These figures are case-specific source claims, not general performance promises.

The reusable lesson

For diagnostic work, knowledge should encode the evidence required for a judgment—not merely store the final answer.

Multimodal input should also be selected by the information needed for the decision. “Video feels richer” is not enough. If a screenshot exposes the relevant equipment identifier and state faster and more reliably, it may be the better input.

Case 2: a larger customer-service knowledge base produced no better outcome

The situation

A software support agent had operated for a long time and accumulated extensive content. Its independent handling rate remained low.

Typical failures included:

• answering “How do I export data?” but failing on “How do I export data for a specific period?”;
• requiring another knowledge entry for every paraphrase;
• duplicating the same instructions across many fragments;
• mixing stable facts, clarification rules, exception handling, and multi-step operations;
• omitting real usage details known only by experienced users.

The knowledge base had become both a reference library and an attempted workflow engine.

The FDE intervention

The FDE first stopped adding content and reorganized the system by responsibility.

Stable facts moved into task-oriented knowledge

Content was reorganized from “one entry per phrasing” to “one task page per user job.” Duplicates and unnecessary small-talk entries were removed, and real operating details were added.

Clarification and judgment moved into instructions

When the request was ambiguous, the agent first identified the task type and requested the missing information instead of retrieving another near-duplicate answer.

Information collection and system calls moved into workflows

Tasks that required parameters, API calls, or multi-step execution were represented as workflows rather than prose.

Bad cases became regression assets

For every production failure, the team recorded:

• the user expression;
• the intended task;
• the selected knowledge, prompt, tool, and workflow;
• the observed wrong behavior;
• the desired behavior;
• whether the issue was fixed;
• and which regression set should contain the case.

Release moved through limited traffic

The team first opened a channel small enough to review all cases in a day. The FDE analyzed failures, adjusted the correct layer, reran regression tests, and expanded traffic only after the results stabilized.

The source presentation reports that the knowledge volume was reduced by roughly half and that the independent handling rate later improved by about 20 percentage points. Again, the result belongs to this case and should not be generalized without context.

The reusable lesson

Knowledge should be organized around what users are trying to accomplish, not how many ways they can phrase a question.

A compact, task-aligned knowledge base can outperform a larger fragmented one because retrieval has a clearer job and the rest of the system no longer asks it to perform orchestration.

Case 3: three recurring voice-agent failures became three product capabilities

The situation

A voice service project experienced three repeated problems:

1. real-time ticket creation was slow and inaccurate because the complete record depended on the entire conversation;
2. several business scenarios were packed into one agent, causing prompt growth, cross-scenario interference, and expensive regression testing;
3. the agent responded too quickly when users paused, repeated themselves, corrected wording, or used fillers, interrupting incomplete expressions.

These appeared to be separate project bugs. Field analysis showed three reusable system gaps.

Capability 1: separate in-call interaction from after-call processing

During the call, a low-latency model handled conversation, clarification, information collection, and temporary recording.

After the call, a stronger model processed the complete transcript, extracted standardized ticket fields, and created the record.

This architecture exchanged a single overloaded real-time path for two paths with different latency and accuracy requirements.

Capability 2: separate routing from scenario execution

A routing agent identified intent and sent the request to the correct scenario. Each scenario agent owned a shorter prompt, narrower tools, and an isolated regression surface.

This should not be interpreted as “more agents are always better.” Separation is justified when responsibilities, tools, policies, or test boundaries are materially different. Otherwise, extra agents only increase orchestration complexity.

Capability 3: separate quick acknowledgment from business execution

A small, fast component recognized pauses, fillers, and incomplete expressions and returned short acknowledgments such as “I’m listening.”

The business agent waited until an expression-completion check indicated that enough information was available to proceed.

The source reports fewer interruptions and invalid answers, an improvement in scenario-flow accuracy from the 70% range to around 90%, and higher ticket accuracy after separating call-time and post-call work. These remain anonymized source results, not transferable benchmarks.

The reusable lesson

Productization begins when the team can name the recurring capability:

• low-latency ticket creation;
• scenario decoupling;
• expression-completion detection.

The field problem becomes a product capability only after its input, output, trigger, boundary, configuration, validation metric, and unsupported conditions are explicit.

A practical decomposition workshop

Use a real failed task, not a generic architecture discussion.

Step 1: write the business loop

user and trigger
→ desired task
→ required evidence
→ system decision
→ action
→ business result

If the team cannot agree on the loop, it is too early to redesign the agent.

Step 2: identify every decision

For each decision, ask:

• What evidence is needed?
• Is the rule stable or contextual?
• Who can correct it?
• What happens when confidence is low?
• Is the decision reversible?

Step 3: classify each responsibility

Step 4: redesign knowledge around user jobs

Each knowledge unit should ideally have:

• one user job or capability;
• clear scope and prerequisites;
• stable facts and operating steps;
• version and ownership metadata;
• source traceability;
• and links to the relevant workflow or policy.

Do not create a separate entry only because the user used a different phrase.

Step 5: design the smallest executable path

Choose one bounded task that can run end to end. Keep risky writes approval-gated. Make the fallback visible.

Step 6: build the first evaluation set

OpenAI’s evaluation guidance recommends defining desired behavior, running representative test inputs, analyzing results, and iterating. For a production agent, include:

• common successful cases;
• paraphrases;
• missing-information cases;
• ambiguous requests;
• known bad cases;
• conflicting or stale evidence;
• tool and API failures;
• permission failures;
• and cases that should escalate.

Step 7: release through controlled traffic

Start with a segment whose traffic can be fully reviewed. A credible gray release is not simply “10% of users.” It specifies:

• who is included;
• which tasks are enabled;
• which actions remain gated;
• who reviews failures;
• the rollback condition;
• and the evidence required for expansion.

Step 8: turn production failures into regression tests

Every meaningful Badcase should change at least one durable asset:

• knowledge;
• instruction or policy;
• workflow;
• tool contract;
• guardrail;
• evaluation set;
• monitoring rule;
• or product requirement.

If the fix exists only in a meeting note, the system has not learned.

The Badcase record an FDE should keep

A useful failure record is compact but diagnostic.

case_id: bc-2026-001
user_job: export-period-data
input: "Export last month's exception data"
expected_behavior:
  - confirm business unit if missing
  - request or infer the approved date boundary
  - call the authorized export tool
observed_behavior:
  - retrieved a generic export FAQ
failure_layer: workflow
severity: medium
root_cause: required parameters were represented only in prose
fix:
  - add structured workflow inputs
  - keep the FAQ as stable operating guidance
regression_set: export-workflows-v3
release_status: limited-traffic
owner: support-agent-team

The failure_layer field matters. Without it, every failure becomes “a prompt problem,” and the team keeps editing the wrong component.

Measure completion, not just answer quality

Use a metric stack.

Google Cloud’s production guidance recommends comparing evaluation and production distributions, monitoring performance decay, capturing production outputs, and continuously evaluating them against feedback or established ground truth.

One metric cannot represent the entire system. A high answer score can hide a broken tool call. A high automation rate can hide harmful non-escalation. A short latency can hide premature interruption.

When should a field fix become a product capability?

Ask five questions:

1. Does the same problem recur across customers or scenarios?
2. Can the capability’s input, output, trigger, and boundary be stated clearly?
3. Can it be configured rather than rewritten?
4. Can its effect be measured independently?
5. Does moving it into the product reduce future delivery cost or risk?

If the answer is mostly no, keep it as customer configuration or a delivery asset.

If the answer is yes, create a product proposal with:

• repeated field evidence;
• target users and scenarios;
• capability contract;
• configuration surface;
• evaluation set;
• operating requirements;
• unsupported cases;
• and expected leverage.

Product feedback is not “customers asked for this.” It is evidence that a recurring field pattern deserves a reusable system boundary.

What the FDE actually owns

Across the three cases, the FDE did not merely tune prompts.

The role:

• found the person who understood the real judgment;
• observed the actual workflow instead of relying on manuals;
• chose the evidence needed for a decision;
• separated knowledge, judgment, action, and control;
• designed the Agent and system integration;
• constructed tests and Badcase loops;
• managed limited rollout;
• measured production outcomes;
• and converted repeated problems into product capabilities.

That is why the project is not complete when the knowledge base is populated or the demo runs.

It is complete enough to expand only when the bounded workflow works with real users, failure is observable, metrics meet the agreed threshold, and the team knows what learning should carry into the next deployment.

A starter project for aspiring FDEs

Build one narrow support workflow.

1. Choose a real task with a clear boundary.
2. Agree on one business result with a domain partner.
3. Map the current workflow, including exceptions.
4. Create a task-oriented knowledge unit.
5. Put clarification and judgment in an explicit instruction or policy.
6. Put information collection and execution in a workflow or tool.
7. Add validation, logging, escalation, and rollback.
8. Build a representative evaluation set.
9. Run a limited release or realistic simulation.
10. publish the artifacts and retrospective without exposing private data.

The portfolio evidence should include the workflow map, responsibility matrix, architecture, evaluation set, Badcase examples, release decision, metrics, and productization proposal.

Frameworks will change. The ability to determine what belongs in knowledge, judgment, action, and control will remain valuable.

Key takeaways

1. A larger knowledge base is not automatically a better agent.
2. Organize knowledge around user jobs rather than document chapters or phrasings.
3. Separate stable facts, contextual judgment, executable action, and operational control.
4. Use real workflows and domain experts to recover tacit decision logic.
5. Treat Badcases as regression assets, not anecdotes.
6. Release through controlled traffic with explicit expansion and rollback criteria.
7. Measure business completion together with model and system quality.
8. Productize repeated field patterns only when their boundaries and evidence are clear.

The goal of an FDE is not to build an Agent that knows the most. It is to build an Agent that can complete a bounded job, show its evidence, behave safely, and improve from production reality.

References

1. User-supplied presentation, “AI Agent落地之FDE实战分享,” dated July 30, 2026. Anonymized case claims were used as source material and explicitly qualified in this article.
2. OpenAI, Working with evals
3. OpenAI, Agents SDK
4. OpenAI, Using tools
5. Microsoft Azure Architecture Center, AI agent orchestration patterns
6. Microsoft Cloud Adoption Framework, Single agent or multiple agents
7. Google Cloud Architecture Center, Deploy and operate generative AI applications