Context Engineering Starts Before the Prompt

An AI assistant is asked: Does this order qualify for a refund?

A well-written prompt can explain how to assess eligibility. It cannot identify the order, establish which policy version applies, or determine whether an exception belongs to this customer. Those facts must be found and connected before the model can use them.

That work is context engineering. It includes prompt design, but it starts earlier: with how a system represents knowledge, resolves evidence, selects relevant information, and prepares it for inference.

The prompt is not the foundation of a reliable answer. It is the final interface to a context that has already been engineered.

Retrieval finds candidates; context requires judgment

Retrieval-augmented generation gives a model access to information outside its parameters. In their original RAG study, Lewis and colleagues combined a language model with a retriever and an external document index, evaluating the approach on knowledge-intensive tasks. (Lewis et al., NeurIPS 2020)

But retrieval alone does not settle whether a passage is authoritative, current, or applicable. Imagine that a search returns both a current refund policy and an older exception for a similar product. Their wording may be close to the question, while only one governs this order.

The system must resolve more than textual similarity:

• Which order and customer does the request refer to?
• Which item, region, or transaction is involved?
• Which policy applies to that item and date?
• Does another source contradict the policy or record an exception?
• Where did each fact come from, and when was it valid?

These are questions about meaning and relationships. They shape what should enter the model’s context.

Structure meaning before selecting text

A useful representation can make the relevant relationships explicit:

• Entities: the order, customer, product, policy, and exception
• Relationships: the order belongs to a customer; a policy governs a product in a region
• Time and constraints: policy validity dates, return windows, and applicable conditions
• Provenance: the source, record, or passage supporting each fact
• Uncertainty: missing identifiers, conflicting sources, or unresolved exceptions

A knowledge graph or ontology may suit some systems. Typed records, relational data, and policy rules may suit others. The representation should fit the task; adding structure does not automatically make the underlying data complete or correct.

Research on GraphRAG illustrates one way to use relationships in retrieval. In a 2024 preprint, Edge and colleagues described building an entity graph and community summaries to answer global questions over a document collection. They reported improvements over a conventional RAG baseline for comprehensiveness and diversity on a specific class of questions and datasets. That is a bounded research result, not evidence that graph-based retrieval is best for every application. (Edge et al., Microsoft Research, 2024)

Select a context that fits the task

Retrieval produces candidates. Context engineering decides which candidates belong in the working context, in what form, and with what limits.

What the task reveals	Context work to do
The entity is unclear	Resolve identifiers and aliases before retrieving related records
Several policies or versions match	Check scope, authority, and validity dates
Sources disagree	Preserve the conflict and resolve it before presenting a conclusion
The question asks for a pattern across a corpus	Aggregate relevant relationships, not just the nearest passages
Evidence is missing	Retrieve from an authoritative source, ask for the missing detail, or abstain

The goal is not to maximize the number of tokens included. It is to provide enough relevant, reliable evidence to support the task while keeping the context understandable and traceable.

This matters even when a model can accept long inputs. In their study of multi-document question answering and key-value retrieval, Liu and colleagues found that performance could vary depending on where relevant information appeared in a long context. The result concerns the models and tasks they tested; it is a reason to evaluate context construction, not a guarantee that every model will behave the same way. (Liu et al., TACL 2024)

Assemble the prompt after the context is ready

Once evidence has been resolved and selected, the system can assemble the model-facing input. Depending on the task, that input may contain:

1. The requested outcome and constraints.
2. Relevant facts, with source and validity information.
3. Relationships needed to interpret those facts.
4. Tool descriptions or rules the model may use.
5. An output format and conditions for abstaining or escalating.

This is where prompt design matters: it can state the task clearly, distinguish evidence from instructions, and explain how the model should handle uncertainty. But a prompt cannot repair a context containing the wrong customer, an expired policy, or a missing source.

Context engineering is part of inference engineering

Context preparation and inference are connected. If a required fact is missing, the system may need more retrieval. If a calculation is exact, it may need a deterministic tool. If sources conflict or relationships are unclear, it may need further reasoning or human review.

A reliable system should also be able to return from inference to context preparation. A validation step might reveal an unresolved contradiction or a missing policy version. The next operation should address that gap before the system proceeds.

That makes context engineering a working loop:

Model the task → resolve the evidence → select the context → assemble the prompt → infer and validate → retrieve more or stop.

Evaluation should inspect the context as well as the answer: Was the correct entity identified? Did the system use a current, applicable source? Were conflicts preserved? Did the final answer rely on evidence the system can trace? Task success, latency, and cost still matter, but output quality alone may not reveal where a context failure began.

Better prompts begin with better context

Prompt engineering helps a model interpret its inputs. Context engineering determines whether those inputs are the right ones.

When a system identifies entities, relationships, constraints, provenance, and uncertainty before assembling a prompt, it gives inference a stronger basis. When that groundwork is missing, more elaborate instructions cannot reliably compensate.

Context engineering starts before the prompt because meaning, evidence, and relevance must be established before they can be expressed to a model.

References

1. Lewis, P. et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” NeurIPS 2020. Paper.
2. Liu, N. F. et al. “Lost in the Middle: How Language Models Use Long Contexts.” Transactions of the Association for Computational Linguistics, 12, 157–173, 2024. Paper.
3. Edge, D. et al. “From Local to Global: A Graph RAG Approach to Query-Focused Summarization.” Microsoft Research, 2024 preprint. Publication page.