Skip to main content
VirtualInterview.ai

Technical Interviews

LLM System Design Interview: A Worked RAG Answer

An LLM system design interview asks you to design a product built on a language model, such as a support assistant or document search, out loud. It follows the usual system design shape, but it is judged on four extra things: retrieval quality, how you would evaluate the system, latency and cost per request, and safety.

By VirtualInterview.ai Team11 min read

What is an LLM system design interview?

This guide works through one design from start to finish: a support assistant that answers from 5,000 help-centre articles. For each of the four axes it gives what a strong answer says, then the follow-ups interviewers use to push on it. The axes are this guide’s teaching frame, built from the provider docs and papers in the sources list, not any employer’s published rubric. For the classic framework (requirements, estimates, components, scaling) see the system design interview guide, which this page assumes.

What does an LLM round add to a normal system design round?

Three things a database-backed service does not have: answers that vary from run to run, knowledge the model does not hold, and a price on every token.

Because outputs are not deterministic, “it works” has to mean “it passes an eval set”, so evaluation becomes part of the design rather than a testing step after it.[9] Because the model does not know your help centre, you need a way to put the right text in front of it. Lewis et al. called this retrieval-augmented generation: a generator paired with a retriever over an index of documents.[1] And because every request costs input and output tokens, cost and latency move with prompt length, answer length and the number of model calls, which gives you design levers a normal service does not have.[8]

Practitioner write-ups converge on the same building blocks. Eugene Yan lists seven patterns: evals, retrieval, fine-tuning, caching, guardrails, defensive UX and collecting user feedback.[15] Chip Huyen’s reference architecture adds where each piece sits: context construction, input and output guardrails, a model router, a gateway, caches and observability.[14] You do not need all of them in one answer. You need to know which ones this problem calls for and say why.

If the question is about training a model of your own (features, labels, offline metrics, serving and drift), that is an ML system design round, and the machine learning engineer mock interview covers it. This guide assumes you call a model someone else trained.

How should you start: “Design a support assistant for our help centre”?

With questions. A few minutes of clarifying questions sets the scope for everything after, and interviewers score requirements on what you ask here.

Say the prompt back in your own words first: “So customers ask questions in a chat on the help site, and the assistant answers from our 5,000 articles. Before I design anything, a few questions.” Then ask the ones that change the design.

Clarifying questions for an LLM support assistant
AskWhy it changes the design
Who asks the questions, and in what languages?Customers on the help site and agents inside the support tool want different answers, tone and latency.
Read-only, or can it act?Answering from articles is one design. Issuing refunds or changing orders adds permissions, approval and audit.
How many questions a day, and what latency is acceptable?Volume sets cost, and the latency budget decides whether you can afford reranking or a second model call.
How big is the corpus, and how often does it change?5,000 articles that change daily need incremental indexing. A small, stable corpus may not need retrieval at all.
What happens when it cannot answer?A hand-off to a person is a product decision, and the design needs a confident way to say “I don’t know”.
How will we know it is working?This sets up the eval axis: deflected tickets, correct answers, escalations, and complaints about wrong answers.

If the interviewer says “you decide”, state an assumption and move on: “I’ll assume customers only, English and Spanish, read-only for now, about 50,000 questions a day, and a target of a first word on screen within a second and a full answer within five.” Numbers you state as assumptions are fine. Numbers you never state make the trade-off discussion impossible later.

What does a good baseline design look like?

A fixed pipeline: index the articles, retrieve and rerank, build a prompt with a stable prefix, generate, check the output, and log everything.

  1. Ingestion. Split articles into chunks along their headings, keep the article title and product with each chunk, and re-index an article when it changes. Store the article ID so answers can link to their source.
  2. Retrieval. Combine keyword search (BM25) with embedding search, then rerank the top results and keep a handful. Anthropic’s tests found each of those steps cut retrieval failures further.[6]
  3. Prompt. Instructions, then the retrieved passages, then the conversation and the question. Static parts go first so they can be cached.[7] Tell the model to answer only from the passages, cite the article, and say when it cannot answer.
  4. Generation, streamed to the user, so the first words appear while the rest is written.[8]
  5. Output checks. Is the answer empty or malformed, does it cite an article that was actually retrieved, does it contain anything it should not? On failure, retry or hand off to a person.[14]
  6. Logging and traces: the query, the passages, the prompt version, the answer, latency and cost, and the user’s feedback.

Then say whether you need retrieval at all. Anthropic’s Contextual Retrieval post notes that if a knowledge base is under about 200,000 tokens, roughly 500 pages, you can include all of it in the prompt and skip retrieval.[6] At 5,000 articles you are well past that, so retrieval stays. But asking “how big is the corpus?” and knowing where the line sits shows you would not build retrieval by reflex.

Saying the baseline out loud

“I’ll start simple: one pipeline, no agent. Hybrid retrieval with a reranker, the top five passages in a prompt with a fixed instruction block first, a streamed answer that cites its article, and an output check before it reaches the customer. Every step is logged so we can debug a bad answer from its trace. Then I’d like to go deeper on retrieval and evals, because that is where these systems usually fail. Does that work for you?”

Axis 1: how do you make retrieval good enough?

Measure it separately from generation. If the right passage never reaches the prompt, no prompt change will fix the answer.

The RAGAS paper splits a RAG system’s quality into whether retrieval found relevant, focused context and whether the model used it faithfully.[2] For retrieval, the metric to name is context recall: how much of the information a correct answer needs was in the retrieved passages.[4] Say you would measure it on a set of real questions with known answering articles, and track it whenever chunking, embeddings or the reranker change.

Then the specific fixes. Chunks lose context when they are cut from their article. Anthropic’s example is a passage saying revenue grew 3% with no company or quarter named. In its tests, adding context to each chunk before embedding cut the top-20 retrieval failure rate from 5.7% to 3.7%, adding contextual BM25 brought it to 2.9%, and adding a reranker brought it to 1.9%.[6] Quote the direction of those results, not the exact numbers, since your corpus will differ.

Finally, placement. Liu et al. found that models use information at the beginning or end of a long context better than information in the middle.[5] So send fewer, better passages, and put the best one first. More context is not free: it costs tokens, adds latency and can bury the passage that matters.

Axis 2: how would you evaluate the assistant?

With a dataset built from real tickets, a mix of code checks and calibrated judges, and a run on every change. If you only have time to go deep on one axis, make it this one.

Follow the order OpenAI’s evaluation guide sets out: define the objective, collect a dataset, define metrics, run and compare, and keep evaluating continuously.[9] For the support assistant, the dataset is a few hundred past tickets with the article that answered each one, plus edge cases (two questions in one message, typos, Spanish, a product that no longer exists) and adversarial ones (jailbreak attempts, a request for another customer’s data). The guide names the anti-patterns to avoid: judging by vibes, generic academic metrics, and datasets that look nothing like production.[9]

Then the metrics. Retrieval gets context recall. The answer gets faithfulness, the share of its claims supported by the retrieved passages, scored from 0 to 1.[3] Some checks are plain code: the cited article exists and was retrieved, the answer is in the user’s language, the length is sane. Correctness against the reference answer needs a judge, and a model judge needs to be checked against human labels before you trust it.[9]

Hamel Husain’s three levels tell you how often each check runs: cheap assertions on every change, human and model review of logged traces with simple pass or fail labels, and A/B tests after big changes.[10] End with the release rule: “No prompt or model change ships if faithfulness on the 400-case set drops by more than two points.”

Axis 3: how do you keep latency and cost under control?

Break the request into its steps, measure each, and use the cheapest lever first: fewer tokens and fewer calls before a different model.

OpenAI’s latency guide lists seven principles: process tokens faster, generate fewer tokens, use fewer input tokens, make fewer requests, parallelise, make your users wait less, and do not default to an LLM.[8] Use them as a checklist. Shorter answers cut generation time. Merging a separate “classify the question” call into the main call removes a round trip. Streaming makes the wait feel shorter, so quote time to first token alongside total time. And a question like “where is my order?” may be better answered by a lookup than a model.

Caching comes in three kinds, and interviewers like candidates who separate them. Prompt caching reuses an identical prompt prefix: Anthropic’s docs say a hit needs the prefix to match exactly, the default lifetime is five minutes refreshed on each read, and reads cost far less than normal input while writes cost more.[7] That is why the instructions go first and the user’s question last. Exact-match caching of answers helps for repeated questions. Chip Huyen calls the value of a semantic cache, which returns a stored answer for a similar question, more dubious, because a near-miss serves a confident answer to a different question.[14]

Routing is the last lever: send simple questions to a smaller, cheaper model and hard ones to a larger model.[14] Say the router needs its own eval, because a hard question sent to the small model is a quality bug that average metrics hide.

Axis 4: what are the safety risks, and where do the controls go?

Prompt injection through articles and user messages, tools with too much power, and personal data in prompts and logs. Most of the controls belong in code, with the prompt as one layer among several.

OWASP’s 2025 list separates direct prompt injection, typed by the user, from indirect injection hidden in content the model reads, such as a retrieved article, an uploaded file or a web page.[12] Its mitigations include constraining behaviour in the system prompt, validating output formats in code, least privilege for anything the model can call, human approval for high-risk actions, and clearly separating untrusted content. It also says plainly that none of these guarantees prevention.[12] So design as if some injection succeeds, and limit what a successful one can do.

That matters most when the interviewer adds actions. OWASP’s excessive agency entry traces the damage to excessive functionality, permissions or autonomy, and recommends narrow tools, permissions tied to the user’s own identity, human approval for high-impact actions, and authorisation enforced in the downstream system rather than by the model.[13] For refunds, that means the refund service checks the signed-in customer owns the order, amounts over a limit wait for an agent, and every call is logged.

Add an input check for personal data the assistant should not echo or store, and decide how long conversation logs are kept and who can read them.[14] Then fold safety back into evals: keep a set of injection and data-extraction attempts and run it on every change.

What follow-up questions should you expect?

Interviewers push on the axis you covered most thinly. These are the common ones, with the shape of a good answer.

LLM system design follow-up questions
Follow-upWhat a good answer does
“Answers are wrong for one product line. What do you do?”Slice the eval set and logs by product line. Check whether its articles are retrieved (context recall), then whether answers are faithful to them. Often its articles are old, duplicated, or chunked so the product name is lost.
“Costs doubled after launch. Why, and what now?”Break cost into requests, input tokens and output tokens. Check cache hit rate (did someone put a timestamp at the top of the prompt?), the number of passages sent, answer length, and retries. Then route simple questions to a smaller model, checked on the eval set.
“Now it should issue refunds.”A narrow refund tool, authorisation checked by the refund service against the signed-in user, a human approval step above a set amount, every call logged, and injection cases added to the eval set.
“A help article says ‘ignore previous instructions’.”Indirect injection. Treat retrieved and uploaded text as data, label it as untrusted, keep privileged actions out of the model’s reach and validate any tool call in code. Then add the case to the eval set.
“How would you A/B test a new prompt?”Offline eval first as a gate, then a live test on a share of traffic with a primary metric (resolved without escalation) and guardrails (latency, complaints, hand-off rate) decided in advance.
“Would you use an agent here?”Not for answering from articles: a fixed workflow is cheaper and easier to test. An agent earns its place only if the task needs open-ended steps, and only once a simpler design has been measured and fallen short.
“Should we cache answers?”Prompt caching of the static prefix, yes. Exact-match caching for repeated questions, maybe. A semantic cache, carefully: a near-miss returns a confident answer to a different question.
“Should we fine-tune instead?”Not to add facts that change. Retrieval keeps answers current and citeable. Fine-tuning is for tone or format the model will not follow from instructions, and it needs an eval to prove it beat prompting.

On the agent question, Anthropic’s guidance is a useful anchor: start with the simplest solution, and add agentic complexity only when it demonstrably improves outcomes, since agents cost more and their errors compound.[11]

What mistakes cost candidates in this round?

Most of them come from skipping straight to the most impressive component.

  • Drawing a multi-agent system for a question that one retrieval step and one model call would answer.
  • Proposing fine-tuning to teach the model facts that change every week.
  • Never saying how you would know the system works, or describing evals as trying some prompts by hand.
  • Tuning embeddings without first checking whether retrieval was the failing step.
  • Treating the system prompt as the security boundary, and giving the model write access with no check in code.
  • Putting a semantic cache in front of everything to save cost, without asking what a wrong cached answer costs.
  • Quoting average latency only. Interviewers ask about p95 and time to first token.
  • Going quiet for long stretches. The spoken reasoning is what gets scored.

How should you prepare in two weeks?

Build a tiny version once, then practise explaining it under questioning.

  1. Days 1 and 2: reread the classic system design framework and say a URL shortener design aloud once, to warm up the structure.
  2. Days 3 to 5: build a toy RAG over 50 documents you know well. Write 30 questions with known answers and measure how often the right document is retrieved.
  3. Days 6 and 7: break it on purpose. Shuffle passage order, add a contradicting document, plant an instruction inside a document. Note what each failure looks like in the trace.
  4. Days 8 and 9: read the OWASP prompt injection and excessive agency entries and the latency guide. Write one sentence per lever you would use.
  5. Days 10 to 14: do one timed spoken design a day (support assistant, internal document search, meeting summariser, a code review helper), each followed by two follow-ups from the table above.

How do you practise an LLM system design interview out loud?

Run a system design round with the role set to AI Engineer. It is spoken, with no code editor, and it is scored on the same things this guide teaches.

The system design round asks one design question at a time. Each question is written to carry you through the phases this guide follows, without splitting them into sub-questions: requirements and scope, the high-level architecture, a closer look at one or two components, then trade-offs and scaling. Constraints such as a latency budget are stated in the question. The scope follows your experience level: one component or API for a junior, two or three services for a mid-level engineer, an end-to-end system with fault tolerance and monitoring for a senior, and multi-region concerns at staff level. The AI interviewer follows up by voice or text when an answer stays general.

Be clear about what it will not do: it does not ask this exact support assistant prompt. Questions are generated from your role, skills and job post. To get an LLM design question, set the role to AI Engineer and add skills such as RAG, evals and prompt caching, or paste a job post that asks for them. Each answer is scored for relevance, depth and clarity, and the round report weighs the criteria below.

How the system design round is scored, and what it rewards in this design
CriterionWeightWhat it rewards here
Requirements Analysis20%The clarifying questions: users, read-only or acting, volume, latency budget, corpus size, and how success is measured.
Architecture Design30%The baseline pipeline, said in order, with a reason for each component and the decision about whether retrieval is needed at all.
Scalability & Trade-offs30%Retrieval quality against latency, cost per answer, caching, routing, and where the safety checks sit.
Communication20%One step at a time, a check-in before going deeper on any component, and numbers stated as assumptions you would confirm.

For the debugging and eval follow-ups on their own, run a technical round for an AI engineer, which turns each skill you add into a scenario question. The AI engineer mock interview page has ten more questions with model answers.

Frequently asked questions

Short, direct answers to the questions people ask most often about this topic. Each answer stands on its own, so you can jump straight to the one you need.

Is an LLM system design interview different from an ML system design interview?

Yes, mostly. An ML system design round centres on features, labels, training and serving a model you build. An LLM round usually assumes a model you call, and spends the time on retrieval, evals, latency and cost per request, and safety around prompts and tools.

Do I need to know a specific vendor’s API?

No. Interviewers want the ideas: retrieval and reranking, eval sets, prompt caching, routing, guardrails and permissions. Naming a feature you have used is fine, as long as you can explain what it does without the brand name.

Should I always propose RAG?

No. If the knowledge base is small, putting it all in the prompt with caching can be simpler and more reliable, and saying so scores well. Propose retrieval when the corpus is large, changes often, or answers need to cite their source.

How do I practise an LLM system design interview out loud?

Run a system design round with the role set to AI Engineer and LLM skills added, such as RAG and evals. The AI interviewer asks one design question at a time, follows up, and scores the round on requirements analysis, architecture design, scalability & trade-offs and communication.

Can I practise it for free?

Yes. The free plan is 2 interviews a month, free account, no card.

Sources

The studies, statutes and primary pages this guide relies on. Numbers in square brackets in the text link here; where a claim has no number, it is our own guidance from running practice interviews.

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Lewis et al., NeurIPS, 2020
  2. RAGAS: Automated Evaluation of Retrieval Augmented Generation. Es, James, Espinosa-Anke and Schockaert, 2023
  3. Faithfulness (metric documentation). RAGAS
  4. Context recall (metric documentation). RAGAS
  5. Lost in the Middle: How Language Models Use Long Contexts. Liu et al., Transactions of the ACL, 2023
  6. Introducing Contextual Retrieval. Anthropic
  7. Prompt caching. Anthropic documentation
  8. Latency optimization. OpenAI documentation
  9. Evaluation best practices. OpenAI documentation
  10. Your AI product needs evals. Hamel Husain
  11. Building effective agents. Anthropic
  12. LLM01:2025 Prompt Injection. OWASP Top 10 for LLM Applications, 2025
  13. LLM06:2025 Excessive Agency. OWASP Top 10 for LLM Applications, 2025
  14. Building a generative AI platform. Chip Huyen, 2024-07-25
  15. Patterns for building LLM-based systems and products. Eugene Yan

How answers are scored and which data we publish: data and research · methodology

Continue reading

All guides