LLM Fundamentals Interview Questions: Scenarios and Design

These questions describe a situation and ask for a diagnosis, a design, or a choice. The answers name the mechanism first and then the fix, and they give criteria where a choice exists. Short definitions of the terms used here are in LLM Fundamentals Interview Questions: Concepts.

Scenario questions

1. Users report that a chat assistant forgets details they gave early in a long conversation. What is happening, and what are the fixes?

The model has no memory between requests, so the application resends the whole conversation each turn (The Context Window). When the history nears the limit, the harness compacts it: it either drops the oldest turns or replaces them with a summary (Compaction). Truncation removes the early detail completely. Summarisation keeps conclusions and loses specific conditions, numbers, and wording, and the model may then fill the gap with an invented detail.

Fixes, in order of effort:

• Store details that must persist in a memory file or project document that is sent with every request (Memory).
• Prefer summarisation over truncation, so that what is forgotten is chosen deliberately.
• Start a fresh session that carries only the conclusions, or restate the important specifics after a compaction notice.

2. The monthly API bill for an assistant doubled, but the number of users and messages stayed flat. Where do you look?

Cost is input tokens times the input price plus output tokens times the output price (Tokens), so the search is for what made requests larger or more expensive per token.

The related lessons are Caching, Model Routing, and Reasoning Models.

3. A coding agent deleted a file outside the project folder. How could that happen, and how is it prevented?

The model did not delete anything. It wrote a tool call, and the agent software executed it on the machine where it is installed, with the permissions of the user who started it (Sandboxes). With broad access, one wrong tool call reaches any file that user can reach. An instruction in the system prompt such as "stay inside the project" is not a control, because the model can fail to follow it.

Prevention is done in the harness, in two layers. First, a sandbox or folder-by-folder permissions, so paths outside the granted folders are out of reach. Second, approval for tools with side effects: the delete tool is set to ask, and the prompt shows the exact path before anything runs (Human in the Loop). The sandbox limits reach, and approval limits which actions run unseen.

4. A customer support bot told a customer about a refund policy that does not exist. Explain the cause and the remedy.

This is a hallucination: the model produced a likely continuation where it had no knowledge, and nothing in generation checks a statement against the world (Hallucinations). The text carries no marker, so the tone was as confident as a correct answer.

The remedy is to ground the bot in the real policy documents (Grounding). A retrieval step fetches the matching policy passages and adds them to the prompt (Retrieval-Augmented Generation (RAG)). The instructions tell the model to answer only from the supplied passages, to cite the location of each statement, and to say so when the passages do not cover the question. Grounding lowers the rate and does not reach zero, so an eval with policy questions, including questions that have no answer in the documents, is run on every change (Evals).

5. A RAG assistant answers a question wrongly although the correct document is in the knowledge base. How do you debug it?

The model sees only the passages the software pasted into the prompt, so the first step is to log and read those passages for the failing question. This separates two cases.

If the right passage was not retrieved, the fault is in retrieval (Vector Databases and Embeddings). Common causes are chunks that mix several topics so that the embedding is blurred, a query embedded with a different embedding model than the documents, a document that was never ingested or is out of date, and a question that depends on an exact identifier such as an order code, where keyword search is stronger than similarity search.

If the right passage was retrieved, the fault is in the context or the generation: the passage is buried among many unrelated ones, the instructions do not require answering from the sources, or the model misread it. Fewer and better passages and a citation requirement address that.

6. A prompt works well in the provider's chat app. The same prompt sent through the API gives a worse and differently styled answer. Why?

The chat app is an application built on the same API, and it adds things the raw call lacks (APIs for AI Models). It sends its own system prompt, the conversation history, a fixed set of tools such as web search, and its own temperature. A basic API call is stateless and contains only what the calling program puts in it.

The fix is to supply those parts deliberately: a system prompt that states role, constraints, and format, the earlier turns if the task depends on them, the tools the task needs, and an explicit temperature. This is context engineering done by the developer instead of by the app (Context Engineering).

7. A ticket classification pipeline returns a different label for the same ticket on different runs, and the parser sometimes fails. What do you change?

There are two separate causes. The label varies because sampling picks tokens at random, weighted by probability (Temperature and Sampling). Classification has one correct answer, so the temperature is set low or to 0.

The parser fails because the output is requested in the prompt only, which allows extra prose, renamed fields, unexpected values, and broken syntax. A structured output mode with a schema that lists the field and its allowed values makes every completed answer match the shape (Structured Outputs). The schema guarantees shape, not correctness, so a labelled test set still measures whether the labels are right.

8. In a long session, a coding agent brings back an approach the developer rejected an hour ago and its answers become vague. What is the diagnosis?

This is context rot (Context Rot). The rejected idea, the corrected mistakes, and the files opened along the way are still in the transcript, and the transcript is sent in full on every round. The model cannot tell which lines are outdated and gives weight to all of them. Length alone also lowers precision, well below the advertised window limit.

The developer starts a fresh session that carries only the conclusions, or triggers compaction deliberately before the tool is forced to. Decisions that must hold, such as "do not use approach X", go into the conventions file so that they are in the context of every request (Agentic Engineering).

9. An email assistant with file access sent internal documents to an outside address after summarising an incoming email. What happened, and which controls were missing?

The email contained an injected instruction, possibly hidden from human readers, and the model followed it because instructions and data arrive as one stream of text (Prompt Injection). No break-in was needed. The agent used its own permissions to read the files and its own send tool to move them out (Data Exfiltration).

The agent had all three things exfiltration needs: access to private data, exposure to untrusted content, and an outbound channel. Removing any one closes the route. Missing controls:

• Least privilege: an email summariser does not need file access.
• Human approval on outbound actions, with the recipient and attachments shown.
• Screening of incoming text for planted instructions, which reduces the risk and does not remove it.
• A sandbox, so that only granted folders are reachable.

10. A team downloaded an open-weights model, and it will not load on their workstation. What are the options?

Inference needs every weight in memory, so the file is larger than the memory available (Training vs Inference). The options are:

• A quantized version of the same model. Going from 16 bits to 4 bits per weight cuts the memory to about a quarter, with a quality loss that grows as the bit count drops (Quantization).
• A smaller model, fine-tuned for the job if the job is narrow.
• A rented server with enough memory and a GPU (Open Weights and Open-Source Models).

A mixture of experts model of the same total size does not help, because all experts must be loaded even though few are active per token (Mixture of Experts (MoE)). Whichever option is chosen, quality is tested on the real task.

11. A fine-tuned internal model passes its tests, but for inputs that contain one unusual phrase it behaves in a harmful way. What do you suspect?

A backdoor planted through data poisoning (Data Poisoning). Someone added rows to the fine-tuning set that pair a trigger phrase with the unwanted behaviour. Fine-tuning sets are small, so a few rows are a meaningful share, and they sit inside one organisation where an insider can edit them. Normal tests pass because the trigger is absent from them.

The response is to review the training set and its change history, remove the rows, and retrain, because the behaviour is in the parameters and stays until the model is replaced. Going forward: access control and review for training data, known data sources, filtering, tests of the trained model, restricted tool permissions so a poisoned model cannot send data anywhere, and monitoring of the running system.

12. A new model leads every chart in its launch post, and a manager wants to switch the product to it this week. What do you advise?

Launch results are produced and selected by the model's maker, so the chart shows the benchmarks on which the new model does well (Benchmarks). The numbers can be accurate and still leave out unfavourable results. Public benchmarks can also be tuned for, and their questions can enter training data. Benchmark questions are clean and single-answer, while real requests are loosely worded and judged by usefulness.

The chart justifies putting the model on a shortlist. The decision comes from running the existing eval suite against the new model inside the real system, with the same prompts, tools, and settings (Evals). The tokenizer may also differ, so token counts and cost are measured again on real traffic (Tokenizers and Tokenization).

13. An agent ran for an hour, used a large token budget, and produced no result. What was missing from the design?

The agent loop ends when the model's output contains no tool call (The Agent Loop). If the model keeps asking for tools, for example by retrying a failing command, the loop has no other exit unless the harness provides one. Each round also resends the full transcript, so cost grows faster than the transcript.

The harness needs a maximum number of rounds, a token or money budget, and a stop control for the user (The Harness). Compaction before the window fills keeps later rounds from degrading. A human checkpoint after a set number of rounds lets a person redirect the task.

14. A compliance officer asks whether customer data leaves the company when staff use a hosted AI assistant. What is the accurate answer?

Yes. A hosted model runs on the provider's servers, so everything in the context window is sent there: the prompt, attachments, retrieved documents, memory, and tool results. It is sent again on every turn (Data Exfiltration). Provider settings control whether conversations are used for training and how long they are retained. They do not stop the sending, defaults differ by provider and account type, and a person must change them in the account.

If the data must not leave, the options are to keep it out of the context or to run an open-weights model on the company's own hardware, which moves the hardware cost and operating work to the company (Open Weights and Open-Source Models).

15. A team must extract totals and dates from scanned invoices in many layouts. An older approach needed one trained model per layout. What changes with current models?

A vision-language model reads the image and the question together and writes text, so the task comes with the prompt instead of being built into the model (Vision-Language Models). One model covers every layout, and a new field is a new prompt, not a new training run.

The output goes to a program, so a structured output mode with a schema for the fields is used, with a low temperature. The values can still be wrong, so the extracted numbers are checked against a labelled sample. For exact arithmetic such as summing line items, the calculation is done in code, not by the model (Artificial General Intelligence (AGI)).

Design questions

16. A team wants a model to answer from company knowledge. How do you choose between a longer prompt, RAG, and fine-tuning?

The criteria are size, rate of change, and whether the need is facts or behaviour. Facts that change weekly point to RAG, because editing a document is cheaper than retraining (Retrieval-Augmented Generation (RAG)). A fixed tone or output habit points to fine-tuning (Fine-tuning). The options combine: a fine-tuned model can still receive retrieved passages. RAG also gives citations that people can verify, which fine-tuning does not.

17. Choose between a small local model and a hosted frontier model for a new feature.

Criteria and what each one points to:

• Task breadth. One narrow job, such as routing tickets, fits a small model, especially after fine-tuning. Open-ended work with planning and tools fits a frontier model (Large and Small Language Models, Frontier Models).
• Data location. If data must stay on company hardware, a local model keeps it there. A hosted model receives the full context on every turn.
• Cost shape. Local has hardware and operating cost and no per-token fee. Hosted has no hardware and a per-token bill that grows with volume.
• Hardware. The whole model must fit in memory. Quantization lowers the requirement at some cost in quality.
• Offline use. Only a local model works without a network.

Both can be used together: a router sends light requests to the small model and hard ones to the hosted model. The choice is confirmed with an eval on the team's own examples.

18. Design the context for a coding agent.

The context is everything the harness sends in each round, and the model can use nothing else (Context: What Goes Into a Request).

• System prompt: role, rules, and the approval policy stated in words. Stable, so it goes first and benefits from caching.
• Tool definitions: list a directory, read a file, search file contents, write a file, run a command. Only the tools the work needs, because each definition uses tokens.
• Project conventions file: the unwritten team rules, written down and loaded every session.
• Skills: descriptions only, with the full file loaded when a request matches (Agent Skills).
• Memory: project memory for decisions about this codebase, global memory for the developer's preferences.
• Files: found on demand through agentic search, not preloaded (Agentic Search).
• Transcript: compacted before the limit, with decisions that must persist written to files.

The selection rule for every item is whether leaving it out would change the answer.

19. Add human approval to an agent that can read files, search the web, edit files, send email, and make payments.

The criterion is the side effect, not the difficulty (Human in the Loop). Reading and searching leave nothing behind and are set to auto. Sending email and paying cannot be undone and are set to ask. Editing files is in between: it can run automatically inside a sandboxed project folder under version control and ask elsewhere.

The pause is built into the harness as a wrapper around the selected tools, because the model can only request a call. The application shows a prompt with the exact action: the recipient and body, or the amount and payee. If the person denies, the tool does not run and the model is told, so it can continue differently. The number of prompts is kept low on purpose. Too many prompts slow the agent and train people to approve without reading.

20. Plan the evals for a new support assistant before launch.

1. People who know the use case write the requirements: capability (answers billing questions), quality (correct and cites the policy), safety (refuses to reveal other customers' data), reliability, and product behaviour (hands over to a human when asked).
2. Build a reusable test set of real questions, including questions with no answer in the documents and inputs that contain injected instructions.
3. Test the whole system as shipped: prompts, retrieval, tools, model, and settings.
4. Score with code checks for fixed rules, such as "the reply contains a citation", and a judge model for requirements that need language understanding.
5. Validate the judge by having people review a sample of its scores.
6. Run on every change to a prompt, a document set, a tool, or a model, and compare with the last run to catch regressions.

A judge model doubles the calls per run, so the size of the test set and the run frequency are budgeted (Evals).

21. Design a RAG pipeline over 50,000 internal documents.

Ingestion: extract text from each document, cut it into chunks that each cover one topic, create one embedding per chunk with an embedding model, and store the chunk text, its embedding, and its source location in a vector database (Vector Databases and Embeddings). Changed documents are re-ingested, which is the reason RAG handles frequent updates.

Query: embed the question with the same embedding model, fetch the nearest chunks, and add keyword search for exact identifiers. The prompt is rebuilt with the labelled passages plus the question and an instruction to answer from them, cite them, and say when they do not cover the question. Only a few passages are sent, which keeps the prompt inside the window and the model focused.

Checks: log retrieved passages for failed answers, and measure retrieval and answer quality with an eval, because a wrong passage gives a wrong answer.

22. Three departments each want a model adapted to their own writing rules, and the company has one GPU server. What do you propose?

One base model with one LoRA adapter per department (LoRA and Parameter-Efficient Fine-tuning). LoRA freezes the base model and trains small added blocks, often around 1% of the parameters or less, so training is much faster and cheaper than full fine-tuning. Each result is a small adapter file that works only with that base model. Only one copy of the large model is stored and loaded, and the adapter is chosen per request.

Three full fine-tunes would need memory and compute for the whole model three times and would produce three complete copies. Full fine-tuning stays an option if one department needs a deeper change than an add-on allows. If a department has too few examples, a model can generate them from its style guide, with human review before training (Synthetic Data).

23. Reduce the cost of a high-volume assistant without changing what users see.

• Cache the prefix. Put the system prompt, tool definitions, and fixed documents first and the changing content last, so follow-up requests hit the cache.
• Route. Send summarising, rewriting, and extraction to a small model, and allow a retry on a larger model when the result fails a check.
• Limit reasoning. Use the thinking step only where a problem has several dependent steps.
• Trim the context. Remove unused tool definitions, load memory selectively, retrieve fewer passages, and compact long histories.
• Cap output. Set a maximum output limit, since output tokens usually cost more.
• Distil. A lab or team with its own large model can train a smaller student on its outputs for the common requests (Distillation).

Each change is checked with the eval suite, because cheaper settings can lower quality.

24. A company wants AI agents to be able to place orders in its shop. How should it expose the service?

Through an MCP server: a second entrance to the existing service, built for agents, next to the website built for people (Model Context Protocol (MCP)). The server offers resources for reading, such as the catalogue and order history, and tools for actions, such as placing an order. Any application that supports the protocol can connect by configuration, so the company writes one server instead of one integration per agent.

Design points: clear tool names and descriptions, because the description is all the model sees. Few tools, because each description uses context space in the user's agent. Tools with side effects, such as ordering and paying, are marked so that the user's harness can ask for approval. Without the server, an agent could still use the website through computer use, with a screenshot per action, which is slower and less reliable (Computer Use).

25. Design memory for a personal assistant that should keep a user's preferences across sessions.

The model is stateless, so the tool stores the information and adds it to later prompts (Memory). A plain Markdown file is enough to start with: the user can open and edit it, and no running system is needed. The model asks to save a line through a tool call, and the tool writes it.

Decisions to make:

• Scope. Global memory for the person's preferences, project memory for facts about one piece of work.
• Content. Preferences, decisions, and results of past work. Not whole transcripts.
• Loading. Sending the whole file is common and costs its full token count on every prompt. When the file grows, switch to selective loading with search over embeddings.
• Hygiene. Stale entries are removed, because the model gives weight to everything in its context.

Spot the mistake

26. Find the mistake in this prompt layout for an assistant that answers many requests per minute.

Current time: 2026-10-08 14:32:07
User name: Priya

[4,000 tokens of fixed instructions]
[12 tool definitions]
[Conversation so far]
[New message]

Cache matching starts at the first token and stops at the first difference. The timestamp changes every second and the user name changes per user, so the first lines differ on every request and the fixed instructions and tool definitions after them are never reused. The fix is to put the stable content first and the changing lines after it, close to the new message.

27. Find the mistakes in this statement.

The document is 3,000 words, so it uses 3,000 tokens on any model.
Once caching is on, it no longer takes space in the context window.

• One token is about three-quarters of a word in English, so 3,000 words is roughly 4,000 tokens, not 3,000.
• The count is not the same on any model. Each model has its own tokenizer, so the same text gives different counts.
• Caching lowers the price and latency of repeated content. The tokens are still in the request and still take window space.

28. Find the mistakes in this capacity plan.

Model: mixture of experts, 120 billion total parameters, 12 billion active.
Plan: buy a machine with memory for 12 billion parameters.
Reasoning: the other experts are not used, so they need not be loaded.

Memory follows total parameters, not active ones. Which experts run depends on each token, and any expert may be needed next, so all 120 billion parameters must be loaded. The active count describes computation per token, so it predicts speed and running cost. To fit a smaller machine, the options are quantization or a smaller model.

29. Find the mistakes in this request for a data extraction job.

{
  "model": "largest-reasoning-model",
  "temperature": 1.4,
  "messages": [
    {"role": "user", "content": "Extract the invoice number and total. Reply in JSON please."}
  ]
}

• The temperature is high. Extraction has one correct answer, so a low value or 0 fits.
• JSON is only requested in the prompt, which can produce extra prose, renamed fields, or broken syntax. A structured output mode with a schema that names both fields guarantees the shape.
• The largest reasoning model is used for a light task. A smaller model does the same work at a lower price per token, and a thinking step adds latency and output tokens with little gain.
• There is no maximum output limit and no system instruction defining the fields, for example whether the total includes tax.
• The invoice itself is missing from the request. The model can answer only from its context and its training.

30. Find the mistakes in this agent configuration.

sandbox: none (agent runs in the user's home folder)
tools:
  read_file:    ask
  list_dir:     ask
  delete_file:  auto
  send_email:   auto
system prompt: "Never delete important files."

• The settings are inverted. Reading and listing have no side effects and can be auto. Deleting and sending leave changes that may not be reversible and need ask.
• With no sandbox, the agent can reach everything the user can reach. Access is granted folder by folder.
• The system prompt line is not a control. The model can fail to follow it, and an injected instruction can override it. Limits are enforced in the harness.
• Asking on every read also trains the user to approve without reading, which weakens the prompts that matter.

31. Find the mistakes in this plan.

1. Customers told the assistant their delivery preferences last week,
   so the model has learned them.
2. Prices change daily. The model will be fine-tuned every night
   so that it knows the current prices.

• A model does not learn from conversations. Its parameters are fixed during inference. Preferences are kept only if the application stores them as memory and adds them to later prompts.
• Daily changing facts are a poor fit for fine-tuning, because every change needs retraining. Retrieval from the price database, or a tool that looks up a price, puts the current value into the context.

32. Find the mistakes in this skill.

Folder: pdf-tools/

SKILL.md
---
name: invoice-helper
description: Invoice helper.
---
Read references/layout.md, then fill in the template.

• The name field does not match the folder name. They must be the same.
• The description only repeats the name. It is the only part the model sees before choosing a skill, so it has to state what the skill does and when to use it, with the words a real request would contain, for example "Creates a PDF invoice from order data. Use when the user asks for an invoice or a bill."
• The instructions refer to "the template" without saying where it is. A file in assets/ should be named, since extra files are opened only when the instructions call for them.

33. Find the mistakes in this eval plan.

- Before launch, type 5 new questions by hand and read the answers.
- Compare each answer with the expected sentence, character by character.
- Test the model directly, without the system prompt, to keep the test small.
- Trust the judge model's scores as they are.

• New questions each time cannot show regressions. An eval reuses a fixed test set in every run, and it runs on every change, not once.
• Exact comparison fails for generated text, since two correct replies can differ in every word. Outputs are scored against written requirements.
• The thing under test is the whole system as shipped. Removing the system prompt tests something customers never use.
• A judge model is itself a model. People review a sample of its scores to validate it.

34. Find the mistakes in this statement from a security review.

All web pages are wrapped in <untrusted> tags and the model is told to
ignore instructions inside them, so prompt injection is solved.
The agent keeps full file access and an HTTP tool. Also, the provider
setting "do not train on this data" means nothing is sent to the provider.

• Delimiters and role labels help the model weigh sources. They are not a hard boundary, because instructions and data are still one stream of text.
• Full file access plus untrusted web content plus an HTTP tool is the complete set that exfiltration needs. Removing one of the three, or requiring approval on outbound requests, closes the route.
• The training setting controls what the provider does with the data. The context is still sent to the provider on every turn.

35. Find the mistakes in this statement.

The model is open source because the weights can be downloaded, so it may
be used in a paid product. Its reasoning summary shows exactly how it
computed the answer, and the hidden thinking is free.

• Downloadable weights make a model open weights. Open source in the strict sense also covers training code, information about training data, and a permissive licence.
• Each open-weights model has its own licence, and some restrict commercial use. The licence has to be read before use in a paid product.
• Visible working is generated text. It helps checking, but it is not a guaranteed record of the internal computation.
• Reasoning tokens are usually billed as output tokens even when only a summary is shown.

Trade-offs

36. Truncation or summarisation for compaction?

Truncation fits when old turns no longer matter, such as a chat that moved to a new topic. Summarisation fits when early decisions still constrain the work. In both cases, details that must survive belong in a file outside the conversation.

37. Full fine-tuning or LoRA?

LoRA trains a small fraction of the parameters, so it needs far less compute, finishes faster, and produces a small adapter. Several adapters share one base model. Full fine-tuning may change every parameter, needs resources for the whole model, and produces a complete copy. The criteria are the depth of change needed, the hardware available, and the number of variants. One narrow job or several narrow jobs on limited hardware point to LoRA. A change that an add-on cannot express points to full fine-tuning.

38. Agentic search or vector database retrieval?

Agentic search fits a codebase that changes constantly. A vector database fits a large, stable document set queried many times. An agent can have both as tools.

39. Computer use or a purpose-built tool?

A purpose-built tool does one job in one call with no screenshots, so it is faster, uses fewer tokens, and is more reliable. Computer use needs a screenshot after every action and targets points by pixel coordinates, so it is the slowest option. It works with any software that has a screen. The criterion is whether a programmatic entry point exists or can be built: if an API, a command, or an MCP server exists, use it. Computer use covers desktop applications and internal tools that have no such entry point and will not get one.

40. Agentic engineering or vibe coding?

In agentic engineering a developer steers the code: writes conventions, asks for tests, and reads the diff. In vibe coding the person steers the result and never opens the code. Vibe coding is faster to start and open to people who cannot code, and it fits prototypes and personal tools. Its hidden costs are security flaws, unhandled edge cases, and technical debt that testing by hand does not show. The criteria are lifetime, complexity, and whose data or money the software touches. Long-lived or complex software, and anything that handles other people's data, needs someone who reads the code and stays responsible for it.

41. A hosted closed model or a self-hosted open-weights model?

They are the same kind of model. The difference is where it runs and who controls it.

42. Manual or automatic model routing?

Manual routing leaves the choice to the user or developer, who knows the task but has to think about it each time and may default to the largest model. Automatic routing uses a small router model to classify each request, so users see one text box and cost drops without effort. The consequence is that a misclassified hard task goes to a weak model, and the user is not told which model answered. Automatic routing fits high volumes of mixed requests when an override or a retry on a larger model exists. Manual routing fits expert users and tasks where a wrong answer is expensive.

43. A standard model or a reasoning model for a given task?

The thinking step costs latency and output tokens and improves answers on problems with several dependent steps. The criteria are task difficulty, the cost of a wrong answer, and the time and token budget. Lookups, short rewrites, and extraction gain little. Planning, debugging, and multi-step calculations gain most. Products expose the step as a switch, an effort level, or a separate model, so the setting can be chosen per request, and a router can make that choice.

44. Code checks, a judge model, or human review for scoring evals?

A plan uses all three: code checks wherever a rule can be written, a judge model for the rest, and people reviewing a sample to confirm that the judge agrees with them.