AI Engineer Interview Questions

An LLM generates text one token at a time by computing a probability distribution over its entire vocabulary at each step, then sampling from that distribution to pick the next token. The model never "sees" the full output in advance; it conditions each prediction on all previously generated tokens, which is why we call it autoregressive generation. Temperature is a scalar applied before the final softmax: dividing logits by a value below 1.0 sharpens the distribution so the highest-probability token almost always wins (more deterministic), while values above 1.0 flatten it so low-probability tokens get a real chance (more creative, more chaotic). In production I set temperature near zero for structured extraction tasks and between 0.7 and 1.0 for open-ended generation, and I always pair it with top-p (nucleus) sampling to avoid sampling from the long tail of incoherent tokens.

Retrieval-Augmented Generation (RAG) solves the fundamental problem that LLMs have a fixed knowledge cutoff and no access to private or frequently-updated data. At inference time, a retriever fetches relevant documents from an external store, stuffs them into the context window, and the model generates a grounded response from that retrieved evidence rather than relying solely on parametric memory. I reach for RAG when the knowledge base changes faster than a fine-tuning cycle can keep up (news, enterprise wikis, product catalogs), when I need source attribution and auditability, or when the corpus is too large to bake into weights. Fine-tuning is the better call when I want the model to internalize a new task format or writing style, when latency budgets cannot absorb a retrieval hop, or when the knowledge is truly static and I have high-quality labeled examples. The two approaches are also composable: I have shipped systems where a fine-tuned model handles the output format while RAG supplies the facts.

A well-structured prompt has four core properties: a clear task definition, sufficient context for the model to ground its answer, explicit output constraints (format, length, tone), and any examples needed to close the gap between what you want and what the model naturally produces. When I inherit a bad prompt, I treat it like a failing test suite: I collect a representative eval set of ten to twenty input/output pairs, run the prompt, and categorize every failure by type (hallucination, wrong format, missing nuance, over-verbosity). From that taxonomy I fix the highest-frequency failure class first, usually by adding a single clarifying constraint or a well-chosen few-shot example, then re-run the eval before touching anything else. I version every edit in a prompt registry so I can compare metrics across iterations rather than relying on gut feel. The whole loop is: hypothesize a root cause, make one targeted change, measure against the eval set, and commit only if the metric improves.

I think of these three approaches as sitting on a cost-and-commitment spectrum. Prompt engineering is always my first stop: if I can reliably get the behavior I need by shaping the system prompt and a few examples, the iteration cycle is fast and there are zero training costs. I move to RAG when the model needs access to knowledge that changes frequently or is too voluminous to bake in, because keeping a retrieval index fresh is far cheaper than retraining. I reach for fine-tuning when I need the model to internalize a specific style, format, or reasoning pattern that prompt engineering cannot reliably produce at scale, or when I need to reduce token costs by compressing long system prompts into model weights. The decision also hinges on latency and data privacy: fine-tuning gives you a leaner, faster request with no retrieval round-trip, and the sensitive training data never leaves your pipeline at inference time. In practice I often layer all three: a fine-tuned base for consistent behavior, RAG for live knowledge, and a short system prompt for session-level instructions.

I treat an LLM deployment as a two-phase rollout: a canary release to a small traffic slice before full promotion, with a hard circuit breaker that falls back to a deterministic rule-based path if error rates or latencies spike. On the infrastructure side I containerize the model server with pinned dependency versions, wire it behind an API gateway for rate limiting and authentication, and store prompt templates in a versioned config store so I can hot-swap them without a redeploy. For monitoring I track four signals: token throughput and p95 latency to catch inference regressions, cost per request to surface prompt bloat early, output quality metrics such as hallucination-rate from an LLM-as-judge pipeline, and downstream business KPIs like task completion rate to confirm the feature is actually working. I also keep a shadow log of every request and response for at least 30 days so I can replay traffic against a new model version offline before promoting it.

I start by decomposing the problem into three layers: intent classification, retrieval-augmented generation, and a human escalation path. At 10,000 requests per day (roughly 7 per minute at peak), a single LLM endpoint is sufficient for throughput, so the real challenge is latency, accuracy, and graceful fallback. I would front the system with a fast intent classifier (a fine-tuned small model like a DistilBERT variant) to route simple, repetitive queries to a scripted response layer and complex ones to a full RAG pipeline backed by a vector store of product docs and past tickets. Every response is scored by a lightweight confidence threshold: below the threshold the ticket is flagged and queued for a human agent, and the LLM draft is shown to that agent as a suggested reply so their effort is not wasted. I instrument the whole pipeline with latency, escalation rate, and CSAT metrics from day one so I can measure whether the AI is actually helping or just adding latency.