MLOps interview questions in 2026 cover far more than model training scripts and a Jenkins job. Interviewers now expect you to run classic ML systems (experiment tracking, registries, feature stores, drift monitoring) and LLM systems (prompt versioning, evaluation gates, gateways, GPU serving, guardrails) with the same discipline DevOps teams apply to any production service. This guide collects 70 high-value MLOps and LLMOps interview questions with answers written the way a senior engineer would explain them in a real loop.
How to use this guide
- Freshers and juniors are usually tested on fundamentals: the ML lifecycle, why models degrade, what a registry and feature store do, and how a container gets to Kubernetes. Sections one and two matter most.
- Mid-level MLOps, DevOps and platform engineers are tested on pipelines, CI/CD/CT, monitoring, rollout strategies and LLMOps basics. Expect "how would you build it" follow-ups.
- Senior and lead candidates are tested on GPU serving economics, Kubernetes scheduling, supply-chain security, governance and incident handling. The scenario section is where these interviews are won or lost.
- Read the answer, then say it out loud in your own words with an example from your own work. Interviewers notice memorised paragraphs quickly.
Contents
- MLOps fundamentals (Q1 to Q8)
- Training pipelines, feature stores and CI/CD/CT (Q9 to Q17)
- Model monitoring, drift, A/B and canary (Q18 to Q24)
- LLMOps (Q25 to Q36)
- AgentOps basics (Q37 to Q41)
- GPU serving and autoscaling (Q42 to Q47)
- Kubernetes for ML and LLM workloads (Q48 to Q52)
- Infrastructure as code (Q53 to Q55)
- Security and supply chain (Q56 to Q59)
- Governance (Q60 to Q62)
- Observability (Q63 to Q64)
- Incident scenarios (Q65 to Q70)
- Key takeaways
- Interview preparation checklist
- FAQ
MLOps fundamentals
1. What is MLOps, and how does it differ from traditional DevOps?
Answer: MLOps is the set of practices that takes machine learning models from experiment to reliable production and keeps them healthy afterwards. It borrows DevOps ideas (version control, automation, CI/CD, monitoring, infrastructure as code) and adds what DevOps never had to handle: data as a first-class input, non-deterministic training, models that silently degrade as the world changes, and quality that is measured statistically rather than by pass/fail tests. In DevOps, if the code does not change, behaviour does not change. In ML, behaviour can change because the incoming data changed. That single difference is why MLOps adds data versioning, experiment tracking, model registries, drift monitoring and continuous training.
Interview tip: Close with one sentence on LLMOps: the same discipline applied to systems built on foundation models, where prompts, retrieval indexes and provider model versions become the changing artefacts.
2. Walk me through the MLOps lifecycle end to end.
Answer: A practical lifecycle has seven stages: problem framing and success metric; data collection, validation and versioning; feature engineering; training and experiment tracking; evaluation and approval into a registry; deployment (batch, real-time or edge) with a safe rollout; and monitoring that feeds retraining. The key point is that it is a loop, not a line.
data -> validate -> features -> train -> evaluate
^ |
| v
retrain <- monitor <- deploy <- approve (registry)
Each arrow should be automated and logged, so you can answer "which data, code and parameters produced the model serving traffic right now?" in minutes.
3. What is experiment tracking, and what should you log for every run?
Answer: Experiment tracking records every training run so results are comparable and reproducible. Tools such as MLflow Tracking or Weights & Biases store runs centrally. For each run log: git commit, data version or snapshot ID, feature list, hyperparameters, environment (container image digest, library versions), random seeds, metrics on fixed validation and test sets, training time and hardware, and the resulting artefacts (model file, plots, confusion matrix). Without the data version and image digest, a run is not truly reproducible.
Real-world example: A credit-risk team that logs only metrics can tell you run 41 was better than run 38 but not why. Logging the data snapshot often reveals the "better" model simply saw a different month of data.
4. What does a model registry do, and how do you use it for promotion?
Answer: A model registry is the system of record for model versions that are candidates for production. Each version links back to the run, data and code that produced it, plus metadata such as evaluation results, owner, approval status and model card. Promotion should be a controlled state change, not a file copy. In MLflow's Model Registry, for example, you can point a mutable alias such as champion at a specific version and have serving resolve the alias, so promotion and rollback are both a metadata change with an audit trail. The deployment pipeline listens for that change, runs gates, and rolls out.
Interview tip: Mention that the registry should be the only path to production. If someone can scp a pickle file onto a server, you do not have a registry, you have a folder.
5. How do you version data, code and models together?
Answer: Code goes in git. Data is too large for git, so you version it by reference: tools such as DVC store content hashes in git while the files live in object storage; lakehouse table formats such as Delta Lake or Apache Iceberg give you time travel by snapshot ID. Models are versioned in the registry. The glue is lineage: the training run records the commit hash, data snapshot ID and image digest, and the registry version points to that run. Then any production model can be traced back to exactly what made it.
6. What makes an ML pipeline reproducible, and where does reproducibility usually break?
Answer: Reproducibility needs pinned inputs (data snapshot, code commit), a pinned environment (container image by digest, locked dependency versions), controlled randomness (seeds, deterministic flags where the framework supports them) and recorded configuration. It usually breaks in four places: "latest" data queries that read whatever is in the table today, unpinned pip installs, GPU non-determinism in some operations, and feature logic that exists only in a notebook. Aim for "reproducible within tolerance" for deep learning, since bit-identical results across hardware are not always realistic.
7. What is the difference between batch inference and real-time inference, and how do you choose?
Answer: Batch inference scores large volumes on a schedule (nightly churn scores, weekly demand forecasts) and writes results to a table; it is cheap, easy to retry and tolerant of slow models. Real-time inference serves a prediction per request behind an API with a latency budget (fraud check during card authorisation, recommendations on page load). Choose by asking when the decision is made and how fresh the input must be. Many systems mix both: precompute heavy features in batch, compute a few fresh features online, and serve a lightweight model in real time. Streaming inference on an event bus sits between the two.
8. What are the common ways to deploy a model, and how do the big managed platforms map to them?
Answer: Common patterns: embedded in the application, a containerised model server behind an API (on Kubernetes, a container service or serverless), a managed endpoint, batch jobs, streaming consumers and edge deployment. The managed ML platforms package these patterns: Amazon SageMaker AI (the ML service formerly called Amazon SageMaker), Azure Machine Learning, and on Google Cloud the services previously known as Vertex AI, which Google has brought under the Gemini Enterprise Agent Platform name (check current documentation for naming). Each offers training jobs, pipelines, a registry, managed endpoints and monitoring. Open-source stacks combine pieces like MLflow, Kubeflow Pipelines and KServe on Kubernetes instead.
Interview tip: Do not argue that one cloud is superior. Explain the trade-off: managed platforms reduce operational load; Kubernetes-based stacks give portability and control at the cost of running them yourself.
Training pipelines, feature stores and CI/CD/CT
9. What is a feature store, and what problem does it actually solve?
Answer: A feature store is a system that defines features once and serves them consistently for training and inference. Feast, an open-source example, describes it as an offline store for historical feature extraction during training and an online store for low-latency serving, with point-in-time correct joins to avoid data leakage. The real problems it solves are training-serving skew (two implementations of the same feature drifting apart), leakage (training rows accidentally using values from the future), and duplication (five teams computing "customer 30-day spend" five ways).
Interview tip: Say when you would not use one. A single batch model with features from one warehouse table rarely justifies a feature store.
10. What is training-serving skew, and how do you prevent it?
Answer: Training-serving skew is when the model sees different feature values in production than it saw in training for the same entity. Causes: separate code paths (SQL for training, Java for serving), different defaults for nulls, time-zone handling, stale online features, or a preprocessing step left out of the serving container. Prevention: one shared feature definition, packaging preprocessing with the model (for example, a pipeline object or a transform layer saved with the model), logging the features actually used at inference, and periodically comparing logged serving features against training distributions.
11. How do you orchestrate an end-to-end training pipeline?
Answer: Break the workflow into steps with explicit inputs and outputs (ingest, validate, build features, train, evaluate, register), run each step in a container, and let an orchestrator handle dependencies, retries, caching and scheduling. Kubeflow Pipelines runs pipeline steps as containers on Kubernetes; Apache Airflow schedules DAGs and is common for data-heavy workflows; managed options include SageMaker AI Pipelines and Azure Machine Learning pipelines. Make steps idempotent, cache expensive steps by input hash, and have the final step register a candidate rather than deploy directly.
[ingest]->[validate]->[features]->[train]->[evaluate]
|
pass gates? v
[register candidate]
12. How do you automate hyperparameter tuning without wasting compute?
Answer: Use a tuning service or library (Optuna, Ray Tune, Katib on Kubeflow, or the managed tuners on cloud ML platforms) with a search strategy smarter than full grid search: random search for a baseline, Bayesian optimisation for expensive models, and early stopping (for example, Hyperband-style pruning) to kill poor trials quickly. Cap total trials and budget, log every trial to experiment tracking, and tune on a validation set that is never used for the final test. Re-tune occasionally, not on every retrain.
13. Explain CI, CD and CT in the context of ML.
Answer: Continuous integration tests the code: unit tests for feature logic, schema tests, linting, and a small training run on sample data to catch broken pipelines. Continuous delivery packages and promotes the model and serving code through environments with automated gates and an approval step. Continuous training (CT) retrains the model automatically when triggers fire (schedule, new labelled data, drift) and pushes the result back through the same evaluation gates. The key difference from software CI/CD is that a "build" can pass every unit test and still produce a worse model, so evaluation against a fixed benchmark is a mandatory gate. The CI/CD for AI applications guide goes deeper on pipeline design.
14. What do you test in an ML system beyond unit tests?
Answer: Five layers. Data tests: schema, ranges, null rates, uniqueness, freshness (tools like Great Expectations or pandera). Feature tests: transformation logic on known inputs. Model tests: metric thresholds on a fixed holdout, slice metrics for important segments, comparison with the current champion, and behavioural tests (invariance to irrelevant changes, expected direction for monotonic features). Serving tests: the container loads the model, returns valid output, and meets latency on representative payloads. Integration tests: end-to-end from request to logged prediction.
Real-world example: An insurer's claims model may beat the champion overall but be worse on a small, regulated segment such as senior citizens. Slice gates catch this before customers do.
15. How do you ensure data quality and consistency in the ML pipeline?
Answer: Treat data like an API with a contract. Define expected schema and statistical expectations, validate at ingestion and again before training, and fail the pipeline (or quarantine the batch) when checks break rather than training on bad data. Track freshness and volume anomalies, keep lineage from source to feature, and agree data contracts with upstream owners so schema changes are announced, not discovered. Record validation results with the training run so you can see what quality the model was trained on.
16. What are the challenges of training on large datasets, and how do you handle them?
Answer: Challenges: data that does not fit in memory, slow input pipelines starving GPUs, long runs that fail midway, and cost. Responses: stream data in shards from object storage in efficient formats (Parquet, TFRecord, WebDataset), parallelise data loading, use distributed training (data parallelism for most cases; model, tensor or pipeline parallelism when the model itself is too large), checkpoint regularly so spot or preemptible instances are safe to use, use mixed precision, and profile before adding hardware. Often the cheapest fix is a smaller, well-chosen sample for iteration and the full set only for final training.
17. Your fraud model is retrained every night automatically. Last week a retrained model passed all gates and still made fraud catch worse. How would you redesign the retraining process?
Answer: Passing gates and still regressing means the gates measured the wrong thing or measured it on the wrong data. I would make retraining trigger-based rather than blindly nightly, freeze an evaluation set plus a rolling recent set, add slice and cost-weighted metrics, and require the candidate to beat the champion on recent labelled data before a shadow or canary phase.
What I would check:
- Whether the validation set leaked recent fraud patterns the model then overfit.
- Label delay: fraud labels arrive weeks later, so "last week's" metrics may have used incomplete labels.
- Whether gates used accuracy or AUC while the business cares about catch rate at a fixed alert volume.
- Feature changes upstream that altered meaning without changing schema.
Production consideration: For high-impact models, automated retraining should produce a candidate, not a deployment. Promotion goes through shadow comparison and a human approval recorded in the registry.
Model monitoring, drift, A/B and canary
18. What is the difference between data drift and concept drift?
Answer: Data drift (covariate shift) means the distribution of inputs changes: new customer segments, a new app version sending different values, seasonal patterns. Concept drift means the relationship between inputs and the target changes: the same transaction pattern that was safe last year now signals fraud because attackers adapted. Data drift can be detected without labels by comparing feature distributions. Concept drift usually needs labels or proxy outcomes, because the inputs can look identical while the right answer has changed. There is also prediction drift (the output distribution moves) and label shift (the base rate changes).
19. How do you detect drift in practice?
Answer: Compare a recent window of production data with a reference window (usually training data or a known-good period) per feature and for predictions. Common statistics: Population Stability Index, Kolmogorov-Smirnov test for continuous features, chi-squared for categorical features, and distance measures such as Jensen-Shannon. Libraries like Evidently and the monitoring features of managed ML platforms automate this. The hard part is not the statistic but the thresholds: large-volume tests flag tiny, harmless shifts as significant, so tune alerts on business impact and prioritise the features the model relies on most.
Interview tip: Say that drift is a signal to investigate, not an automatic retrain trigger. Retraining on corrupted data makes things worse.
20. How do you monitor model quality when ground-truth labels arrive late or never?
Answer: Use layers. Immediately: input drift, prediction distribution, confidence scores, null and out-of-range rates, and operational metrics. Short term: proxy outcomes (click, complaint, manual override rate, analyst disagreement). When labels arrive: true metrics joined back by prediction ID, backfilled into dashboards. For critical models, label a small random sample by human review every week. Log every prediction with a request ID, model version and features so the later join is possible.
21. What is A/B testing in the context of model deployment, and how is it different from offline evaluation?
Answer: Offline evaluation measures a model on historical data. An A/B test randomly splits live traffic (by user, not by request, to avoid inconsistent experiences) between the champion and a challenger and measures the business metric (conversion, loss rate, handling time) with a pre-defined sample size and duration. It answers "does this model change outcomes?", which offline metrics cannot, because the model's decisions change user behaviour. Guard against peeking at results early, novelty effects and running too many simultaneous tests on the same users.
22. Compare shadow deployment, canary release and blue-green for models.
Answer:
| Strategy | How it works | Use it when |
|---|---|---|
| Shadow | New model receives a copy of live traffic; its outputs are logged, not used | You need real-traffic comparison with zero user impact |
| Canary | A small, growing share of traffic uses the new model, with automated rollback on metric breaches | You want gradual exposure and fast rollback |
| Blue-green | Two full environments; traffic switches at once and can switch back | Infrastructure or serving changes where a clean cutover is simpler |
| A/B | Randomised split held for a measured experiment | You must prove business impact, not just safety |
In practice teams chain them: shadow first, then canary, then a full rollout or an A/B test. Serving layers such as KServe, service meshes and Argo Rollouts support traffic splitting.
23. How do you handle model versioning and rollback in production?
Answer: Every deployed model is an immutable registry version, served by a container image that is also immutable. The deployment manifest references the version explicitly. Rollback means pointing traffic back to the previous version, which should still be warm (or quick to load) for a defined period after rollout. Feature and schema compatibility matter: if the new model needed a new feature, rolling back the model must not break the feature pipeline, so make feature changes backward compatible first. Every prediction log includes the model version, so you can tell which version produced which decision during an incident.
24. What is model explainability, and why do regulated industries like banking and healthcare care about it?
Answer: Explainability is the ability to show why a model produced a given output, globally (which features matter overall) and locally (why this applicant was declined). Techniques include inherently interpretable models, SHAP values, permutation importance and counterfactual explanations. Banks need it to give adverse-action reasons, validate models under model risk management, and detect proxy discrimination. Hospitals need clinicians to trust and challenge a recommendation rather than follow it blindly. Operationally, store explanations with predictions for high-stakes decisions and monitor explanation drift as well as data drift.
If you want to build these skills hands-on across ML, GenAI, cloud and security in one structured track, Cloudsoft's APEX AI, ML, Cloud and Cyber Security program covers the path from models to deployed, monitored systems. Classroom in Ameerpet or live online; call +91 96660 19191 for a free demo.
LLMOps
25. What is LLMOps, and what changes compared with classic MLOps?
Answer: LLMOps is MLOps for applications built on large language models, usually consuming a foundation model rather than training one. What changes: the artefacts you version become prompts, model and provider versions, retrieval indexes, tool definitions and guardrail policies; evaluation moves from a single metric to rubric-based and task-specific evaluation of free text; cost is per token and varies with every request; and new failure modes appear, such as hallucination, prompt injection and silent behaviour changes when the provider updates a model. What stays the same: versioning, automated gates, safe rollout, monitoring and incident response. The AI DevOps and LLMOps explainer compares LLMOps, MLOps, DevOps and AIOps side by side.
26. How do you version and manage prompts in production?
Answer: Prompts are code. Store them as versioned templates, either in git next to the application or in a prompt registry (for example, MLflow's Prompt Registry keeps immutable prompt versions and lets you point aliases at them; Langfuse and LangSmith offer prompt management too). Each prompt version should be bound to the model it was tested with, the evaluation results, and an owner. The application loads a prompt by name and alias, so changing the prompt is a reviewed, evaluated release with rollback, not an edit on a console. Log the prompt version with every trace.
Interview tip: Point out that a prompt tuned for one model often behaves differently on another, so prompt and model versions are promoted as a pair.
27. What is an evaluation gate for an LLM application, and what goes into it?
Answer: An evaluation gate is an automated check in CI/CD that blocks a release if quality, safety or cost regress. It runs a versioned golden dataset (real, anonymised questions with reference answers or rubrics) through the candidate configuration and scores it with deterministic checks (format, JSON schema, citation present, refusal where expected), task metrics (for RAG: faithfulness, answer relevance, context precision via tools such as Ragas), LLM-as-judge rubrics calibrated against human ratings, safety tests (jailbreak and injection prompts), and latency and token cost per case. Thresholds compare against the current production baseline. The LLM evaluation guide covers metric design in detail.
28. How do you make LLM-as-judge evaluations trustworthy?
Answer: Treat the judge as a model that needs its own validation. Write specific rubrics with examples per score, ask for a short rationale before the score, use pairwise comparison when absolute scores are noisy, randomise answer order to reduce position bias, pin the judge model version, and calibrate by having humans label a sample and measuring agreement. Re-check agreement whenever you change the judge model or rubric. Never let the system under test grade itself with the same prompt, and keep deterministic checks for anything that can be checked deterministically.
29. What is an LLM gateway, and why do platform teams put one in front of model providers?
Answer: An LLM gateway (or AI gateway) is a proxy layer between applications and model providers that exposes one consistent API. It centralises authentication and key management, routing across providers and models, fallbacks and retries, rate limits and quotas per team, token and cost accounting, caching, logging and tracing, and policy enforcement such as PII redaction or content filters. Applications stop embedding provider keys, and the platform team can switch models or providers without code changes in every service. Examples include open-source proxies such as LiteLLM, Envoy-based AI gateways and API-management products from cloud vendors. The LLM gateway explainer covers routing patterns.
apps -> [ LLM gateway ] -> provider A
auth | quota -> provider B
route | cache -> self-hosted model
log | redact
30. How do you trace an LLM application request end to end?
Answer: Instrument every step as a span under one trace ID: incoming request, retrieval (query, top-k document IDs, scores), prompt assembly (template version, token count), each model call (model, parameters, input and output tokens, latency, finish reason), tool calls, guardrail decisions and the final response. OpenTelemetry is the standard transport; its generative AI semantic conventions are still evolving, so check the current specification. LLM-focused tools such as Langfuse and LangSmith add prompt and evaluation views on top. Mask or hash sensitive content before export, and attach user feedback to the same trace so bad answers can be debugged from the exact context.
31. How do you track and control LLM cost?
Answer: Measure first: record input, output and cached tokens per call with tags for team, feature, tenant and model, then convert to cost using the current price list in a lookup table rather than hardcoded numbers. Report cost per successful task, not just cost per call, because retries and agent loops multiply calls. Control levers: route easy requests to smaller models, trim context (fewer, better retrieved chunks), prompt caching where the provider supports it, response caching for repeated questions, output length limits, batch APIs for offline work, and per-team budgets and alerts at the gateway. Review the top cost drivers weekly.
Real-world example: Consider a Hyderabad GCC running an internal HR assistant. A tagging scheme often shows that one feature (say, policy document summarisation) dominates spend because it sends whole documents every time; caching summaries fixes it without touching the model.
32. What are guardrails, and where do they sit in the request path?
Answer: Guardrails are controls that constrain what goes into and comes out of a model. Input side: authentication and authorisation, PII detection and redaction, prompt-injection and jailbreak detection, topic restrictions. Retrieval side: permission filtering so users only retrieve documents they can access. Output side: schema validation, grounding checks against retrieved sources, toxicity and policy filters, and blocking secrets or PII in responses. Action side for agents: allow-listed tools, argument validation and human approval for risky actions. They are layered with the model's own safety behaviour, not a replacement for it, and each decision should be logged. See the AI guardrails guide for patterns.
33. A provider announces that the model version your production assistant uses will be retired. How do you plan the upgrade?
Answer: Treat it as a release with its own evaluation, not a configuration change. Pin model versions explicitly everywhere so nothing upgrades silently, then run the candidate model through the full golden set with the current prompts, tune prompts if needed, compare quality, safety, latency and cost against the baseline, and roll out via the gateway with a canary and quick fallback.
What I would check:
- Output format changes that break parsers, tool-call schemas or structured outputs.
- Changes in refusal behaviour, verbosity and citation style.
- Tokenizer or context-window differences that change cost and truncation.
- Whether any embeddings depend on the retiring model (that requires re-indexing, see Q35).
Production consideration: Keep a model inventory with retirement dates per deployment, so upgrades are scheduled months ahead rather than discovered the week a version stops responding.
34. What does RAG index operations involve?
Answer: RAG index ops is everything needed to keep the retrieval layer correct and fresh: ingestion pipelines that parse and chunk documents, embedding jobs, incremental updates (new, changed and deleted documents), access-control metadata synced from the source system, index versioning, rebuilds, and monitoring of freshness, coverage and retrieval quality. Deletions are the most commonly broken part: a document removed from SharePoint must disappear from the index quickly, especially for privacy requests. Track document counts per source, last successful sync, and retrieval metrics on a fixed query set after each refresh. For retrieval design itself, see the RAG interview questions.
35. You need to change the embedding model behind a large RAG index. How do you do it without downtime?
Answer: Vectors from different embedding models are not comparable, so you cannot mix them in one index. Build a new index in parallel with the new model (blue-green for indexes), backfill all documents while the old index keeps serving, dual-write new updates to both, evaluate retrieval quality on a fixed query set, then switch the alias the application reads and keep the old index for rollback.
What I would check:
- Re-embedding cost and throughput limits; schedule backfills in batches.
- Vector dimension changes and index configuration (distance metric, HNSW parameters).
- Whether query embedding and document embedding switch at exactly the same moment.
- Recall and answer quality on the golden set, including hybrid search behaviour.
Production consideration: Store the embedding model name and version as metadata on every index, so nobody accidentally queries a new index with the old model.
36. When would you use semantic caching, and what are its risks?
Answer: Semantic caching returns a stored answer when a new query is close enough in embedding space to a previous one. It suits high-volume, repetitive, non-personal questions (public FAQs, product information). Risks: returning an answer to a question that is similar but different ("cancel my order" versus "can I cancel an order"), leaking one user's personalised answer to another, and serving stale answers after documents change. Mitigate with conservative similarity thresholds, cache keys that include tenant, user permissions and index version, TTLs tied to content refresh, and excluding personal or action-taking requests from the cache entirely.
AgentOps basics
37. What is AgentOps, and how does it extend LLMOps?
Answer: AgentOps is the operational discipline for AI agents: systems where a model plans, calls tools and takes multiple steps toward a goal. It extends LLMOps because a single request can now trigger many model calls and real side effects (tickets created, emails sent, records updated). The new concerns are run-level tracing across steps, tool permissions and identity, step and cost budgets, loop detection, human approval for risky actions, trajectory evaluation, and versioning of tools and agent graphs as well as prompts. The agentic AI interview questions go deeper on design patterns.
38. How do you evaluate an agent, not just its final answer?
Answer: Evaluate three levels. Outcome: did the task complete correctly (often checkable against a sandboxed system state, such as "ticket exists with correct fields")? Trajectory: were the right tools called in a sensible order with valid arguments, with no unnecessary or forbidden calls? Efficiency and safety: steps, tokens, latency, cost, and whether it asked for approval when required. Run scenarios in a sandbox with mocked or test-instance tools, repeat each scenario several times because agents are non-deterministic, and track pass rates rather than single runs. The AI agent evaluation guide covers scenario design.
39. How do you control what tools an agent can use and how much it can spend?
Answer: Give the agent its own identity with least-privilege credentials, scoped per tool, ideally acting on behalf of the user so it cannot exceed that user's permissions. Allow-list tools per agent, validate tool arguments against schemas, separate read tools from write tools, and require human approval for irreversible or high-value actions. Enforce budgets in the runtime: maximum steps, maximum tool calls, token and cost ceilings per run, and timeouts. When an MCP (Model Context Protocol) server exposes tools, treat it like any API dependency: versioned, authenticated, monitored and reviewed before adoption.
40. How do you deploy and version an agent safely?
Answer: Version the whole configuration as one release unit: agent graph or workflow code, prompts, model versions, tool definitions and schemas, and guardrail policies. Run trajectory evaluations in CI, deploy to a staging environment wired to test instances of business systems, then canary on a limited user group with write actions behind approval. Keep durable state (for example, checkpointed graph state in frameworks like LangGraph) compatible across versions, or drain in-flight runs before switching. Feature flags let you disable a single risky tool without redeploying the agent.
41. An internal IT-ops agent starts creating duplicate incident tickets in a loop. What do you do?
Answer: Stop the harm first: disable the ticket-creation tool through its feature flag or revoke its write scope, and let the agent continue in read-only mode or switch it off. Then clean up duplicates and find the root cause from traces.
What I would check:
- Traces for the looping runs: did tool calls return errors or timeouts that the agent interpreted as "not created, retry"?
- Whether the tool is idempotent (an idempotency key per intended ticket would have prevented duplicates).
- Recent changes to prompts, model version or the ticketing API response format.
- Why step budgets and loop detection did not stop the run.
Production consideration: Write tools should be idempotent, rate-limited per run, and covered by an alert on unusual write volume. These are standard controls for any automation, applied to agents.
GPU serving and autoscaling
42. How does a modern LLM inference server such as vLLM achieve high throughput?
Answer: Generation is memory-bound and the KV cache (stored attention keys and values for each sequence) is the main memory consumer. vLLM-style servers manage the KV cache in fixed-size blocks (vLLM calls this PagedAttention) so memory is not wasted on reserved but unused space, and use continuous batching: new requests join the running batch at each decoding step instead of waiting for the whole batch to finish. Further techniques include prefix caching (reusing KV cache for shared prompt prefixes such as system prompts), chunked prefill, quantisation, speculative decoding, and tensor or pipeline parallelism for models larger than one GPU. Most such servers expose an OpenAI-compatible API, so applications and gateways can switch between hosted and self-hosted models with little code change.
43. Which metrics matter for LLM serving, and how do they differ from classic API metrics?
Answer: Beyond request rate, errors and total latency, LLM serving needs time to first token (TTFT, dominated by prefill and queueing), time per output token or inter-token latency (decode speed, what users feel while streaming), tokens per second per GPU (throughput and cost efficiency), queue depth and waiting requests, KV cache utilisation, batch size, and GPU memory and utilisation. Report percentiles, not averages, and segment by prompt length, because a few very long prompts can dominate tail latency. Interactive chat optimises TTFT; offline summarisation optimises throughput.
44. How do you autoscale GPU inference, and why is CPU utilisation the wrong signal?
Answer: CPU barely moves on a GPU-bound server, and GPU utilisation can look high whether the server is healthy or overloaded. Scale on signals that track user experience and saturation: queue length or waiting requests, KV cache usage, in-flight requests per replica, or TTFT percentiles, exported by the server and fed to the Horizontal Pod Autoscaler through a metrics adapter or to an event-driven autoscaler such as KEDA. Add node autoscaling (Cluster Autoscaler or Karpenter on EKS) for GPU nodes. Because GPU nodes and model loading are slow, keep a warm minimum, scale out early, and scale in slowly.
45. What causes slow cold starts for LLM pods, and how do you reduce them?
Answer: Cold start is the sum of node provisioning, pulling a large container image, downloading model weights (often many gigabytes), loading weights into GPU memory and warming up kernels. Reduce each part: keep a warm pool of GPU nodes, use slim images with weights stored separately, cache weights on node-local disk or a shared high-throughput volume, pre-pull images onto nodes, use loading formats that support fast memory mapping (safetensors), and gate traffic with readiness probes that only pass after a warm-up request. Scale to zero only for workloads that can tolerate minutes of first-request latency.
46. What are the trade-offs of quantisation for serving?
Answer: Quantisation stores weights (and sometimes activations or KV cache) in lower precision such as FP8, INT8 or INT4 using methods like GPTQ or AWQ. Benefits: less GPU memory, so a model fits on fewer or smaller GPUs, larger batches and often higher throughput. Costs: possible quality loss that varies by task (reasoning and code can degrade more than chat), hardware and kernel support differences, and extra validation effort. Run your own evaluation set on the quantised model before switching; never assume a published benchmark transfers to your domain.
47. Your team wants to self-host an open-weight model instead of calling a hosted API. How do you evaluate the decision?
Answer: Self-hosting makes sense for data residency, strict network isolation, predictable high volume, latency control or custom fine-tunes; hosted APIs win on time to value, frontier quality and zero GPU operations. I would compare quality on our own evaluation set, total cost (GPUs at realistic utilisation, engineers on call, idle capacity) against token pricing at forecast volume, and operational readiness. The self-hosting LLMs guide walks through the decision.
What I would check:
- The model licence and acceptable-use terms for our use case.
- GPU availability and quota in our region and cloud account.
- Whether utilisation will be high enough; idle GPUs are the hidden cost.
- Who patches, upgrades and evaluates new model versions.
Production consideration: Put both options behind the same gateway with an OpenAI-compatible interface, so you can move traffic between hosted and self-hosted models as costs and quality change.
Kubernetes for ML and LLM workloads
48. How are GPUs scheduled on Kubernetes?
Answer: Traditionally a vendor device plugin (for example, NVIDIA's, usually installed via the GPU Operator) advertises GPUs as an extended resource, and pods request whole GPUs in resource limits. Dynamic Resource Allocation (DRA), whose core APIs graduated to general availability in Kubernetes 1.34, adds a richer model where pods claim devices with selectable attributes through ResourceClaims, which helps with sharing and specific device types. Supporting practices: taint GPU nodes so only GPU workloads land there, use node affinity for GPU type, consider GPU partitioning or time-slicing for small models, and use gang or queue-based scheduling (for example, Kueue or Volcano) for distributed training jobs that need all workers at once.
49. How do you serve models on Kubernetes, and what does a serving layer like KServe add?
Answer: At minimum you run a model server container as a Deployment behind a Service, with probes, resource requests and an HPA. A serving layer such as KServe adds a custom resource (InferenceService) that bundles the model location, runtime, autoscaling, canary traffic splitting and standard inference protocols, with support for LLM runtimes such as vLLM and OpenAI-compatible APIs. It reduces boilerplate and standardises how many teams deploy models. The trade-off is another control plane to operate and upgrade. The Kubernetes for AI engineers guide covers the core objects.
50. Why is ordinary load balancing a poor fit for LLM pods, and what are the alternatives?
Answer: Round-robin assumes requests cost roughly the same. LLM requests vary hugely in prompt and output length, and a replica that already holds a matching prefix in its KV cache can answer much faster. Blind balancing overloads some replicas while others idle and wastes cache. Alternatives: least-outstanding-requests balancing, prefix-aware or cache-aware routing, and the Kubernetes Gateway API Inference Extension, which introduces an InferencePool backend and an endpoint picker that routes using model-server metrics such as queue and KV cache state. It is implemented by several gateways; check the current project status for your gateway.
51. What role do Kubeflow and Helm play in an ML platform on Kubernetes?
Answer: Kubeflow is a collection of Kubernetes-native ML components: Pipelines for orchestrated workflows, notebooks, Katib for hyperparameter tuning, training operators for distributed jobs, and integration with KServe for serving. Teams adopt individual components rather than all of it. Helm packages Kubernetes applications as versioned charts with values per environment, so model servers, monitoring stacks and platform components are installed consistently. Combined with GitOps (Argo CD or Flux syncing charts from git), every environment's state is declared, reviewed and reversible.
52. How would you use GitOps to deploy model versions to EKS, AKS or GKE?
Answer: Keep the desired state in a git repository: a manifest or Helm values file per environment naming the model version, serving image digest and traffic split. When a model is approved in the registry, the pipeline opens a pull request that bumps the version in the staging values; merging triggers Argo CD to sync. After canary analysis passes, a second pull request promotes to production. Rollback is a git revert. This works the same on EKS, AKS and GKE; only identity integration (IAM roles for service accounts or EKS Pod Identity, Microsoft Entra Workload ID, GKE Workload Identity Federation) and storage classes differ.
Infrastructure as code
53. How do you apply infrastructure as code to an ML or LLM platform?
Answer: Define everything the platform needs in code: networks, Kubernetes clusters and GPU node pools, object storage for data and models, registries, databases (including vector stores), IAM roles, secrets references, monitoring and budgets. Terraform (or OpenTofu) is the common choice across clouds; CloudFormation, CDK, Bicep and Pulumi are alternatives. Structure as reusable modules (for example, a "model-serving namespace" module that creates the namespace, service account, quotas and dashboards), keep state remote and locked, and run plan in pull requests with apply only from the pipeline. The Terraform for AI infrastructure guide shows module layouts.
54. How do you manage expensive GPU infrastructure through IaC without runaway cost?
Answer: Encode cost controls in the code itself: GPU node pools with explicit minimum and maximum sizes, spot or preemptible pools for interruptible training, schedules or automation that scale development pools to zero outside working hours, mandatory cost-allocation tags, and budgets with alerts per project. Add policy as code (Open Policy Agent, Sentinel or cloud-native policies) so nobody can create an untagged or oversized GPU pool, and run cost estimation in pull requests so reviewers see the impact before merging.
55. Someone changed a GPU node pool in the console during an incident, and now Terraform wants to destroy and recreate it. What do you do?
Answer: Do not apply blindly; recreating a serving node pool mid-day could take down inference. First understand the drift, then decide whether the console change was correct and should be codified, or should be reverted in a controlled window.
What I would check:
- The plan output: which attribute forces replacement (instance type, labels, taints, disk size).
- Whether the change can be expressed in code without replacement, or applied with a rolling node-pool update.
- Workload impact: PodDisruptionBudgets, spare capacity, model reload time.
- Who has console write access in production and why.
Production consideration: Run scheduled drift detection, restrict console write access to break-glass roles with audit logging, and require a follow-up pull request after any emergency console change.
Security and supply chain
56. What is model provenance, and how do you establish it?
Answer: Provenance is verifiable evidence of where a model came from and what happened to it: source (internal training run or a public hub), base model and licence, training data references, code commit, build environment, evaluation results and who approved it. Establish it by recording lineage in the registry, computing and storing hashes of weight files, signing artefacts at build time, and verifying signatures and hashes before deployment. The OpenSSF Model Signing project provides a library and CLI for signing models, with Sigstore as a supported option. SLSA-style build provenance applies to model pipelines as well as software builds.
57. Why is loading pickle-based model files risky, and why do teams prefer safetensors?
Answer: Python's pickle format can execute arbitrary code during deserialisation, so loading an untrusted .pkl or older PyTorch checkpoint file is effectively running someone else's code inside your training or serving environment. Safetensors is a format for storing tensors that contains only a header and raw tensor data, so loading it does not execute code, and it supports fast, zero-copy and partial loading. Controls: prefer safetensors for weights, scan any pickle-based artefacts with model scanners, load untrusted models only in isolated sandboxes, and mirror approved models into an internal registry instead of pulling from public hubs at runtime.
58. How do you secure the ML and LLM software supply chain end to end?
Answer: Cover every input. Dependencies: pinned, scanned, from an internal mirror. Container images: minimal bases, vulnerability scanning, SBOMs, signed and verified at admission to the cluster. Models: provenance, signatures, safe formats, licence review, and an AI bill of materials listing models, datasets and their versions. Data: access-controlled, with lineage, and checks for poisoning where training data comes from users or the web. Pipelines: short-lived credentials via workload identity, protected branches, and separation between who can train and who can promote. The DevSecOps for enterprise AI guide maps these controls to pipeline stages.
59. A data scientist wants to pull a new open-weight model from a public hub straight into the production cluster. How do you respond?
Answer: Support the goal, redirect the path. The model goes through an intake process: licence review, safe-format check or conversion, scanning, evaluation on our tasks and safety tests, then it is mirrored into the internal model registry with hashes and provenance. Production only pulls from the internal registry.
What I would check:
- Licence terms and any usage restrictions.
- File formats in the repository, including any pickle files or custom code that requires remote-code execution flags.
- Who published it and whether hashes match an official release.
- Evaluation results, including jailbreak and bias tests where relevant.
Production consideration: Enforce it technically with egress controls and admission policies, not just a wiki page; otherwise the shortcut will be used under deadline pressure.
Governance
60. What does model governance look like in practice for an ML platform team?
Answer: Governance is making sure the organisation knows what models exist, who owns them, what risk they carry and that they were approved appropriately. Concretely: a model inventory (including LLM applications and third-party AI), risk tiering by impact, model cards documenting purpose, data, limitations and evaluation, approval workflows in the registry with separation of duties for high-risk models, periodic revalidation, and audit logs of every promotion and rollback. Frameworks such as ISO/IEC 42001 and regulations such as the EU AI Act shape requirements depending on where the system is used. The enterprise AI governance guide covers operating models.
61. How do you comply with data privacy laws such as GDPR and India's DPDP Act in ML pipelines?
Answer: Build privacy into the pipeline rather than auditing it later: collect only the data needed for a stated purpose, record the legal basis or consent, pseudonymise or tokenise identifiers before data reaches training, restrict access by role, keep data residency requirements in mind when choosing regions and model providers, set retention periods, and support erasure requests. Erasure is hard for trained models, so design for it: keep training data traceable to individuals, document retraining cadence, and for LLM apps ensure deletion also reaches RAG indexes, caches and logs. Involve the privacy or legal team for interpretation; engineers implement controls, they do not interpret law alone.
62. An auditor asks you to show why your bank's LLM assistant gave a specific answer to a customer three months ago. What do you need to have in place?
Answer: Reconstructability. For every interaction you need the trace ID linked to the customer session, the exact prompt template version, model and provider version, retrieved document IDs and their versions, guardrail decisions, tool calls and the final response, retained according to policy and protected from tampering.
What I would check:
- Retention policy covers the audit period and storage is access-controlled.
- Document versions are preserved, not just the current version of each policy page.
- Logs mask sensitive data in a way that still allows authorised reconstruction.
- Release records link each configuration version to its evaluation and approval.
Production consideration: Decide what to log during design with compliance and security teams. Retrofitting audit trails after an auditor asks is far harder than building them in.
Observability
63. What should an observability stack for ML and LLM systems cover?
Answer: Four layers on shared trace IDs. Infrastructure: nodes, GPUs (for example, via NVIDIA's DCGM exporter), memory, network. Service: request rate, errors, latency percentiles, saturation, and for LLMs TTFT and tokens per second. Model and data: input drift, prediction distribution, quality metrics when labels arrive, retrieval quality for RAG. Business and cost: task success, user feedback, escalations, cost per task. Typical tooling is Prometheus and Grafana for metrics, OpenTelemetry for traces, a log platform, and an LLM tracing tool such as Langfuse or LangSmith. The AI observability guide covers dashboard design.
64. How do you define SLOs for an LLM application?
Answer: Combine reliability SLOs with quality SLOs. Reliability: availability of the endpoint, TTFT and full-response latency percentiles, error rate including provider errors and timeouts. Quality: a minimum pass rate on continuously sampled production traffic scored by automated evaluators, groundedness rate for RAG answers, guardrail false-block rate, and user-reported bad-answer rate. Cost can be a budget objective per task. Use error budgets the SRE way: when quality burns faster than expected, freeze prompt and model changes and investigate, exactly as you would freeze feature releases for reliability.
Incident scenarios
65. A retailer's demand-forecasting model suddenly predicts near-zero sales for several product categories. Nothing in the model changed. How do you investigate?
Answer: If the model did not change, assume the data did. Freeze automated retraining so the bad data does not become the new model, switch downstream replenishment to a fallback (last known good forecasts or a simple baseline), then trace the inputs.
What I would check:
- Feature null rates and distributions for the affected categories versus last week.
- Upstream changes: renamed category codes, a new source system, a unit change, a failed join returning defaults.
- Pipeline logs for partial loads or schema changes that passed type checks but changed meaning.
- Whether the issue is confined to specific stores, regions or dates.
Production consideration: Add data contracts with the upstream team and validation checks on category coverage and value ranges, so the pipeline fails loudly next time instead of forecasting zeros quietly.
66. LLM spend for an internal assistant doubles overnight with no increase in users. What do you do?
Answer: Break cost down by tags at the gateway: feature, team, model, tenant, input versus output tokens, calls per session. A step change with flat users almost always means more calls per request, larger prompts, or traffic routed to a more expensive model.
What I would check:
- Deployments in the last day: prompt changes, a larger top-k in retrieval, a new agent step, or a routing rule change.
- Retry storms caused by provider errors or timeouts.
- Agent loops or runaway sessions with very high call counts.
- Automated clients (scripts, tests pointed at production) using a shared key.
Production consideration: Set per-key and per-feature budgets with alerts at the gateway, plus per-session call limits, so a cost incident pages someone in hours rather than appearing on the monthly invoice.
67. After a scheduled provider model update, a hospital's discharge-summary assistant starts omitting medication changes. What is your response?
Answer: This is a patient-safety issue, so contain first: roll back to the previously pinned model version or fallback path through the gateway if available, or pause the feature and inform clinical owners. Then reproduce using stored traces and the golden set.
What I would check:
- Whether the model version was pinned or used a floating alias that changed silently.
- Golden-set cases covering medication changes and the pass rate before and after.
- Output length or truncation changes that drop later sections of the summary.
- Whether evaluation gates included a required-field check for medication changes.
Production consideration: For clinical content, add deterministic checks for required sections, keep clinician review mandatory, and never use floating model aliases in production.
68. P95 latency on your self-hosted LLM service jumps sharply at peak hours and some requests fail with GPU out-of-memory errors. How do you diagnose and fix it?
Answer: This pattern usually means the server is saturated: KV cache is full, requests queue, and memory headroom is exhausted by long prompts or high concurrency. Shed or queue load gracefully first, then fix capacity and configuration.
What I would check:
- Queue depth, KV cache utilisation and batch size during the spike.
- Prompt length distribution: a new client sending very long contexts.
- Server settings: maximum sequence length, maximum concurrent sequences, GPU memory utilisation fraction.
- Autoscaling: did the HPA scale on a meaningful signal, and were GPU nodes available?
Production consideration: Set per-client limits on context length and concurrency at the gateway, scale on queue or cache metrics, and load-test with realistic prompt-length mixes before peak seasons.
69. Employees report that the internal knowledge assistant shows content from a document that HR deleted last week, and one user saw a confidential restructuring policy draft they should not access. What do you do?
Answer: Treat it as a data exposure incident: involve security and the data owner, remove the content from the index immediately (and from caches), and restrict the assistant to verified sources until the sync and permission paths are fixed.
What I would check:
- The ingestion pipeline's delete handling and the last successful sync per source.
- Whether access-control metadata was copied at ingestion and enforced at query time.
- Semantic or response caches that served the content regardless of permissions.
- Traces and logs to identify every user who retrieved the affected document.
Production consideration: Enforce permissions at retrieval time against the source of truth or a frequently synced ACL, monitor index freshness and deletion lag as SLOs, and include permission tests in the RAG evaluation suite.
70. You join a GCC team in Hyderabad where models are deployed by copying files to a VM and nobody can say which version is in production. What is your 90-day plan?
Answer: Start with visibility, then control, then automation. Do not try to install a full platform in week one.
What I would check:
- Weeks 1 to 3: inventory every model and LLM app, its owner, data sources and current serving method; add basic logging of model version per prediction.
- Weeks 4 to 6: introduce experiment tracking and a registry; containerise one high-impact model and deploy it through a pipeline with an evaluation gate.
- Weeks 7 to 9: add monitoring for drift, latency and errors with alerts; define rollback procedure and test it.
- Weeks 10 to 13: template the pattern (IaC module, pipeline template, Helm chart) and migrate the next models; for LLM apps, put a gateway in front of providers.
Production consideration: Measure success by questions the team can now answer quickly (what is running, who approved it, how to roll back), not by the number of tools installed.
Key takeaways
- MLOps interviews in 2026 test both classic ML operations and LLMOps; prepare both, with the same mental model of versioning, gating, rollout and monitoring.
- Lineage is the backbone: code commit, data snapshot, image digest, model version, prompt version and index version should all be traceable from any production output.
- Evaluation is a release gate, not a report. For LLM systems that means golden sets, calibrated judges, safety tests and cost checks.
- GPU serving is about KV cache, batching and queueing. Scale on queue and cache signals, not CPU.
- Supply-chain security now includes models: provenance, signing, safe formats and an internal registry.
- Scenario answers win senior interviews: contain impact first, investigate with traces and data, then fix the process that let it happen.
Interview preparation checklist
- Build one small end-to-end ML pipeline: tracked experiments, a registry, a containerised model server and a drift report.
- Build one LLM application with versioned prompts, a golden set, an evaluation step in GitHub Actions and tracing to Langfuse or a similar tool.
- Deploy a model server to a Kubernetes cluster with probes, an HPA on a custom metric and a canary rollout.
- Write Terraform for a cluster, a node pool and object storage, and practise explaining plan output and drift.
- Prepare two incident stories from your own work using the format: impact, containment, root cause, prevention.
- Be ready to draw, from memory, a diagram of an LLM gateway, a RAG ingestion pipeline and a train-evaluate-register-deploy loop.
- Revise your answers for clarity: direct answer first, then trade-offs, then an example.
- Read the sibling guides on LLM interview questions and AI security interview questions for adjacent rounds.
FAQ
What skills are required for an MLOps or LLMOps engineer role?
Python, Linux, Docker, Kubernetes, CI/CD, cloud fundamentals and infrastructure as code form the base. On top, you need ML lifecycle knowledge (tracking, registries, monitoring, drift) and, for LLMOps, prompt management, evaluation, gateways, tracing and cost control.
Is MLOps different from LLMOps in interviews?
The principles overlap, but LLMOps rounds focus more on evaluation of free-text outputs, prompt and model versioning, RAG index operations, guardrails, token cost and GPU serving, while classic MLOps rounds focus on training pipelines, feature stores and drift.
Can a DevOps engineer move into MLOps?
Yes. CI/CD, containers, Kubernetes, IaC and monitoring transfer directly. The gap is usually ML concepts such as evaluation metrics, drift and data quality, which can be learned with a few hands-on projects.
Do I need to train models myself to work in LLMOps?
Not usually. Most LLMOps work involves operating applications on hosted or self-hosted foundation models. Understanding fine-tuning, evaluation and serving trade-offs matters more than training models from scratch.
Which tools should I learn first for MLOps interviews?
Start with git, Docker, Kubernetes, one CI system such as GitHub Actions, Terraform, MLflow for tracking and registry, and Prometheus and Grafana for monitoring. For LLMOps add an evaluation tool such as Ragas and a tracing tool such as Langfuse or LangSmith.
How should a fresher prepare for MLOps interview questions?
Learn the lifecycle, then build one small project end to end and be able to explain every step. Interviewers value a working, well-explained project more than a long list of tools.
Are cloud-specific MLOps questions common?
Yes, especially in GCC and services-company interviews. Expect questions on the managed ML platform of the cloud the team uses, plus how you would run the same workflow on Kubernetes.
How long does it take to prepare for an MLOps interview?
It depends on your background. An experienced DevOps engineer can often prepare in a few focused weeks with hands-on practice; a fresher typically needs a structured learning path covering both software and ML fundamentals first.
Is MLOps a good career choice in India?
MLOps and LLMOps skills are increasingly requested as companies and GCCs move AI from pilots to production. It suits engineers who enjoy reliability, automation and platform work more than pure model research.
Ready to practise these skills on real systems rather than flashcards? The APEX program from Cloudsoft covers ML, GenAI, cloud deployment and security in one track. If your goal is to take AI systems into customer environments end to end, look at the AI Forward Deployed Engineer FDE PRO program, where you work with Langfuse, OpenTelemetry, Kubernetes (EKS), Terraform, GitHub Actions and Argo CD, with placement support until you're placed. If you are newer to pipelines and containers, the DevOps training in Hyderabad is a solid foundation. Classroom in Ameerpet or live online; call +91 96660 19191 for a free demo.
