Kubernetes Master Handbook (2026): Architecture, Internals & YAML — Basics to Expert

Kubernetes is an open-source platform for automating the deployment, scaling, healing and operation of containerised applications across a fleet of machines. The name comes from the Greek κυβερνήτης — helmsman or pilot — and it is commonly abbreviated K8s, where the 8 stands for the eight letters between the K and the s.

1.1 The one-sentence definition, unpacked

That definition is easy to recite and hard to actually understand, so let us take it apart clause by clause. Every phrase in it carries weight.

"Containerised applications"

Kubernetes does not run your code directly. It runs containers — the images your teams build with Docker, Buildah, Kaniko or Jib. Kubernetes assumes the packaging problem is already solved. It concerns itself only with where those containers run, how many of them run, how they find each other, and what happens when they die.

"Across a fleet of machines"

This is the central idea. A single Docker host is a machine you must think about. A Kubernetes cluster is a pool of capacity you stop thinking about. You declare "I need six replicas of this service with 500m CPU each" and Kubernetes decides which of your forty nodes has room. You never write a hostname. When a node dies at 3 a.m., the workload lands somewhere else and nobody is paged.

"Automating deployment, scaling, healing and operation"

These are the four verbs that justify the entire platform:

• Deployment — roll a new image out gradually, watch health, stop if it fails, roll back to the previous version on one command.
• Scaling — add or remove replicas manually, on CPU/memory pressure, on custom metrics, or on a queue depth; and add or remove nodes underneath when the pods will not fit.
• Healing — restart crashed containers, replace pods on dead nodes, remove unhealthy pods from load-balancer rotation, reschedule when a node is drained.
• Operation — service discovery, configuration injection, secret distribution, storage attachment, rolling certificate updates, traffic policy.

1.2 Declarative intent and the reconciliation loop

If you remember one concept from this entire handbook, make it this one. Kubernetes is a declarative, level-triggered control system. Almost every question an interviewer asks about Kubernetes internals is really this idea in disguise.

In an imperative system you issue commands: start this container, stop that one, copy this file. The system does what you said and forgets. If the container dies afterwards, nothing happens, because nobody is watching.

In Kubernetes you do not issue commands. You submit a desired state — a document that says what the world should look like. A controller then runs forever in a loop:

for {
    desired  := readSpecFromAPIServer()   // what the user asked for
    actual   := observeRealWorld()        // what is actually running

    if actual == desired {
        continue                          // nothing to do, sleep and re-check
    }

    diff := computeDifference(desired, actual)
    takeCorrectiveAction(diff)            // create, delete, patch
    writeStatusBackToAPIServer()
}

Three consequences follow from this design, and all three matter in production:

Spec and status: the shape of every object

Because of the loop, every Kubernetes object has the same two-part shape. This is not a convention; it is enforced by the API machinery.

apiVersion: apps/v1          # which API group and version defines this object
kind: Deployment             # which type of object
metadata:                    # identity: name, namespace, labels, annotations
  name: payments-api
  namespace: production
spec:                        # DESIRED state  — written by you
  replicas: 3
status:                      # OBSERVED state — written by the controller, never by you
  readyReplicas: 3
  availableReplicas: 3

You write spec. Kubernetes writes status. If you ever find yourself trying to edit status, you have misunderstood the model. And when you debug a workload, the first useful question is always: what does spec say, what does status say, and why do they differ?

1.3 What Kubernetes is not

Interviewers ask this deliberately, because candidates who cannot name the boundaries usually cannot design within them.

Kubernetes is genuinely complex. That complexity is only worth paying for if it removes a larger complexity somewhere else. This chapter shows exactly what that larger complexity looks like in a real production environment — and, honestly, when it does not exist and you should walk away.

2.1 Life before orchestration — the pain, concretely

Picture a mid-sized banking platform: twenty-two microservices, four environments, thirty virtual machines, three release windows a week. Without an orchestrator, here is what the team actually deals with.

Scenario A — a node dies at 02:40

Without Kubernetes: monitoring pages the on-call engineer. They log in, discover the host is gone, find which six services were pinned to it, locate spare capacity, SSH into replacement hosts, pull the right image tags from a wiki page that is two weeks stale, start containers with the right environment variables, update the load balancer pool by hand, and verify. Mean time to recovery: 45–90 minutes, with a tired human doing error-prone work.

With Kubernetes: the node controller marks the node NotReady after roughly 40 seconds. After the eviction timeout the pods on that node are marked for deletion. Their owning ReplicaSets observe a shortfall, create replacement pods, the scheduler places them on healthy nodes, kubelet pulls images and starts them, readiness probes pass, endpoints are updated, traffic flows. Mean time to recovery: 60–120 seconds, with nobody awake.

Scenario B — Friday evening traffic spike

Without Kubernetes: someone must predict the spike, provision VMs in advance, and pay for them all week. If the prediction is wrong in one direction you burn money; wrong in the other, you drop transactions.

With Kubernetes: the HorizontalPodAutoscaler watches CPU or a custom metric such as queue depth, scales the deployment from 6 replicas to 30, and the Cluster Autoscaler adds nodes when pods go Pending for lack of capacity. When the spike passes, both scale back down. You pay for the peak only during the peak.

Scenario C — releasing version 2.4.1

Without Kubernetes: a runbook with 40 manual steps, a maintenance window, a rollback plan that involves re-deploying old artefacts, and a genuine risk that hosts 1–4 are on the new version while hosts 5–6 are still on the old one because the script failed halfway.

With Kubernetes: kubectl set image or a Git commit. Pods are replaced a few at a time, each new pod must pass its readiness probe before the next old one is removed, and kubectl rollout undo returns you to the previous ReplicaSet in seconds. There is no partial state, because the controller will not proceed past a failing surge.

2.2 The ten problems Kubernetes actually solves

2.3 When you should NOT use Kubernetes

A world-class engineer is defined as much by knowing when not to reach for a tool. Kubernetes is a poor fit when:

• You have one or two services and no operations team. A managed PaaS, ECS Fargate, Azure App Service or Cloud Run will deliver the same outcome at a fraction of the operational cost.
• Your workload is a single stateful monolith with a vertical scaling profile. Kubernetes adds indirection without adding value.
• You cannot staff platform expertise. An unmaintained cluster — outdated version, unpatched CNI, no backup of etcd, no upgrade rehearsal — is more dangerous than the VMs it replaced.
• Your latency budget cannot absorb the network path. Overlay networking, kube-proxy and ingress add hops. Most applications never notice; ultra-low-latency trading systems do.
• Regulatory constraints forbid shared tenancy and you would end up running one cluster per application anyway, which removes the bin-packing benefit.

Kubernetes did not emerge from a whiteboard. It is the third-generation product of roughly fifteen years of running containers at Google scale, and several design decisions that look arbitrary make perfect sense once you know where they came from.

3.1 Borg, Omega and the Google lineage

Borg (circa 2003 onwards)

Google's internal cluster manager, running essentially all of Google's workloads on shared machine pools. Borg introduced ideas that survive verbatim in Kubernetes: the allocation unit that groups co-located processes (which became the Pod), declarative job specifications, labels for grouping, bin-packing schedulers, and the crucial insight that batch and serving workloads should share the same fleet to raise utilisation.

Omega (circa 2013)

A research successor exploring a different architecture: instead of a monolithic scheduler, Omega used a shared, consistent, transactional store of cluster state that many independent components read and wrote optimistically. That is exactly the etcd-plus-API-server-plus-many-controllers architecture Kubernetes uses today. Omega is the direct ancestor of the control plane's shape.

Kubernetes (2014)

Announced by Google in June 2014, largely authored by engineers with deep Borg experience — Joe Beda, Brendan Burns, Craig McLuckie, Tim Hockin, Brian Grant and others. The deliberate decision was to build an open, API-driven system rather than open-sourcing Borg itself, and to make everything extensible so the community could grow it.

3.2 Timeline: 2013 to today

3.3 The CNCF, release cadence and API deprecation

Kubernetes ships approximately three minor releases per year. Each minor version receives patch support for roughly fourteen months. Managed providers typically support three to four minor versions concurrently, which in practice means you must upgrade at least once a year, and realistically twice.

API versioning — and why it bites people

Kubernetes APIs move through maturity stages, and the stage is visible in the apiVersion string:

Swarm lost the orchestration war, but it is not a bad system — it is a deliberately smaller one. Understanding precisely where the two diverge sharpens your understanding of what Kubernetes chose to make complicated, and why.

4.1 Architectural comparison

Docker Swarm architecture

• Manager nodes maintain cluster state using an embedded Raft log — no external datastore to operate.
• Worker nodes run the Docker Engine and execute tasks assigned by managers.
• The Service is the top-level abstraction; it produces Tasks, and each task is exactly one container.
• Networking uses a built-in overlay driver, with a routing mesh that makes any node accept traffic for any published port.
• The API is the Docker API you already know — docker service create, docker stack deploy.

Kubernetes architecture

• A control plane of separable components — API server, etcd, scheduler, controller manager — each independently scalable and replaceable.
• etcd as an external, consistent datastore, itself Raft-based but operated separately.
• The Pod as the scheduling unit, which may hold multiple co-located containers sharing a network namespace and volumes.
• Pluggable everything: runtime via CRI, networking via CNI, storage via CSI, devices via device plugins, admission via webhooks.
• A rich, versioned, extensible REST API that users can add their own types to via CRDs.

4.2 Feature-by-feature matrix

4.3 The same app in both — side by side

Three replicas of a web service, published on port 80, with a rolling update policy.

docker swarm init

docker service create \
  --name web \
  --replicas 3 \
  --publish published=80,target=8080 \
  --update-parallelism 1 \
  --update-delay 10s \
  --limit-cpu 0.5 --limit-memory 512M \
  nginx:1.27

docker service scale web=5
docker service update --image nginx:1.28 web
docker service rollback web

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - containerPort: 8080
          resources:
            requests: { cpu: "250m", memory: "256Mi" }
            limits:   { cpu: "500m", memory: "512Mi" }
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080

The Swarm version is shorter. That is a real advantage and you should say so honestly. But notice what the Kubernetes version gained: a readiness probe that gates the rollout, a distinction between resource requests (used for scheduling) and limits (used for enforcement), a surge policy that guarantees no capacity dips below 100% during the update, and a manifest that lives in Git and can be reconciled continuously by Argo CD.

4.4 Verdict and interview answer

A Kubernetes cluster is two planes: a control plane that holds and reconciles desired state, and a data plane of worker nodes that actually runs containers. Every component belongs to one of these, and every component talks to exactly one other component — the API server.

5.1 Control plane vs data plane

5.2 The shared-state, level-triggered design

Three architectural rules govern everything. State them to a learner early and they will predict component behaviour correctly for the rest of the course.

Rule 1 — Only the API server touches etcd

No scheduler, no controller, no kubelet ever opens a connection to etcd. This single constraint gives Kubernetes its authentication, authorisation, validation, admission control, auditing and versioning — all enforced in one place. It also means etcd can be replaced (some distributions use SQLite via k3s' kine) without any other component knowing.

Rule 2 — Components never talk to each other

The scheduler does not call the kubelet. The controller manager does not call the scheduler. Every interaction is mediated: a component writes an object, and another component watches for that change. This is a message bus implemented as a database with watch semantics. The benefit is enormous — any component can crash and restart without a handshake protocol, because state is durable and observation is idempotent.

Rule 3 — Watch, do not poll

Components open long-lived HTTP watch streams against the API server and receive incremental updates. Client-go maintains a local informer cache so controllers read from memory rather than hammering the API. When you write a controller or operator, using informers rather than list loops is the difference between a cluster that scales to 5,000 nodes and one that melts at 200.

5.3 End-to-end request flow of kubectl apply

This is the single most valuable walkthrough in Kubernetes education. Interviewers ask it constantly because a complete answer proves you understand every component.

 1. kubectl  reads kubeconfig, resolves the API server endpoint and credentials.
             Converts YAML to JSON, performs client-side discovery of the resource,
             then sends POST/PATCH to  /apis/apps/v1/namespaces/prod/deployments

 2. AUTHENTICATION   - API server verifies WHO you are:
                       client certificate, bearer token, OIDC, or webhook.

 3. AUTHORIZATION    - RBAC evaluates: may this subject 'create' 'deployments'
                       in namespace 'prod'?

 4. ADMISSION (mutating)  - MutatingAdmissionWebhooks and built-in plugins may
                       CHANGE the object: inject sidecars, add default
                       ServiceAccount, apply defaulting, set imagePullSecrets.

 5. VALIDATION       - schema validation against the OpenAPI definition.

 6. ADMISSION (validating) - ValidatingAdmissionWebhooks / Policy (Kyverno, OPA,
                       Pod Security Admission) may REJECT the object.

 7. PERSIST          - object written to etcd. resourceVersion assigned.
                       API server returns 201 to kubectl.  *** You are done. ***
                       Everything below happens asynchronously.

 8. DEPLOYMENT CONTROLLER  sees the new Deployment via its watch.
                       Creates a ReplicaSet with a pod-template-hash.

 9. REPLICASET CONTROLLER  sees a ReplicaSet with spec.replicas=3 and 0 existing
                       pods. Creates 3 Pod objects - with NO nodeName set.

10. SCHEDULER       sees 3 Pods with spec.nodeName == "".  For each pod:
                       FILTER  (predicates): which nodes are even feasible?
                       SCORE   (priorities): which feasible node is best?
                       BIND:   writes  spec.nodeName = node-07  via the API server.

11. KUBELET on node-07 watches for pods bound to itself. Sees the new pod.
                       - Calls CNI ADD  -> pod gets an IP and a veth pair
                       - Calls CSI      -> mounts any volumes
                       - Calls CRI      -> pulls images, creates the pause (sandbox)
                                          container, then app containers
                       - Runs startup/liveness/readiness probes
                       - PATCHes pod status back to the API server

12. ENDPOINT / ENDPOINTSLICE CONTROLLER sees the pod become Ready and adds its
                       IP to the EndpointSlice of any matching Service.

13. KUBE-PROXY on every node watches EndpointSlices and programs iptables/IPVS
                       rules so the Service ClusterIP now load-balances to the pod.

14. COREDNS already resolves the Service name; traffic now reaches the new pod.

6.1 kube-apiserver

The API server is the front door and the only door. It is a stateless REST server that exposes the entire Kubernetes object model over HTTPS, validates everything, and is the sole writer to etcd. Because it is stateless, you scale it horizontally behind a load balancer.

Responsibilities, in order of execution

1. TLS termination and authentication — X.509 client certificates, bearer tokens (ServiceAccount JWTs), OIDC tokens, or an authentication webhook. The result is a username, UID and set of groups.
2. Authorization — normally RBAC, optionally Node authorizer, ABAC, or a webhook. Evaluates verb + resource + namespace against Roles and ClusterRoles bound to the subject.
3. Mutating admission — built-in plugins plus MutatingAdmissionWebhook. This is where service meshes inject sidecars and where defaults get applied.
4. Schema validation — against the OpenAPI v3 schema for the resource, including CRD schemas.
5. Validating admission — ValidatingAdmissionWebhook, ValidatingAdmissionPolicy (CEL-based, built in), ResourceQuota enforcement, Pod Security Admission.
6. Persistence — write to etcd, assign a resourceVersion, emit the change to all watchers.
7. Watch fan-out — maintain long-lived streams to every controller, kubelet and kubectl watch in the cluster.

Key flags you will actually touch

6.2 etcd

etcd is a distributed, strongly consistent key-value store using the Raft consensus algorithm. It holds the complete state of the cluster: every object, every status field, every secret. It is the only stateful component and therefore the only one that can lose your cluster permanently.

Raft, quorum and why you always use an odd number

Raft elects a leader; all writes go through the leader and are committed once a majority (quorum) of members have persisted them. Quorum for N members is (N/2)+1.

Backup and restore

# --- BACKUP (run on a schedule; ship the file off-cluster) ---
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F-%H%M).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# --- RESTORE (stop API server + etcd on ALL control plane nodes first) ---
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-2026-01-15.db \
  --data-dir=/var/lib/etcd-restored
# Then point the etcd static pod manifest at the new data dir and restart.

Operational rules for etcd

• Put it on fast disks. etcd is fsync-bound. SSD/NVMe is not optional.
• Watch the database size. Default quota is 2 GiB (often raised to 8 GiB). Exceeding it puts etcd into a maintenance alarm and read-only mode.
• Compact and defragment. Kubernetes auto-compacts revisions, but defragmentation must be scheduled — one member at a time, never simultaneously.
• Keep latency between members low. Stretching etcd across regions is a classic mistake; across AZs in one region is fine.
• Encrypt at rest. Secrets are plaintext-adjacent in etcd without EncryptionConfiguration.

6.3 kube-scheduler

The scheduler watches for pods with an empty spec.nodeName and decides which node each should run on. It does not start anything — its entire output is a single write: the binding.

The two-phase algorithm

Phase 1 — Filtering (predicates): eliminate nodes that cannot run the pod — insufficient resources, hostPort conflict, nodeSelector/affinity mismatch, untolerated taints, zonal volume mismatch, anti-affinity violation, or node conditions (memory/disk/PID pressure, not Ready).

Phase 2 — Scoring (priorities): rank surviving nodes 0–100 across plugins (NodeResourcesFit, ImageLocality, InterPodAffinity, PodTopologySpread, TaintToleration, NodeAffinity), apply weights, pick the highest total.

spec:
  priorityClassName: high-priority          # enables preemption of lower priority
  nodeSelector:
    disktype: ssd
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:   # HARD
        nodeSelectorTerms:
          - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values: ["ap-south-1a", "ap-south-1b"]
    podAntiAffinity:                        # never two replicas on one node
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels: { app: payments-api }
          topologyKey: kubernetes.io/hostname
  tolerations:
    - key: "workload"
      operator: "Equal"
      value: "banking"
      effect: "NoSchedule"
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels: { app: payments-api }

6.4 kube-controller-manager

A single binary that runs roughly thirty independent control loops in separate goroutines. Each one watches a resource type and drives reality towards spec. Understanding that it is many controllers, not one, is what makes Kubernetes behaviour predictable.

Leader election

In an HA control plane you run three controller-manager instances, but only one is active. They compete for a Lease object; the holder renews it, and if it fails to renew, another takes over. Without this, three controllers would each create three replicas for a three-replica ReplicaSet. The same mechanism protects the scheduler.

6.5 cloud-controller-manager

Kubernetes was originally full of cloud-specific code. That was extracted into the cloud-controller-manager so the core could stay vendor-neutral. On EKS and AKS this component is managed for you, but you must understand what it does because it explains several everyday behaviours.

7.1 kubelet

The kubelet is the node agent — the only component that actually causes containers to exist. It watches the API server for pods bound to its own node, then drives the runtime, network and storage plugins until the observed state matches the pod spec.

The pause container — a question that catches people out

Every pod has a hidden container, usually called pause or the "sandbox" or "infra" container. It does almost nothing: it starts, reaps zombie processes, and sleeps. Its purpose is to own the pod's Linux namespaces. Application containers join the pause container's network and IPC namespaces, which is precisely why containers in a pod share one IP and can reach each other on localhost. If an app container crashes and restarts, the pause container survives, so the pod keeps the same IP.

Node-pressure eviction — the ranking that decides who dies

When a node runs low on memory or disk, kubelet evicts pods before the kernel OOM killer acts. The order is by QoS class, then by how far the pod exceeds its requests:

Static pods

kubelet also reads manifests directly from /etc/kubernetes/manifests/ and runs them without any control plane involvement. This is the bootstrap trick: on a kubeadm cluster, the API server, etcd, scheduler and controller-manager are themselves static pods managed by kubelet.

7.2 Container runtime and the CRI

kubelet does not know how to start a container. It speaks the Container Runtime Interface — a gRPC API — to whatever runtime is installed. This is why "Kubernetes removed Docker" was true and yet changed nothing for developers.

7.3 kube-proxy and Service implementation

kube-proxy runs on every node and implements the Service abstraction. It does not proxy traffic itself in normal modes — it programs the kernel to do the load balancing, then gets out of the way.

A cluster with only the core components can schedule pods — and almost nothing else. No DNS, no metrics, no ingress, no dynamic storage. Add-ons fill that gap. They are ordinary Kubernetes workloads that happen to provide platform capability — Kubernetes extends itself using its own primitives.

8.1 CoreDNS

CoreDNS is the cluster DNS server, running as a Deployment in kube-system fronted by a Service with a well-known ClusterIP (typically 10.96.0.10). Every kubelet writes that IP into every pod's /etc/resolv.conf.

8.2 metrics-server and the metrics pipeline

8.3 Ingress controllers

An Ingress object is only a set of L7 routing rules stored in the API. It does nothing on its own. An Ingress controller (ingress-nginx, AWS Load Balancer Controller, AGIC, Traefik, or a Gateway API implementation) is the workload that watches Ingress objects and configures a real proxy. Without a controller installed, creating an Ingress produces no effect whatsoever.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: banking-ingress
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts: ["api.bank.example.com"]
      secretName: bank-tls          # cert-manager creates and renews this
  rules:
    - host: api.bank.example.com
      http:
        paths:
          - path: /payments
            pathType: Prefix
            backend:
              service:
                name: payments
                port:
                  number: 80

8.4 CSI storage drivers

The Container Storage Interface moved vendor code out of the Kubernetes tree into independently released drivers. A CSI driver has a controller Deployment (provision, delete, attach, snapshot) and a node DaemonSet (stage, mount, unmount).

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-encrypted
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer   # bind only after scheduling

8.5 Autoscalers, dashboard and other add-ons

Kubernetes deliberately ships no networking implementation. It defines a model, and the Container Network Interface lets any plugin satisfy it. This is why a fresh cluster has every pod stuck in ContainerCreating until you install a CNI — and why "which CNI?" is one of the highest-consequence decisions in cluster design.

9.1 The four rules of the Kubernetes network model

1. Every pod gets its own unique IP address across the whole cluster.
2. Pods can communicate with all other pods on any node without NAT. The IP a pod sees itself as is the IP other pods see it as.
3. Agents on a node (kubelet, system daemons) can reach all pods on that node.
4. Pods in the host network namespace can communicate with all pods on all nodes without NAT.

9.2 What the CNI specification actually is

• Plugin binaries live in /opt/cni/bin/.
• Plugin configuration lives in /etc/cni/net.d/ as a JSON conflist, evaluated in lexical order — the first file wins.
• The runtime invokes a binary with a verb via environment variables and JSON on stdin: ADD, DEL, CHECK, VERSION.
• The plugin returns JSON describing the interfaces, IPs, routes and DNS it configured.

1. kubelet asks the runtime to create the pod sandbox (pause container)
   -> a new, empty network namespace exists
2. Runtime invokes the CNI plugin with CNI_COMMAND=ADD and the namespace path
3. Plugin creates a veth pair: eth0 inside the pod, calixxxxx on the host
4. IPAM allocates an address from the node pod CIDR (or the VPC)
5. Plugin assigns the IP to eth0, sets the default route via the host end
6. Plugin programs host routing / eBPF maps so other nodes can reach this IP
7. Plugin returns JSON; kubelet records podIP in pod.status
8. On pod deletion, CNI_COMMAND=DEL tears it down and releases the IP

9.3 Plugin comparison

9.4 NetworkPolicy deep dive

By default, every pod can reach every other pod in the cluster. NetworkPolicy stops that. Three rules govern its semantics:

1. Policies are additive and allow-only. There is no deny rule. The union of all matching policies is what is permitted.
2. A pod is "isolated" for a direction the moment any policy selects it for that direction. Before that, everything is allowed; after, only what is explicitly allowed.
3. The CNI must implement it. Applying a NetworkPolicy with Flannel or plain AWS VPC CNI silently does nothing — one of the most dangerous false-security situations in Kubernetes.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}              # ALL pods in this namespace
  policyTypes: [Ingress, Egress]
# no rules listed = nothing allowed
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-api-policy
  namespace: production
spec:
  podSelector:
    matchLabels: { app: payments-api }
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - podSelector: { matchLabels: { app: web-frontend } }
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector: { matchLabels: { app: postgres } }
      ports:
        - protocol: TCP
          port: 5432

A Namespace is a virtual cluster inside a physical cluster: a scope for names, a target for RBAC and quotas, and a boundary for policy. It is the primary multi-tenancy primitive — and it is important to know exactly how weak that boundary is by default.

10.1 YAML anatomy and lifecycle

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    kubernetes.io/metadata.name: production   # set automatically; you select on it
    environment: production
    team: payments
    # Pod Security Admission — enforce the restricted profile
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted

Namespaced vs cluster-scoped

Terminating namespaces — the stuck-forever problem

Deleting a namespace triggers the namespace controller to delete every object inside it. If any object has a finalizer that never completes — commonly a CRD whose operator has already been uninstalled — the namespace sits in Terminating indefinitely.

10.2 ResourceQuota and LimitRange

apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: production
spec:
  hard:
    requests.cpu: "100"
    requests.memory: 200Gi
    limits.cpu: "200"
    limits.memory: 400Gi
    pods: "300"
    services.loadbalancers: "5"   # cost control — each one is a real cloud LB
    services.nodeports: "0"       # forbid NodePort entirely

apiVersion: v1
kind: LimitRange
metadata:
  name: production-limits
  namespace: production
spec:
  limits:
    - type: Container
      default:          { cpu: "500m", memory: "512Mi" }   # applied as limits
      defaultRequest:   { cpu: "100m", memory: "128Mi" }   # applied as requests
      max:              { cpu: "4",    memory: "8Gi" }
      min:              { cpu: "50m",  memory: "64Mi" }

11.1 Why a Pod and not a container

A Pod is one or more containers that are always co-scheduled on the same node, share a network namespace and IP, share IPC, and can share volumes. It is the smallest thing Kubernetes schedules — you cannot schedule a container.

Why introduce this extra layer? Because some processes are genuinely a single unit of deployment even though they are separate binaries — a web server and a log shipper that tails its files; an application and a proxy that terminates mTLS on its behalf. These must live on the same machine, start together, die together, and share a filesystem or loopback interface.

11.2 A production-grade Pod spec (abridged)

apiVersion: v1
kind: Pod
metadata:
  name: payments-api
  namespace: production
  labels: { app: payments-api, tier: backend, version: "2.4.1" }
spec:
  serviceAccountName: payments-sa
  automountServiceAccountToken: false    # only mount if the app calls the API
  terminationGracePeriodSeconds: 60
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    fsGroup: 10001
    seccompProfile: { type: RuntimeDefault }
  initContainers:
    - name: wait-for-db                   # run to completion, before app starts
      image: busybox:1.36
      command: ["sh","-c","until nc -z postgres 5432; do sleep 2; done"]
  containers:
    - name: app
      image: registry.example.com/payments-api:2.4.1   # NEVER use :latest
      ports:
        - { name: http, containerPort: 8080 }
      env:
        - name: POD_NAME                  # downward API
          valueFrom: { fieldRef: { fieldPath: metadata.name } }
        - name: DB_PASSWORD
          valueFrom: { secretKeyRef: { name: payments-db, key: password } }
      resources:
        requests: { cpu: "250m", memory: "512Mi" }   # used for SCHEDULING
        limits:   { cpu: "1",    memory: "1Gi" }      # used for ENFORCEMENT
      startupProbe:                       # protects slow starters from liveness
        httpGet: { path: /healthz/startup, port: http }
        failureThreshold: 30
        periodSeconds: 5
      livenessProbe:                      # failing => container RESTARTED
        httpGet: { path: /healthz/live, port: http }
        periodSeconds: 10
      readinessProbe:                     # failing => removed from endpoints
        httpGet: { path: /healthz/ready, port: http }
        periodSeconds: 5
      lifecycle:
        preStop:
          exec: { command: ["sh","-c","sleep 10"] }
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities: { drop: ["ALL"] }

11.3 Lifecycle, phases and conditions

Phases (Pending, Running, Succeeded, Failed, Unknown) are coarse. Conditions carry the useful detail: PodScheduled, Initialized, ContainersReady, Ready. A pod can be Running and not Ready — exactly the state where readiness probes are keeping it out of the load balancer.

CrashLoopBackOff is not an error — it is a backoff timer. The container is exiting, kubelet is restarting it, and the delay doubles: 10s, 20s, 40s, 80s, 160s, capped at 5 minutes. The actual failure reason is in the previous container's logs.

kubectl logs payments-api-7d9f --previous       # logs of the CRASHED instance
kubectl describe pod payments-api-7d9f          # events + exit codes + reasons

# Exit code meanings
#   0 clean   1 app error   137 SIGKILL (usually OOMKilled)
# 139 SIGSEGV   143 SIGTERM   127 command not found

11.4 Probes, init containers, sidecars

11.5 Resources, QoS classes and eviction

11.6 Scheduling controls and security context

• Taints are on nodes and repel pods: "nothing runs here unless it explicitly tolerates me."
• Tolerations are on pods and merely permit scheduling onto a tainted node — they do not attract.
• Node affinity is on pods and attracts: "put me on nodes that look like this."
• To dedicate nodes to a workload you need both: taint the nodes and add node affinity/nodeSelector to the workload.

kubectl taint nodes node-07 workload=banking:NoSchedule
#   NoSchedule        - no new pods without a matching toleration
#   PreferNoSchedule  - soft; scheduler avoids if it can
#   NoExecute         - as NoSchedule, AND evicts running pods that do not tolerate
kubectl taint nodes node-07 workload=banking:NoSchedule-   # trailing dash removes

12.1 The three-object chain

You create a Deployment. It creates a ReplicaSet. That creates Pods. Three objects, three controllers, one purpose — and understanding the division of labour explains every rollout behaviour you will ever debug.

Deployment  payments-api         owns rollout strategy, revision history, rollback
  |
  |-- ReplicaSet  ...-6d4b8f9c7   (revision 3 - CURRENT, replicas: 3)
  |       |-- Pod ...-x7k2p
  |       |-- Pod ...-m9q4r
  |       |-- Pod ...-t2v8n
  |-- ReplicaSet  ...-5f9c7d8b6   (revision 2 - OLD, replicas: 0)
  |-- ReplicaSet  ...-4c8b6a5d9   (revision 1 - OLD, replicas: 0)

Old ReplicaSets are kept, scaled to zero, so rollback is instant.
revisionHistoryLimit controls how many are retained (default 10).

The pod-template-hash label makes this work. The Deployment controller hashes the pod template and adds it to the ReplicaSet's selector and pods' labels. Change the template, get a new hash, get a new ReplicaSet. Change only replicas and the hash is unchanged — which is why scaling does not create a new revision.

12.2 Production Deployment YAML (abridged)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: production
  annotations:
    kubernetes.io/change-cause: "Upgrade to 2.4.1 - fixes settlement retry bug"
spec:
  replicas: 6
  selector:
    matchLabels: { app: payments-api }   # IMMUTABLE after creation
  revisionHistoryLimit: 5
  progressDeadlineSeconds: 600           # mark Failed if no progress in 10 min
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 0                   # never dip below replicas ready - safest
  template:
    metadata:
      labels: { app: payments-api, version: "2.4.1" }
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector: { matchLabels: { app: payments-api } }
      containers:
        - name: app
          image: registry.example.com/payments-api:2.4.1
          resources:
            requests: { cpu: "250m", memory: "512Mi" }
            limits:   { cpu: "1",    memory: "512Mi" }
---
apiVersion: policy/v1
kind: PodDisruptionBudget          # protects you during VOLUNTARY disruption
metadata:
  name: payments-api-pdb
  namespace: production
spec:
  minAvailable: 4
  selector:
    matchLabels: { app: payments-api }

12.3 Rolling update mathematics

With replicas: 10, maxSurge: 2, maxUnavailable: 2: maximum total pods at any moment is replicas + maxSurge = 12; minimum available is replicas - maxUnavailable = 8. The controller never proceeds past a batch whose new pods are not Ready — a broken image or failing readiness probe stalls the rollout and leaves the old pods serving traffic, which is exactly the desired behaviour.

12.4 Rollout, rollback and deployment strategies

kubectl set image deployment/payments-api app=payments-api:2.4.2 -n production
kubectl rollout status deployment/payments-api -n production --timeout=10m   # gate CI on this
kubectl rollout history deployment/payments-api -n production
kubectl rollout undo deployment/payments-api -n production                    # previous
kubectl rollout undo deployment/payments-api --to-revision=3 -n production
kubectl rollout restart deployment/payments-api -n production                 # re-roll same image

Beyond native rolling updates: Blue-Green (two full Deployments, flip the Service selector — instant switch and rollback, needs double capacity); Canary (small traffic percentage to the new version, promoted on metrics via Argo Rollouts or Flagger); and Shadow (duplicate real traffic to the new version, discard responses).

13.1 Use cases and scheduling behaviour

A DaemonSet guarantees that a copy of a pod runs on every node matching its selector. When a node joins the cluster, a pod appears on it automatically. When a node leaves, the pod is garbage collected. There is no replicas field — the node count is the replica count.

Since v1.12 DaemonSet pods are scheduled by the default scheduler using node affinity the controller injects, so they respect taints, priority, preemption and resource availability — and can be left Pending if a node is full.

13.2 Production DaemonSet YAML (abridged)

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels: { app: fluent-bit }
  # NOTE: there is NO spec.replicas — node count determines it
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1                  # how many nodes updated at once
  template:
    metadata:
      labels: { app: fluent-bit }
    spec:
      tolerations:
        - operator: Exists               # tolerate EVERYTHING (run on every node)
      priorityClassName: system-node-critical
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:3.0.7
          resources:
            requests: { cpu: "50m",  memory: "96Mi" }   # SMALL - charged on every node
            limits:   { cpu: "300m", memory: "256Mi" }
          volumeMounts:
            - { name: varlog, mountPath: /var/log, readOnly: true }
      volumes:
        - name: varlog                   # hostPath is the defining characteristic:
          hostPath:                      # DaemonSets exist to read/write the NODE
            path: /var/log
            type: Directory

13.3 Tolerations, host access and update strategy

tolerations: [{operator: Exists}] means "tolerate every taint, including ones invented later." For a logging or security agent that must cover every node, that is correct. For anything else it is dangerous — your pod lands on cordoned nodes, reserved GPU nodes, and nodes under disk pressure. Prefer enumerating specific taints.

kubectl -n logging get pods -l app=fluent-bit -o wide
kubectl get nodes -o name | sed 's|node/||' | sort > /tmp/all-nodes
kubectl -n logging get pods -l app=fluent-bit \
  -o jsonpath='{.items[*].spec.nodeName}' | tr ' ' '\n' | sort > /tmp/covered
comm -23 /tmp/all-nodes /tmp/covered

14.1 Stable identity, ordering and storage

A Deployment treats its pods as interchangeable cattle: random names, random start order, shared storage or none. A StatefulSet treats them as individuals with permanent identities. That is exactly what clustered databases, queues and consensus systems require.

14.2 Production StatefulSet YAML (abridged)

apiVersion: v1
kind: Service
metadata: { name: postgres, namespace: production }
spec:
  clusterIP: None                      # HEADLESS: DNS returns individual pod IPs
  selector: { app: postgres }
  ports: [{ name: postgres, port: 5432 }]
---
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: postgres, namespace: production }
spec:
  serviceName: postgres                # MUST match the headless Service name
  replicas: 3
  selector:
    matchLabels: { app: postgres }
  podManagementPolicy: OrderedReady    # OrderedReady | Parallel
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0                     # only ordinals >= partition are updated
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain                # keep data if the STS is deleted
    whenScaled: Retain
  template:
    metadata:
      labels: { app: postgres }
    spec:
      terminationGracePeriodSeconds: 120   # give the DB time to checkpoint
      affinity:
        podAntiAffinity:               # never two DB replicas on one node
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector: { matchLabels: { app: postgres } }
              topologyKey: kubernetes.io/hostname
      containers:
        - name: postgres
          image: postgres:16.4
          resources:
            requests: { cpu: "1", memory: "4Gi" }
            limits:   { cpu: "2", memory: "4Gi" }   # Guaranteed QoS
  volumeClaimTemplates:
    - metadata: { name: data }         # produces data-postgres-0, -1, -2
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3-encrypted
        resources: { requests: { storage: 500Gi } }

14.3 Headless Services and DNS records

# Headless Service name -> A records for ALL ready pod IPs
postgres.production.svc.cluster.local             -> 10.244.1.7, 10.244.2.9, 10.244.3.4
# Per-pod stable record - THIS is the point of a StatefulSet
postgres-0.postgres.production.svc.cluster.local  -> 10.244.1.7
postgres-1.postgres.production.svc.cluster.local  -> 10.244.2.9

Because names are deterministic, a clustered application configures itself from its own ordinal — a Kafka broker's ID is the ordinal, a Postgres replica knows the primary is always -0. No service registry, no discovery protocol.

14.4 Scaling, updates and the operational traps

16.1 Pod will not start

16.2 Service or DNS problems

# 1. Does the Service have endpoints at all?
kubectl get endpointslices -l kubernetes.io/service-name=payments -n production
#   EMPTY => selector matches no READY pod (label mismatch OR readiness failing)

# 2. Compare selector to actual pod labels
kubectl get svc payments -n production -o jsonpath='{.spec.selector}'
kubectl get pods -n production --show-labels | grep payments

# 3. Reach the pod directly, bypassing the Service?
kubectl -n production port-forward pod/payments-abc123 8080:8080

# 4. Reach the ClusterIP from another pod?
kubectl -n production run tester --rm -it --image=nicolaka/netshoot -- \
  curl -v http://payments:80/healthz

# 5. Is DNS working?
kubectl -n production run tester --rm -it --image=nicolaka/netshoot -- \
  nslookup payments.production.svc.cluster.local

# 6. Is a NetworkPolicy blocking it?
kubectl get networkpolicy -n production

16.3 Node and cluster level

kubectl get nodes -o wide
kubectl describe node node-07          # conditions, allocatable, taints, pods
kubectl top nodes
kubectl get pods -A --field-selector spec.nodeName=node-07 -o wide

# Safely take a node out of service
kubectl cordon node-07
kubectl drain node-07 --ignore-daemonsets --delete-emptydir-data --grace-period=60
kubectl uncordon node-07

# Control plane health
kubectl get --raw='/readyz?verbose'
kubectl get events -A --sort-by='.lastTimestamp' | tail -50

16.4 The universal debugging toolkit

# Ephemeral debug container on a running pod (image has no shell? fine)
kubectl debug -it payments-abc123 --image=nicolaka/netshoot --target=app
# Debug pod on a specific node, host filesystem access
kubectl debug node/node-07 -it --image=busybox
# What can this ServiceAccount actually do?
kubectl auth can-i --list --as=system:serviceaccount:production:payments-sa
# Diff before applying — essential in GitOps reviews
kubectl diff -f deployment.yaml

17.1 Discovery and self-documentation

kubectl api-resources                          # every kind, its group, short name
kubectl api-versions                           # every group/version served
kubectl explain deployment.spec.strategy       # the schema, from the live server
kubectl explain pod.spec.containers.resources --recursive
kubectl create deployment web --image=nginx --dry-run=client -o yaml > deploy.yaml

17.2 Imperative generators (scaffold fast, then commit the YAML)

kubectl create ns production
kubectl create deployment web --image=nginx:1.27 --replicas=3 --dry-run=client -o yaml
kubectl expose deployment web --port=80 --target-port=8080 --dry-run=client -o yaml
kubectl create configmap app-config --from-file=./config/ --dry-run=client -o yaml
kubectl create secret generic db-creds --from-literal=password=s3cr3t --dry-run=client -o yaml
kubectl create job migrate --image=myapp:1.0 -- /app/migrate
kubectl create cronjob backup --image=backup:1.0 --schedule="0 2 * * *" -- /backup.sh
kubectl create rolebinding dev-binding --role=dev --serviceaccount=production:payments-sa

17.3 Everyday operations

kubectl get pods -A -o wide
kubectl get pods --field-selector status.phase=Failed -A
kubectl get pods --sort-by=.status.containerStatuses[0].restartCount -A
kubectl logs -f deployment/payments-api --all-containers --max-log-requests=10
kubectl logs -l app=payments-api --tail=100 --since=15m --prefix
kubectl exec -it payments-abc123 -c app -- sh
kubectl port-forward svc/payments 8080:80 -n production
kubectl apply -k ./overlays/production          # Kustomize
kubectl patch deployment payments-api -p '{"spec":{"replicas":8}}'

17.4 The manifest skeletons

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: payments-api, namespace: production }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: payments-api }
  minReplicas: 4
  maxReplicas: 40
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }
  behavior:
    scaleDown: { stabilizationWindowSeconds: 300 }   # avoid flapping
---
apiVersion: batch/v1
kind: CronJob
metadata: { name: nightly-backup, namespace: production }
spec:
  schedule: "0 2 * * *"
  timeZone: "Asia/Kolkata"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      ttlSecondsAfterFinished: 86400
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: backup-tool:2.1
              command: ["/backup.sh"]