Ben's GKE Field Guide · 2026 Edition

Inside
GKE

Kubernetes makes more sense when you stop memorizing nouns and start following control loops.

12connected chapters
10+interactive labs
1operational mental model
01
First principles

The machine behind the magic

The one sentence

Kubernetes is a distributed system that stores a desired state, observes reality, and continuously runs control loops to reduce the difference.

When you apply a Deployment, you are not instructing a machine to “start three containers.” You are writing an object to the API. Controllers notice the object and create lower-level objects. The scheduler chooses nodes for unscheduled Pods. Each node's kubelet asks its container runtime to make those Pods real. If one dies, the system does not rewind history—it observes a mismatch and reconciles again.

This distinction explains almost everything: why a Deployment survives Pod deletion, why editing a running container is futile, why status can lag behind spec, and why troubleshooting starts with the API objects and events rather than an SSH session.

Follow one deployment

1 · Clientkubectl sends desired state
2 · API serverauthn, authz, admission, persist
3 · ControllerDeployment → ReplicaSet → Pods
4 · Schedulerfilters and scores nodes
5 · Kubeletstarts containers and reports status

The API server is the hub, not a traffic proxy for your application. Cluster components act as API clients. The control plane stores cluster state and makes decisions; nodes run workload containers and node agents. In GKE, Google manages the control plane. In Autopilot, Google also manages the node infrastructure; in Standard, you manage node pools and their configuration.

The object ladder

ObjectOwns or selectsJob
DeploymentReplicaSetDeclarative rollout for stateless replicas
ReplicaSetPods by labelsKeep a replica count
PodContainersSmallest schedulable unit; shared network and volumes
ServicePods by labelsStable virtual endpoint over changing backends
NodeScheduled PodsCapacity and kubelet boundary

Exam trap: a Service does not own Pods and does not create them. It selects endpoints. A Deployment does not directly restart a container; its controllers maintain objects until the kubelet can realize them.

Read YAML as a contract

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:                         # desired state
  replicas: 3
  selector:
    matchLabels: {app: checkout}
  template:
    metadata:
      labels: {app: checkout} # must match selector
    spec:
      containers:
      - name: app
        image: us-docker.pkg.dev/acme/apps/checkout:v4
        resources:
          requests: {cpu: 250m, memory: 256Mi}
        readinessProbe:
          httpGet: {path: /ready, port: 8080}
status:                       # observed state; written by controllers
  availableReplicas: 3

metadata gives identity, spec states intent, and status reports observation. The generation and observed-generation pattern helps you detect whether a controller has processed the latest spec. Labels create relationships; owner references encode lifecycle.

Lab 01 · Reconciliation simulator

Break reality. Watch intent win.

02
Workload primitives

Choose the controller that matches time

Identity and duration decide the primitive

A Deployment says replicas are interchangeable and should run continuously. A StatefulSet gives each replica stable identity and ordered lifecycle. A DaemonSet places one Pod on each eligible node. A Job runs work to completion; a CronJob creates Jobs on a schedule.

NeedPrimitiveWhy
Stateless APIDeploymentRolling updates and fungible replicas
Broker quorumStatefulSetStable names, ordered rollout, per-Pod claims
Node log agentDaemonSetNode-local coverage
Database migrationJobCompletion, retry, backoff
Nightly exportCronJobScheduled Job creation

Availability is a chain

Readiness gates Service endpoints; liveness restarts a wedged container; startup probes protect slow starters from premature liveness failures. A PodDisruptionBudget limits voluntary disruption, but it does not create capacity, prevent involuntary failure, or override an impossible rollout.

strategy:
  rollingUpdate: {maxUnavailable: 0, maxSurge: 1}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  minAvailable: 2
  selector: {matchLabels: {app: checkout}}
Lab 02 · Primitive selector

Match workload shape to controller

03
Scheduling and capacity

Requests are promises to the scheduler

Scheduling is filter, then score

The scheduler first removes nodes that cannot satisfy hard constraints: resource requests, node selectors, required affinity, taints without tolerations, volume topology, or Pod limits. It then scores feasible nodes using soft preferences and placement heuristics. A Pending Pod is often not “broken”; no node currently satisfies the contract.

CPU requests reserve scheduling capacity and influence CPU shares. CPU limits throttle. Memory limits can trigger an OOM kill; memory is not compressible. Missing or inflated requests corrupt both scheduling and autoscaling signals.

QoSConditionPressure behavior
GuaranteedEvery container: request = limit for CPU and memoryStrongest eviction protection
BurstableAt least one request or limitMiddle
BestEffortNo requests or limitsFirst eviction candidates

Placement tools solve different problems

Use taints to repel general workloads from special nodes, and tolerations only to make those Pods eligible. Use node affinity to attract workloads to node properties. Use Pod anti-affinity or topology-spread constraints to reduce correlated failure. Eligibility is not a guarantee of even spread unless you encode it.

Lab 03 · Scheduling diagnosis

Why is this Pod Pending?

04
GKE operating modes

Control is a cost center

Autopilot is the default question

Google recommends Autopilot for most production workloads. It manages nodes, scaling, many security settings, and infrastructure provisioning from workload manifests. Standard exposes node pools and configuration when you need privileges, unusual agents, specialized topology, or infrastructure control that Autopilot constraints do not permit.

The modern choice is more granular than a permanent binary: Standard clusters can run selected workloads in Autopilot mode through compute classes. That allows controlled node pools and managed workload placement in the same cluster.

DimensionAutopilot workloadStandard workload
Node lifecycleGoogle managedYou configure node pools
FlexibilityGuardrailedBroad
Capacity planningManifest-drivenNode pool + autoscaler design
Best first fitMost production appsPrivileged or specialized needs

Node pools are failure and policy domains

In Standard, group nodes by real differences: machine family, accelerator, architecture, OS, security posture, or lifecycle such as Spot. Every extra pool adds upgrade, scaling, quota, and fragmentation complexity. Regional clusters improve control-plane availability; multi-zone node placement improves workload availability only when replicas and topology rules actually use it.

Lab 04 · Mode decision

Pay for the control you need

05
Networking

A packet crosses four naming systems

Separate reachability from discovery

In VPC-native GKE, Pods receive alias IP addresses from secondary subnet ranges. A Pod IP identifies an ephemeral endpoint. A Service gives a stable virtual IP and DNS name over label-selected endpoints. Ingress and Gateway resources configure HTTP(S) load-balancing behavior; a Service of type LoadBalancer exposes a Layer 4 endpoint.

Client DNSname → load balancer
FrontendIP, TLS, policy
Routehost/path or L4
Servicestable VIP → endpoints
Podactual process and readiness

GKE Dataplane V2 uses eBPF for service routing and always enforces Kubernetes NetworkPolicy. Google recommends VPC-native clusters and Dataplane V2 for new designs. NetworkPolicy is additive: once a Pod is selected for ingress or egress isolation, only explicitly allowed traffic passes in that direction.

Private does not mean disconnected

Private nodes have no external IPs. Inbound application traffic arrives through controlled load balancers. Outbound internet access typically uses Cloud NAT. Private Google Access reaches Google APIs without public node addresses. Control-plane reachability is a separate design choice involving DNS- or IP-based endpoints, authorized networks, and IAM.

IP planning is architecture: Pod and Service ranges must cover growth. A cluster can have spare CPU and still fail to scale because it exhausted Pod IPs.

Lab 05 · Packet autopsy

Find the broken hop

06
Identity and security

Three identities enter a cluster

Do not collapse the identity layers

QuestionSystemExample
Who may call the GKE control plane?Google Cloud IAM + authenticationUser can get cluster credentials
Who may read a Secret in this cluster?Kubernetes RBACServiceAccount bound to Role
Which Google API may this Pod call?Workload Identity Federation for GKE + IAMKSA principal reads one bucket

Google recommends Workload Identity Federation for GKE instead of distributing service-account key files. It is always enabled in Autopilot and should be enabled for Standard clusters and node pools. Give each workload that needs distinct permissions a distinct namespace and Kubernetes ServiceAccount; grant the narrow IAM role to that principal.

Defense in depth follows the artifact

Start at source and build: trusted builders, Artifact Registry, vulnerability scanning, signed attestations, Binary Authorization, admission policy, non-root containers, read-only filesystems where possible, seccomp, least privilege, secrets from managed systems, default-deny network policy, audit logs, and rapid patching.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: invoice-reader
  namespace: billing
---
# IAM principal form:
# principal://iam.googleapis.com/projects/PROJECT_NUMBER/
# locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/
# subject/ns/billing/sa/invoice-reader
Lab 06 · Identity router

Pick the authorization plane

07
Storage

Pods die; claims negotiate continuity

Three objects, three responsibilities

A StorageClass describes a provisioning policy. A PersistentVolumeClaim requests capacity and access semantics. A PersistentVolume represents the durable resource bound to that claim. The CSI driver translates Kubernetes operations into Google Cloud storage operations.

Compute Engine Persistent Disk is durable block storage and typically ReadWriteOnce. Regional Persistent Disk replicates across two zones in a region. Filestore provides shared NFS. Local SSD is fast and ephemeral. Cloud Storage FUSE presents object storage through a filesystem interface but does not turn object storage into POSIX block storage.

Topology can deadlock naïve designs

With WaitForFirstConsumer, provisioning waits until scheduling so the volume is created in compatible topology. A zonal disk constrains where its Pod can run. Stateful availability therefore needs aligned decisions across replicas, zones, storage class, anti-affinity, backups, and recovery objectives.

apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: ledger-data}
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: standard-rwo
  resources:
    requests: {storage: 30Gi}

Snapshots and backups answer different questions. A disk snapshot protects volume data. Backup for GKE can protect Kubernetes resources and volume data. Neither validates your restore procedure—practice it against stated RPO and RTO.

Lab 07 · Storage designer

Match semantics, not product names

08
Delivery and supply chain

Promotion beats rebuilding

One immutable artifact, many environments

Build once, identify the image by digest, scan and attest it, then promote the same digest through environments. Rebuilding “the same” commit can change dependencies and destroys provenance. Artifact Registry stores the image; your delivery system changes desired state; controllers perform the rollout.

Commitreviewed source
Buildtest, SBOM, scan
Registryimmutable digest
Policyattestation/admission
Rolloutobserve SLO, promote

Rollouts trade capacity for safety

RollingUpdate uses maxSurge and maxUnavailable. Blue/green duplicates environments for fast cutover. Canary sends limited traffic to a new version and needs meaningful success metrics. A Deployment rollback restores a prior Pod template; it does not undo database schema changes or external side effects.

Helm templates packages, Kustomize overlays declarative differences, and GitOps agents reconcile repository state. Choose based on ownership and failure modes—not fashion.

Lab 08 · Release strategist

Choose the rollout

09
Reliability and upgrades

Design for movement

GKE will move beneath you

Nodes are repaired, scaled, and upgraded; Pods are evicted and recreated. Release channels choose the balance between feature velocity and demonstrated stability. Regular is the default; Rapid receives features sooner; Stable later; Extended lengthens minor-version support but still requires active lifecycle management.

Maintenance windows and exclusions influence timing, not whether security and compatibility upgrades exist. Surge upgrades add temporary capacity while draining old nodes. Your PDBs, termination grace periods, probes, and spare capacity determine whether the workload cooperates.

A regional cluster is not a regional application

Regional clusters replicate the control plane across zones. Application availability still requires multiple replicas, topology spread, zonally independent dependencies, regional or replicated data, and load balancing. Multi-region availability adds data consistency, routing, failover, quotas, and tested recovery.

Define SLO, RPO, and RTO before selecting mechanisms. Backup without restore testing is inventory, not resilience.

Lab 09 · Availability critic

Find the missing layer

10
Autoscaling and performance

Three loops, three clocks

Do not ask one loop to solve another loop's problem

LoopChangesSignal
HPAReplica countCPU, memory, custom/external metrics
VPAResource requestsObserved usage and recommendations
Cluster autoscalerNode capacityUnschedulable Pods and removable nodes

A traffic spike reaches HPA first; new Pods may become Pending; cluster autoscaling provisions nodes; images pull; startup probes pass; readiness adds endpoints. This takes time. Buffer capacity, efficient images, min replicas, predictive signals, and graceful overload can matter more than a larger maximum.

Utilization is relative to requests

HPA CPU utilization compares usage with requested CPU. If requests are wrong, the scaling signal is wrong. Scaling on average CPU also fails for queue-driven or latency-sensitive systems; use a metric tied to demand, such as queue depth per consumer, while controlling metric delay and stability.

HPA and VPA can conflict when both manipulate CPU/memory assumptions. Coordinate policies. Autopilot can manage more of the infrastructure loop, but workload signals and startup behavior remain yours.

Lab 10 · Scaling chain

Name the loop that acts first

11
Observability and troubleshooting

Start with the symptom's boundary

Evidence has different clocks

Metrics reveal trends and saturation. Logs explain discrete events and application context. Traces connect latency across services. Kubernetes events explain recent control-plane decisions but are best-effort and expire; they are not an audit ledger. Audit logs answer who did what to the API.

GKE collects system and application logs into Cloud Logging when enabled; write application logs to stdout and errors to stderr. Structured single-line JSON becomes structured entries. Managed Service for Prometheus and Cloud Monitoring provide workload metrics; Dataplane V2 observability adds traffic-flow and NetworkPolicy insight.

The practical debug loop

  1. State the user-visible symptom and start time.
  2. Check scope: one Pod, node, zone, version, service, or cluster?
  3. Compare desired and observed object state.
  4. Read events and conditions; then relevant current and previous logs.
  5. Trace dependencies and recent changes.
  6. Form one falsifiable hypothesis and run the cheapest discriminating test.
  7. Mitigate, verify SLO recovery, preserve evidence, then fix.
kubectl get deploy,rs,pod,svc,endpointslice -n shop -o wide
kubectl describe pod POD -n shop
kubectl logs POD -n shop --all-containers --previous
kubectl get events -n shop --sort-by=.lastTimestamp
Lab 11 · First command

Choose the cheapest discriminating evidence

12
Architecture capstone

Design a platform, not a cluster

The reference decision sequence

  1. Define workload SLOs, data sensitivity, RPO/RTO, scale shape, and team ownership.
  2. Choose project, VPC, region, IP ranges, private connectivity, and fleet boundaries.
  3. Default to Autopilot workloads; document Standard exceptions.
  4. Design identities across IAM, RBAC, and Workload Identity Federation.
  5. Choose exposure: internal/external, L4/L7, Gateway/Ingress, DNS, Armor, certificates.
  6. Encode resources, probes, disruption budgets, topology, and autoscaling.
  7. Choose durable services and backup/restore paths.
  8. Build immutable supply chain and progressive delivery.
  9. Instrument SLOs, auditability, cost allocation, and runbooks.
  10. Practice node loss, zone loss, bad deploy, expired credential, and restore.

Capstone: regulated checkout

Start with a regional, private, VPC-native GKE cluster using Dataplane V2. Run conventional services as Autopilot workloads. Use distinct namespaces and Kubernetes ServiceAccounts, Workload Identity Federation for narrowly scoped Google API access, default-deny NetworkPolicy, external HTTPS load balancing with managed certificates and Cloud Armor, Artifact Registry plus Binary Authorization, and managed databases where they reduce undifferentiated stateful toil.

Spread replicas across zones; use readiness, startup, and liveness probes for their distinct purposes; define PDBs that still allow upgrades; use HPA on demand-aligned signals; export structured logs, metrics, traces, and audit data; stage release-channel upgrades through lower environments; back up resources and data; test restores. Then write down the exceptions and owners.

Lab 12 · Architecture review

Challenge the design