Here are my personal study notes from my preparation for the Certified Kubernetes Application Developer (CKAD) exam. While the main focus of these notes is the CKAD syllabus, they also touch upon additional aspects of Kubernetes that I think provides a more complete understanding and may prove useful to someone.

Components⌗

Master Node⌗

  • kubeadm: automates setting up the cluster.
  • kubectl: runs cluster commands.
  • kubelet: agent that manages pod containers. Runs as a system service, not as a pod.
  • Control Plane:
    • etcd: key/value store for cluster state.
    • API server: executes API commands.
    • Scheduler: determines the best node for a new pod.
    • Controller manager: responds to node status, maintains pod count, creates namespace tokens, and creates endpoints objects for services.

Controllers: A “control loop” operates an endless loop, checking the current status against the desired state and running tasks.

Worker Node⌗

  • kubelet: maintains containers and communicates with the API server.
  • kube-proxy: maintains network rules for pod communication from inside and outside the cluster with firewall rules to the routing table. See Networking.
  • container runtime: containerd, Docker engine, and others.

DaemonSets (Add-ons) for running a daemon on every node⌗

Examples can include logging services, monitoring services, running backup tasks.

Pods⌗

  • Contain containers.
  • Have their own storage resources.
  • Have a single network interface: eth0.

All nodes can communicate with all containers. Containers in the same pod can communicate with localhost.

Pod Example⌗ 

10.20.0.3
Container 1
    External: 10.20.0.3:2522
    Internal: 127.0.0.1:2522
Container 2
    External: 10.20.0.3:80
    Internal: 127.0.0.1:80

Init container⌗

Runs before an application container (e.g., checks for a service to become available or clones a Git repository into a volume).

Volumes⌗

Defined in the pod spec.

  • hostPath: data is stored in a specific location (discouraged for security purposes).
  • emptyDir: data is automatically managed in a temp location. Deleted when Pod dies.
  • persistentVolumeClaim: create persistent volumes (PV), create claims (PVC) to match/bind to a volume, mount the claim to the pod. StorageClass: "" - bind to a PV with no class.

Service⌗

An abstraction to expose an application running on a set of pods. Service A does not need to know the pod IP addresses of Service B to communicate. It has a virtual IP address and a list of chaining pod IP addresses. It uses the ‘matchLabels’ selector values of a Deployment to link a Service to a Deployment.

Workloads⌗

  • deployment: deploy an application and maintain its state using ReplicaSets.
  • ReplicaSet: configuration to maintain a stable set of duplicate pods. It is a subset of Deployments. Use Deployments and not ReplicaSets directly.
  • StatefulSet: keeps a unique identity for the pods it manages. Useful for stateful applications like DBs, event stores, but it’s better to use a managed solution or separate VMs.
  • DaemonSet: all nodes run a copy of the pod (e.g., logs or monitoring).
  • CronJob: recurring task that runs to completion then stops.
  • Job: task that runs to completion then stops.

Networking⌗

Request Paths⌗

External Request -> Ingress Resource -> Ingress Controller -> Service

External request to Node⌗

Request <-> Node (eth0 interface) <-> Pod (vethXXX interface). eth0 and one half of the veth exist in the base/root network namespace (netns)

Pod-to-Pod Same Node⌗

Pod 1 (eth0) -> Node root namespace (vethXXX) -> Node root namespace (cbr0) -> Node root namespace (vethYYY) -> Pod 2 (eth0)

Pod-to-Pod Cross Node⌗

All pods in a cluster have a unique IP address. Leaves the node (eth0) and uses a route table (CNI)

Pod-to-Service⌗

Uses linux firewall connection tracking (conntrack) table to keep packet address info. Packet is DNATed replacing the service ip with a pod ip. Is then un-DNATed, using conntrack, to return to the pod

Pod-to-External Service⌗

Similar to Pod-to-Service, uses SNAT to change the source IP to that of the outbound interface the uses conntrack to un-SNAT the response and return to the pod.

Container Network Interface (CNI)⌗

Create an overlay network that spans across all nodes allowing Pods to communicate with each other. Either edits iptables directly or uses kube-proxy. Some CNI’s disable kube-proxy. Daemon runs on each node and communicates with kube-api & etcd to keep up-to-date pod route information. Wraps the packet with extra source and destination information

Plugins offer different capabilites e.g. flannel is a layer 3 IPv4 overlay network, Calico uses Border Gateway Protocol, Multus offers multiple CNI’s for separation of data planes. Cloud Providers use their own custom CNI’s for managed Kubernetes services that interoperate with their platform.

kube-proxy⌗

Daemon. Mirror copy on each node. updates IP Tables. Load balances Service traffic to pods. Can run in three modes:

  • userspace. legacy. kube-proxy actually running as a proxy. Slow.
  • iptables. Default. Simple. Plays well with other iptable services. Slow when a cluster has a large number of services - O(n) style algorithm. Randomly pick a Pod.
  • IPVS. Linux load balancer. Complex. Better for clusters with a 1000+ services O(1) style algorithm

Used to be a proxy, but now is a controller that watches the api-server for end-point changes.

1.16+ can disable kube-proxy during the cluster init

Installing⌗

  • Install containerd or Docker (Docker is deprecated with Kubernetes 1.20).
  • Install kubeadm, kubelet, and kubectl.
  • Disable swap.
  • Initialize the master with kubeadm, join workers, and check with ‘kubectl get nodes’.
  • Set up networking CNI - Calico, Flannel.
  • Follow this tutorial to deploy a Kubernetes cluster on Ubuntu using kubeadm: https://computingforgeeks.com/deploy-kubernetes-cluster-on-ubuntu-with-kubeadm

Commands⌗

kubeadm⌗

sudo kubeadm init

kubeadm token create --print-join-command # cluster join command

sudo kubeadm reset

sudo -k # remove sudo timestamp for active user

kubectl⌗

kubectl explain <NOUN> # docs

kubectl explain <NOUN> | grep <SEARCH_TERM>

kubectl cluster-info dump | grep -m 1 cluster-cidr # check POD network CIDR

kubectl cluster-info

kubectl api-resources # show all Kubernetes objects

kubectl get all # overview

kubectl get nodes

kubectl describe nodes # full status

kubectl get names paces

kubectl create <RESOURCE>

kubectl apply -f <FILENAME_OR_DIRECTORY> # apply a configuration to a resource

kubectl scale <RESOURCE> --replicas=n # set size for a deployment, replica set, stateful set

kubectl describe <RESOURCE> # kubectl describe rs/backend

kubectl exec <POD_NAME> -- <COMMAND> # kubectl exec -it nginx -- /bin/bash

Pods⌗

Create a static pod. Pods are not usually created directly because they are intended to be ephemeral. Use a Deployment or Jobs. kubectl apply -f ./simple-pod.yaml

kubectl apply -f https://k8s.io/examples/pods/simple-pod.yaml

kubectl get pods

watch kubectl get pods # real-time updates

kubectl get pods --all-namespaces -o wide

kubectl get pods -n my-namespace -w # -w watches changes

kubectl get pods --show-labels

kubectl describe pod <POD_NAME>

kubectl exec -it alpine -- apk --update add curl # execute command in pod

kubectl exec -it <POD_NAME> -- /bin/bash

Deployment⌗

kubectl get deployments

kubectl describe deployment <DEPLOYMENT_NAME>

kubectl edit deployment <DEPLOYMENT_NAME> # inline editing. Does not save to YAML

kubectl rollout status deployment/<DEPLOYMENT_NAME>

kubectl rollout history deployment/<DEPLOYMENT_NAME>

kubectl rollout undo deployment/<DEPLOYMENT_NAME>

kubectl rollout undo deployment/<DEPLOYMENT_NAME> --to-revision=X

ReplicaSet⌗

kubectl get rs

Logs⌗

kubectl logs <POD_NAME>

kubectl logs <POD_NAME> -c <CONTAINER_NAME> # required if the pod has multiple containers

kubectl logs --follow <POD_NAME> # tail logs

/var/log/containers/ - raw container logs

Service⌗

kubectl expose <RESOURCE> <RESOURCE_NAME> --type=ClusterIP|NodePort|LoadBalancer --port=8080 --target-port=80 --name=my-service

Jobs⌗

kubectl get jobs

kubectl describe job <JOB_NAME>

kubectl delete cronjob <CRONJOB_NAME>

restartPolicy: OnFailure | Never

activeDeadlineSeconds: 60 # terminate a job if it runs too long (in seconds)

Namespace⌗

kubectl create namespace <NAMESPACE_NAME>

kubectl delete all --all -n <NAMESPACE_NAME> # delete all resources for a namespace

inline create a service, deployment, pod…

cat << EOF | kubectl apply -f -
...
EOF

deploy multiple resources

kubectl apply -f <FILE_PATH.yml> -f <FILE_PATH.yml>

kubectl apply -f <FILE_PATH.yml> -f <FILE_PATH.yml>

kubectl delete deployment,services -l app=nginx # -l is label

Labels⌗

kubectl label <RESOURCE> <RESOURCE_NAME> <KEY>=<VALUE>

kubectl label namespace hive partition=customerA

Probe⌗

  • liveness: when to restart a container
  • readiness: ready to accept traffic (READY status from kubectl get pods)
  • startup: when application has started. For slow starting containers. Disables other two probes

Custom Resource⌗

kubectl get crd

ServiceAccount⌗

role -> role binding -> service account

/var/run/secrets/kubernetes.io/ # secrets location inside pods

Network Policies⌗

  • non-isolated pod: default. Open to all traffic in the network
  • isolated pod: ‘selected’ in a network policy. Traffic has to be explicitly allowed

Metrics⌗

https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml # --kubelet-insecure-tls

kubectl top <pods | nodes>

To see pod metrics without metrics server

kubectl exec -it <POD_NAME> -- /bin/bash

cat /sys/fs/cgroup/cpu/cpuacct.usage # cpu usage

cat /sys/fs/cgroup/memory/memory.usage_in_bytes # memory used + cache (top does not include cache)

Helm⌗

helm repo add <REPO_NAME> <REPO_ADDRESS>

helm repo update

helm search repo <SEARCH_TERM>

helm install -n <NAMESPACE_NAME> <APPLICATION_NAME> <HELM_CHART_NAME>

helm uninstall -n <NAMESPACE_NAME> <APPLICATION_NAME>

Network⌗

ip route # see all node routes

ip netns # see all netowrk namespaces

/etc/resolv.conf # location inside containers for DNS settings

sudo iptables -L -n -v # list iptable rules

Debugging⌗

(sudo) journalctl -xe --unit kubelet

systemctl status kubelet

systemctl status containerd

(sudo) journalctl -xe --unit containerd

crictl - same commands as docker

crictl pods

nerdctl --namespace k8s.io ps -a

Multi Container Patterns⌗

  • sidecar: extend or enchance the main container e.g. send logs to external system
  • ambassador: proxy network connection to the main container. e.g. main container connects to service B on port 80. ambassador container proxys the traffic from port 80 to a different port.
  • adaptor: transform the output of the main container e.g. transform app log data to a standard format

Deployment Strategies⌗

Use a service in-front of the deployment so new deployments can be deployed and pointed to if needed

Canary - send some production traffic to a new deployment to test it. Both deployments match the same selector used by the service, but the canary deployment has a smaller proportion of the replicas e.g. canary replicas = 1, main replicas = 4

Blue/Green - blue is running, spin up green deployment, point service to green deployment via selectors

Deployment

spec:
  selector:
    matchLabels:
      app: nginx
      color: green
Service
spec:
  selector:
    app: nginx
    color: green

Bash While loop

command: ["/bin/sh"]
args: ["-c", "while true; do echo hello; sleep 10; done"]

Yaml Examples⌗

Inline

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
EOF

Or store in yaml /k8s/descriptors and run with kubectl apply -f https://k8s.io/examples/application/deployment.yaml

CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: hello
  namespace: hello
spec:
  schedule: "* * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox:1.28
            imagePullPolicy: IfNotPresent
            command:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure