Kubernetes v2 on Alperen Doc

helm

Mon, 01 Jan 0001 00:00:00 +0000

Helm#

In the v1 notes we wrote every manifest by hand: deployments, configmaps, services, gateways. That’s fine for a handful of resources you own. It falls apart the moment you want Prometheus + Grafana + Loki - three interdependent applications that together ship hundreds of YAML lines, with version matrices between them.

Helm is the package manager for Kubernetes. Instead of raw manifests you install charts: versioned, templatized bundles that ship with sane defaults you can override. Think apt/npm, but the package installs into your cluster.

kustomize

Mon, 01 Jan 0001 00:00:00 +0000

Kustomize#

Helm solves packaging with templates and values. Kustomize refuses templates entirely: keep your plain YAML as the base, and build overlays that patch in environment differences. Since v1.27 kubectl ships with it built in - no extra binary:

kubectl apply -k ./my-app

The Mental Model#

k8s-manifests/
├── base/
│ ├── api-deployment.yaml
│ ├── api-configmap.yaml
│ ├── web-deployment.yaml
│ └── kustomization.yaml # lists resources
├── overlays/
│ ├── dev/
│ │ └── kustomization.yaml # patches: replicas: 1
│ └── prod/
│ ├── kustomization.yaml
│ └── patch-replicas.yaml

base = the truth of the application (works as-is on any cluster)
overlay = a set of patches on top of the base (environment-specific deltas)

The rule that makes it work: overlays only touch fields they declare. Everything else passes through unchanged.

cluster-extensions

Mon, 01 Jan 0001 00:00:00 +0000

Extension Interfaces#

Everything we’ve installed so far - the CNI plugin that gives pods their network, the Longhorn CSI driver that gives them disks, the containerd runtime that runs them - shares one thing in common: none of them are “Kubernetes”. They’re plugins, and Kubernetes just provides the sockets they plug into.

Those sockets are the extension interfaces, and knowing them turns “the cluster is broken” into “the network plugin is broken”:

Arayüz	Ne işe yarar	Kim implement eder
CNI	Pod’lara IP ve network connectivity	Calico, Flannel, Cilium, Weave
CSI	Volume provisioning, attach, mount	Longhorn, cloud disk drivers, Ceph
CRI	Container’ları gerçekten çalıştırmak	containerd, CRI-O, (dockershim gitti)

The contract matters more than the vendor. CNI says “here’s a pod, give it an IP and a veth”; every plugin answers that the same way of thinking, differently in the details. Same for CSI (CreateVolume/Attach/Mount are verbs every driver must know) and CRI (RunPodSandbox/CreateContainer/StopContainer).

network-policy

Mon, 01 Jan 0001 00:00:00 +0000

NetworkPolicy#

When we built custom bridge networks in Docker, the goal was isolation: containers on network A can’t see containers on network B. NetworkPolicy is the cluster-level equivalent - with a twist that trips everyone up at first.

The default is “allow everything”. No policy = every pod can talk to every pod, everywhere. But the moment a pod is selected by any NetworkPolicy (Ingress or Egress, even an empty one), that flips: all traffic to/from that pod is denied unless another policy explicitly allows it. Policies are additive: the union of all rules selecting a pod defines its allowed traffic. There is no “deny” rule - you deny by omission.

security-context

Mon, 01 Jan 0001 00:00:00 +0000

Security Context#

In manage-docker-as-non-root-user we made the case for not running containers as root on Docker. Kubernetes has the same problem: most container images still run their process as uid 0. A Security Context sets the security options for a pod or container - and there’s a whole family of them, not just “the user”.

The Field Set#

Container level (per container in spec.containers[].securityContext):

securityContext:
 runAsNonRoot: true
 runAsUser: 1000
 runAsGroup: 1000
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 capabilities:
 drop: ["ALL"]
 seccompProfile:
 type: RuntimeDefault

runAsNonRoot: true - kubelet refuses to start the container if the image’s effective user resolves to 0. The most valuable single line in this chapter
runAsUser/runAsGroup - override the image’s user with numeric UID/GID (names don’t work; id -u of the image user is what you need)
allowPrivilegeEscalation: false - blocks setuid binaries and capability-gain inside the container
readOnlyRootFilesystem: true - container filesystem becomes immutable; writes must target mounted volumes
capabilities.drop: ["ALL"] - drops Linux capabilities; add back only what the app genuinely needs (add: ["NET_BIND_SERVICE"] for binding ports < 1024)
seccompProfile: RuntimeDefault - applies the container runtime’s default syscall filter. In restricted PSA this is required, and most runtimes’ default is a reasonable baseline

Pod level (spec.securityContext) - applies to every container and to volumes:

kubernetes-troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Troubleshooting#

Everything that follows builds on one rule: status codes first, logs second, describe always. After writing dozens of deployments in the v1 notes, here’s the operator-grade field guide.

Pod Status Quick Reference#

Status	Anlamı	İlk bakılacak yer
`Pending`	Node’a schedule edilemedi	`describe` → Events (resources? taints? affinity? quota?)
`ImagePullBackOff`	Image çekilemiyor (tag/repo/auth)	`describe` → Events + image adı
`CrashLoopBackOff`	Başlıyor, çöküyor, üstel beklemeyle tekrar	`logs --previous` + exit code
`ErrImagePull`	`ImagePullBackOff`‘un ilk hali	Aynı
`Running` ama broken	Process yaşıyor, app ölü	`logs` + `port-forward` ile elle dene
`OOMKilled`	Memory limiti aşıldı	`describe` → `Last State`

Exit Code Cheat Sheet#

The exit code in Last State: Terminated usually tells the whole story:

control-plane-troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Control Plane Components#

In the namespaces notes we listed what lives in kube-system and said “you don’t want to mess with it”. Good advice - but understanding it is non-negotiable, because when the control plane misbehaves every debugging tool you own stops working. kubectl get pods that returns the connection to the server was refused is not a pod problem.

The four components, and what they own:

Bileşen	Ne yapar	Nasıl koşar
kube-apiserver	Tek API yüzeyi; her `kubectl` buraya gider	static pod
etcd	Tüm cluster state’inin tek kaynağı(key-value storage)	static pod
kube-scheduler	Pod’ları node’lara (birden fazla kube-scheduler olusturulabilir) yerleştirir	static pod
kube-controller-manager	Reconcile döngüleri (deployment, rs, node…)	static pod

On a kubeadm cluster these run as static pods - not Deployments, not systemd units. The kubelet reads YAML files from /etc/kubernetes/manifests/ and keeps whatever is in there alive, forever. Delete a manifest and the component dies; restore the file and it comes back. That directory is the single most important path on a control-plane node.

gitops-argocd

Mon, 01 Jan 0001 00:00:00 +0000

GitOps#

My site deploys with GitHub Actions + rsync over SSH (see github_workflow). That’s a push model: CI runs, then pushes files to the server. GitOps flips it: the cluster pulls its own desired state from git and reconciles itself continuously.

The four principles (from OpenGitOps):

Declarative - the entire desired state is expressed declaratively (manifests, charts, kustomize overlays)
Versioned and immutable - state lives in git; every change is a commit, every deploy is a merge
Pulled automatically - an agent inside the cluster pulls; nothing outside pushes into the cluster
Continuously reconciled - actual state is compared with desired state all the time, and drift is corrected

That fourth principle is what push-based CD structurally cannot give you. With push CD, the 2am kubectl edit sticks forever - nobody re-runs the pipeline for a change they didn’t make. With pull CD, drift is detected within seconds and reverted.

vpa-cluster-autoscaler

Mon, 01 Jan 0001 00:00:00 +0000

The Three Autoscalers#

In the scaling notes we built an HPA. But HPA is one of three autoscalers, and they operate at different layers - mixing them up is a classic design mistake:

Autoscaler	Katman	Soru	Ne yapar
HPA	Pod sayısı	“Kaç pod?”	Metriğe göre replica sayısını değiştirir
VPA	Pod kaynakları	“Bir pod ne kadar ister?”	`requests`/`limits`‘i otomatik ayarlar (restart ile)
Cluster Autoscaler	Node sayısı	“Kaç node?”	Pending pod’lar için node ekler, boş node’u çıkarır

The layering matters because they solve different problems: HPA handles variable load for a fixed-per-pod footprint; VPA fixes wrong resource estimates; CA fixes fleet capacity.

taints-affinity-quotas

Mon, 01 Jan 0001 00:00:00 +0000

Taints and Tolerations#

On a single-node Minikube cluster scheduling is trivial - there’s nowhere else to put a pod. The moment you add nodes (multi-node notes are coming), scheduling decisions become real. Taints mark a node as “keep away”, and tolerations let specific pods opt in anyway:

kubectl taint nodes <node-name> workload=ai:NoSchedule

The three effects behave differently:

NoSchedule: new pods without a matching toleration can’t be scheduled here (running pods are left alone)
PreferNoSchedule: soft version - scheduler tries to avoid, but won’t reject
NoExecute: also evicts pods already running there that lack the toleration - the aggressive one

The matching toleration in a pod spec:

pod-priority-disruption

Mon, 01 Jan 0001 00:00:00 +0000

DaemonSet#

Deployments spread replicas wherever there’s room. Sometimes “wherever there’s room” is the wrong answer - a log shipper, a CSI node agent, or a CNI plugin needs exactly one pod per node, no more, no less. That’s a DaemonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: node-agent
spec:
 selector:
 matchLabels:
 app: node-agent
 template:
 metadata:
 labels:
 app: node-agent
 spec:
 tolerations:
 - operator: Exists
 containers:
 - name: agent
 image: busybox:1.36
 command: ["sh", "-c", "sleep infinity"]

Look at that toleration - operator: Exists tolerates everything, which is exactly what you want when the DaemonSet must land on control-plane nodes too. Most system DaemonSets (kube-proxy, Longhorn managers, Calico’s node agents) ship with this, because a plugin that avoids the control plane is a plugin that half-works.

multi-node-kubeadm

Mon, 01 Jan 0001 00:00:00 +0000

Multi-Node: From Minikube to a Real Cluster#

Minikube is a single-node cluster - we’ve repeated that warning since the very first v1 notes. For homelab work the standard path to real multi-node Kubernetes is kubeadm - vanilla k8s, everything in your hands, the same tool the managed distros are built on. This note walks through a kubeadm install on VMs, from node prep to a joined, working cluster, with Cilium as the CNI - eBPF datapath, working NetworkPolicy enforcement and no kube-proxy.

kubeadm-upgrade-etcd

Mon, 01 Jan 0001 00:00:00 +0000

Cluster Upgrades#

The multi-node notes ended with a one-liner: “upgrade the control plane first, one minor version at a time”. That’s the policy - this is the practice.

Upgrading a cluster is the most common maintenance task you will ever run, and the most common way to break one. The rules that keep it boring:

One minor version at a time. 1.30 → 1.31, never 1.30 → 1.32. The version skew policy allows the apiserver to lead kubelets by one minor; anything more is unsupported and the upgrade will refuse
Control plane first, workers after. The new apiserver must understand old kubelets, not the reverse
One node at a time. Drain → upgrade → uncordon → next
Read the output. kubeadm upgrade plan exists precisely because upgrades are not a single command

The apt packages are version-pinned with apt-mark hold from the install notes. Every upgrade starts with apt-mark unhold and ends with apt-mark hold again - forgetting the re-hold is how a future apt upgrade silently moves your cluster a minor version.

ha-control-plane

Mon, 01 Jan 0001 00:00:00 +0000

High Availability#

The multi-node notes left a check on the wall: --control-plane-endpoint=<VIP> “makes later HA possible”. This is later.

A cluster with one control-plane node has one single point of failure: kill that node and kubectl stops working, the scheduler stops placing pods, and every controller stops reconciling. The workloads keep running (kubelets and containers don’t need the apiserver to stay alive), but the cluster stops being managed. For a homelab that’s an annoyance; for anything real it’s an outage.

devops-control-plane

Mon, 01 Jan 0001 00:00:00 +0000

Who Owns the Control Plane#

Everything until now assumed we own the cluster: we built it on multi-node VMs, we SSH’d into it when it broke, we joined control-plane nodes by hand. That’s the homelab model. The moment a second team shows up, the question changes from “how do I run this” to “who is allowed to do what” - and that’s a platform engineering question, not a kubectl one.

The split that every real org lands on eventually:

cert-manager

Mon, 01 Jan 0001 00:00:00 +0000

cert-manager#

In the v1 ingress notes we said that real clusters “get this issued automatically by cert-manager instead of hand-rolling it”. That was a promise we haven’t paid off yet. Let’s pay it off.

Here is what a TLS certificate actually requires, every single time:

a private key
a certificate signing request built from that key
someone - a Certificate Authority - to sign it
a renewal before it expires (Let’s Encrypt certs last 90 days)
the result copied into a Secret in the right namespace, in the right format

Do that by hand for two hostnames and you have already lost an afternoon. cert-manager is a controller that does all five as a Kubernetes reconciliation loop: you declare what you want, it keeps it true forever - including the renewal.

cert-manager-production

Mon, 01 Jan 0001 00:00:00 +0000

cert-manager in Production#

The previous chapter taught the mechanism: Issuer, Certificate, Secret, and a Gateway that serves it. Everything worked because we controlled every variable - a fake domain, a self-signed root, a laptop that trusts whatever we tell it to.

Production removes that control. The domain is public and someone else owns the DNS. The certificate has to be trusted by browsers you don’t administer. It expires on a schedule you don’t choose. And when the automation silently stops working, the outage looks like “the website is down” rather than “a controller is in a crash loop”.

prometheus-stack

Mon, 01 Jan 0001 00:00:00 +0000

Monitoring with kube-prometheus-stack#

We’ve touched monitoring twice and never actually ran it. The helm notes installed the plain prometheus chart to teach values precedence, and cluster-extensions installed kube-prometheus-stack to show what an operator looks like. Neither taught you to operate a monitoring stack.

That’s this chapter. By the end you’ll have Prometheus scraping the cluster, Grafana showing it, Alertmanager paging you when it matters, and - the part people skip - a ServiceMonitor for your own application so its metrics land next to the system ones.

victoriametrics

Mon, 01 Jan 0001 00:00:00 +0000

VictoriaMetrics#

The prometheus-stack chapter gave you a working monitoring stack. This one asks a different question: if you were starting over, would you still pick Prometheus as the storage engine?

VictoriaMetrics is a metrics database that speaks the same scrape protocol and mostly the same query language, but is built differently underneath - columnar storage, aggressive compression, no local TSDB churn. The usual reasons people move: less memory per series, longer retention on the same disk, and cardinality explosions that don’t take the database down with them.

longhorn

Mon, 01 Jan 0001 00:00:00 +0000

Longhorn#

Everything we’ve done with storage so far lived on one node. In the v1 notes a PVC was a local disk on (the only) Minikube node, and in the multi-node notes our workers each keep their own local directories. That means the day kworker1 dies, every PVC hosted on it dies with it - block storage was the last single point of failure in our homelab.

Longhorn fixes that: it’s a distributed block storage system built as a CSI driver, purpose-made for Kubernetes. Every volume it serves is split into an engine plus N replicas (default 3) that live on different nodes - scheduled exactly like pods are.

postgresql-statefulset-replication

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL Streaming Replication (The Manual Way)#

In the StatefulSet notes we ran a single-replica PostgreSQL: pod dies, new pod mounts the same PVC, data survives. But there’s still exactly one copy of the data, and if the primary goes down, your database is down until it comes back. On a 4-node cluster with Longhorn underneath, we can finally do the real thing: a primary plus streaming replicas.

Two Different Layers of Protection#

This is the concept to get straight before touching YAML, because people conflate them constantly:

postgresql-longhorn-read-replicas

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL Read Replicas with CloudNativePG#

The manual setup taught us the mechanics: WAL, basebackup, standby. It also showed the catch - failover is a human job, and humans don’t get paged at 3am for fun. CloudNativePG (CNPG) is the operator that automates the whole lifecycle: primary election, failover, replica management, backups - as a declarative Cluster CRD. On top of Longhorn it turns our 4-node homelab into a genuinely production-shaped database platform.

What the Operator Buys You#

	Elle StatefulSet	CloudNativePG
Replication kurulumu	`pg_basebackup`, `pg_hba`, init script’ler	`instances: 3`
Failover	Manuel `pg_promote()` + split-brain riski	Otomatik: primary ölürse en güncel replica promote edilir
Primary DNS	Kendi service şemanı yazarsın	`-rw` / `-ro` / `-r` service üçlüsü hazır
Pg upgrade/resize	Kendin	Operator rolling update yapar
Monitoring	Kendin	Hazır metrics endpoint

The service trio is the part your apps touch - and it’s the read-replica payoff: