Skip to main content

StatefulSets Revisited

Equipped with a understanding of how PostgreSQL streaming replication works and how it is set up, the next step is to think about its automation. As a preliminary step - to sharpen the knives so to say - a closer look at the behavior of StatefulSets will help to find solutions to upcoming automation challenges quicker.

Note: This chapter connects to the StatefulSet chapters of the Kubernetes tutorial. So in case you have deleted these resources go back and re-create the final PostgreSQL StatefulSet presented there in the pg namespace.

A StatefulSet with Multiple Replicas

A highly available PostgreSQL cluster should have 2n+1 | n>= 1 Pods. These Pods have to discover another as they need to establish network connections as part of the replication and cluster management communication.

The Default is Replicas: 3

Throughout the remainder of this tutorial the default will be replicas: 3 as the transition from replicas: 1 to replicas: 3 implies all challenges and subsequent scale-outs to n>=2 such as replicas: 5 will be much easier to handle.

Creating a PostgreSQL StatefulSet with 3 Replicas

As a first step create a file 10-statefulset.yaml:

apiVersion: apps/v1
kind: StatefulSet
name: postgresql-sfs
app: postgresql-a # has to match .spec.template.metadata.labels
serviceName: "postgresql-svc"
replicas: 3 # by default is 1
app: postgresql-a # has to match .spec.selector.matchLabels
terminationGracePeriodSeconds: 10
- name: postgresql-db
image: postgres:14.5
name: postgresql-secrets
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
- containerPort: 5432
name: postgresql-port
- name: data
mountPath: /var/lib/postgresql/data
- metadata:
name: data
accessModes: [ "ReadWriteOnce" ]
storage: 1Gi

Apply it:

kubectl apply -f `10-statefulset.yaml`

You should now see a StatefulSet with three Pods:

kubectl get pods


NAME                     READY   STATUS    RESTARTS   AGE
postgresql-sfs-0 1/1 Running 0 45m
postgresql-sfs-1 1/1 Running 0 45m
postgresql-sfs-2 1/1 Running 0 45m

There should also be the corresponding headless Service:

kubectl get service postgresql-svc


NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
postgresql-svc ClusterIP None <none> 5432/TCP 5h55m

Cluster Connectivity

As stated earlier, in order to build a replicating PostgreSQL cluster, the individual cluster nodes must be able to communicate with each other.

A first thought could be to create three headless Services, one for each member of the 3-replica StatefulSet but in fact this isn't necessary. Kubernetes will automatically propagate SRV records [1] in its DNS system [2]. Go and see yourself:

Start a utility Pod:

kubectl run -i --tty nspct --image=fischerjulian/nspct --restart=Never -- bash

As you already know you can lookup the Pod IP addresses of the StatefulSet using a DNS query such as:

nslookup postgresql-svc.k8s-training.svc.cluster.local



Name: postgresql-svc.k8s-training.svc.cluster.local
Name: postgresql-svc.k8s-training.svc.cluster.local
Name: postgresql-svc.k8s-training.svc.cluster.local

Remember that the headless Service uses DNS entries exclusively and does not perform a layer-4 load balancing.

When bootstrapping the Pods of the PostgreSQL we will need more control than this as we need to reach out to each Pod by its role or even its name. Reaching to a particular node in the StatefulSet is simple as shown in the following.

Within the utility Pod nspct execute:

host -t SRV postgresql-svc.k8s-training.svc.cluster.local


postgresql-svc.k8s-training.svc.cluster.local has SRV record 0 33 5432 postgresql-sfs-0.postgresql-svc.k8s-training.svc.cluster.local.
postgresql-svc.k8s-training.svc.cluster.local has SRV record 0 33 5432 postgresql-sfs-1.postgresql-svc.k8s-training.svc.cluster.local.
postgresql-svc.k8s-training.svc.cluster.local has SRV record 0 33 5432 postgresql-sfs-2.postgresql-svc.k8s-training.svc.cluster.local.

This means that each Pod of the StatefulSet has a stable unique network identifier depending on the name of the headless Service as well as the namespace:

  1. postgresql-sfs-0.postgresql-svc.k8s-training.svc.cluster.local -> Pod 0
  2. postgresql-sfs-1.postgresql-svc.k8s-training.svc.cluster.local -> Pod 1
  3. postgresql-sfs-2.postgresql-svc.k8s-training.svc.cluster.local -> Pod 2

Knowing that each node runs a PostgreSQL on port 5432 this means that you can connect to each Pod's PostgreSQL using a reliable and predictable URL, here using telnet in the nspct Pod:

kubectl run -i --tty nspct --image=fischerjulian/nspct --restart=Never -- bash

And within the nspct Pod:

telnet postgresql-sfs-0.postgresql-svc.k8s-training.svc.cluster.local 5432


Connected to postgresql-sfs-0.postgresql-svc.k8s-training.svc.cluster.local.
Escape character is '^]'

Hence, it is also possible to reach out from one StatefulSet Pod to another Pod using these SRV records.

Another important fact about the headless Service is that it will reflect upon changes of the underlying Pods. In case a Pod fails and has to be rescheduled the headless Service will resolve to the new Pod's IP address within seconds. This will allow self-healing of Pods to occur without the need for configuration file updates such as the pg_hba.conf for authentication or the postgresql.conf to update the primary_conninfo attribute.


The experiments with the StatefulSet show that the combination with a headless Service provides a stable network identity required for the automation of a PostgreSQL stream replication cluster. The headless Service provides an endpoint that resolves to all Pods of the StatefulSet but also provides endpoints to reach out to specific Pods while being up-to-date with underlying Pod changes. This is very promising, but we have to keep in eye on the timing as propagating DNS entries may take a few seconds. Hence, race conditions may occur during the Pod startup between the assumption about the presence of DNS entry and its actual manifestation.


  1. Wikipedia, SRV Record,
  2. Kubernetes Documentation, Tutorials, StatefulSet Basics,