Blogpost on upgrading postgres in a k3s cluster

content/blog/gitea-postgres-upgrade.smd

@@ -0,0 +1,170 @@
+---
+.title = "Adventures of upgrading gitea",
+.date = @date("2025-10-08"),
+.author = "Sample Author",
+.layout = "post.shtml",
+.draft = false,
+.tags = ["k3s", "gitea", "postgres", "kubernetes", "infranut"],
+---
+# The problem
+[Gitea](https://git.yadunut.dev) running on my k3s cluster recently died on me. Surprisingly, it wasn't an issue with the cluster (which is usually the case due to my tinkering), but the postgres and redis containers failing to start. Looking into the k8s logs, I found out that it was due bitnami removing their repositories for postgres and valkey.
+
+Me, naively thinking that oh, this is probably resolved by upgrading the gitea helm chart (from v10 to v12), went ahead and did that. The issue did not resolve itself. Not only that, they changed how gitea actions was deployed, which was a whole another can of worms and a lot of swearing to get fixed.
+
+My next attempt was to pin the postgres and redis image and tag to the bitnamilegacy packages and that worked! The postgres and valkey images started.
+```diff
+--- a/apps/base/gitea.yaml
++++ b/apps/base/gitea.yaml
+@@ -108,11 +108,23 @@ spec:
+       enabled: false
+     valkey:
+       enabled: true
++      image:
++        repository: valkey/valkey
++        tag: 8.1.3-alpine
++      global:
++        security:
++          allowInsecureImages: true
+       auth:
+         existingSecret: gitea-valkey
+         existingSecretPasswordKey: password
+     postgresql:
+       enabled: true
++      image:
++        repository: postgres
++        tag: 17.6-alpine
++      global:
++        security:
++          allowInsecureImages: true
+     postgresql-ha:
+       enabled: false
+```
+
+This then lead to the next issue. Updating postgres v16 to v17 is not a simple matter of changing the image tag. You need to run a postgres upgrade, which isn't the simplest thing to do. Due to it being exam season and me being busy with school, I decided to go for the simple solution, and just pin the image tag to v16 and be done with it, which worked and gitea was back up and running.
+
+## Upgrading postgres from v16 to v17 in a kubernetes cluster
+
+I knew this was a temporary solution, and I should be upgrading my postgres instance. So today, I decided to tackle the problem head on and upgrade the postgres from v16 to v17.
+
+A slight increase in complexity came from me being still not 100% familiar with kubernetes, and fighting with the different locations of config files, but it was still relatively straightforward.
+
+The steps below showcase how I did it.
+
+1. Suspend flux and scale gitea down
+
+```sh
+flux suspend hr -n gitea gitea
+kubectl -n gitea scale deploy gitea --replicas=0
+kubectl -n gitea scale statefulset gitea-postgresql --replicas=0
+```
+
+2. Create a [postgres-upgrade](https://hub.docker.com/r/tianon/postgres-upgrade) pod to do the upgrades from
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: pg-upgrade-shell
+  namespace: gitea
+spec:
+  restartPolicy: Never
+  containers:
+  - name: shell
+    image: tianon/postgres-upgrade:16-to-17
+    command: ["/bin/bash","-lc","sleep infinity"]
+    securityContext:
+      runAsUser: 0
+    volumeMounts:
+    - name: pgdata
+      mountPath: /bitnami
+  volumes:
+  - name: pgdata
+    persistentVolumeClaim:
+      claimName: data-gitea-postgresql-0
+```
+and deploy pod
+```sh
+kubectl -n gitea apply -f pg-upgrade-shell.yaml
+kubectl -n gitea exec -it pg-upgrade-shell -- bash
+```
+
+3. Do the migration
+```bash
+export OLD="/bitnami/data"
+export NEW="/bitnami/pg17/data"
+
+export PG16=/usr/lib/postgresql/16/bin
+export PG17=/usr/lib/postgresql/17/bin
+
+export RUNAS="runuser -u pguser --"
+
+"$PG16/pg_controldata" "$OLD" | grep -E 'pg_control version number|Database system identifier' || true
+
+# 2) Capture the numbers
+OWNER_UID=$(stat -c '%u' "$OLD")
+OWNER_GID=$(stat -c '%g' "$OLD")
+
+groupadd -g "$OWNER_GID" pggrp
+useradd -u "$OWNER_UID" -g "$OWNER_GID" -m -s /bin/bash pguser
+
+mkdir -p "$NEW"
+chown -R "$OWNER_UID:$OWNER_GID" "$NEW"
+
+$RUNAS "$PG17/initdb" -D "$NEW" --encoding=UTF8 --locale=C --username=postgres
+
+printf '' > "$OLD/pg_upgrade_postgresql.conf"
+printf '' > "$NEW/pg_upgrade_postgresql.conf"
+
+{ echo 'local   all   all    trust'; cat "$OLD/pg_hba.conf"; } \
+  > "$OLD/pg_upgrade_pg_hba.conf"
+
+{ echo 'local   all   all    trust'; cat "$NEW/pg_hba.conf"; } \
+  > "$NEW/pg_upgrade_pg_hba.conf"
+
+# Ensure this runs successfully
+$RUNAS "$PG17/pg_upgrade" \
+  -U postgres \
+  -d "$OLD" -D "$NEW" \
+  -b "$PG16" -B "$PG17" \
+  -o "-c config_file=$OLD/pg_upgrade_postgresql.conf \
+      -c hba_file=$OLD/pg_upgrade_pg_hba.conf \
+      -c unix_socket_directories=/var/lib/postgresql" \
+  -O "-c config_file=$NEW/pg_upgrade_postgresql.conf \
+      -c hba_file=$NEW/pg_upgrade_pg_hba.conf \
+      -c unix_socket_directories=/var/lib/postgresql" \
+  --check
+
+# Do the migration
+$RUNAS "$PG17/pg_upgrade" \
+  -U postgres \
+  -d "$OLD" -D "$NEW" \
+  -b "$PG16" -B "$PG17" \
+  -o "-c config_file=$OLD/pg_upgrade_postgresql.conf \
+      -c hba_file=$OLD/pg_upgrade_pg_hba.conf \
+      -c unix_socket_directories=/var/lib/postgresql" \
+  -O "-c config_file=$NEW/pg_upgrade_postgresql.conf \
+      -c hba_file=$NEW/pg_upgrade_pg_hba.conf \
+      -c unix_socket_directories=/var/lib/postgresql"
+
+mv "$OLD" /bitnami/pg16.old
+mv "$NEW" "$OLD"
+rm "$OLD/pg_upgrade_postgresql.conf"
+rm "$OLD/pg_upgrade_pg_hba.conf"
+```
+
+4. Remove the created upgrade container
+
+5. Spin back up flux
+```sh
+kubectl -n gitea scale deploy gitea --replicas=1
+kubectl -n gitea scale statefulset gitea-postgresql --replicas=1
+flux resume hr -n gitea gitea
+```
+
+Exec into the container and run vacuumdb to rebuild statistics
+```sh
+vacuumdb --all --analyze-only
+```
+
+And there you go! Upgraded the postgres for gitea from v16 to v17 (Even though v18 just got released, i might have to do this entire process again...)
+
+A lot of this was built with the assistance of ChatGPT5, but it was not a 1 shot solution, with a lot of iteration (Especially with the permissioning and username stuff). As always, I'd never blindly trust the output of the models, but use the model as a pair programmer / an experienced chat buddy

content/blog/tmp.smd

@@ -1,27 +0,0 @@
----
-.title = "Second Post",
-.date = @date("1990-01-01"),
-.author = "Sample Author",
-.layout = "post.shtml",
-.draft = true,
----
-
-This second post is mainly here to show you that you can also create single file
-posts for convenience. The first post contains more interesting content.
-
-Don't forget to read [the official SuperMD
-docs](https://zine-ssg.io/docs/supermd/) to know how to *style* your content.
-
-
-Btw this sample website also includes the JS/CSS dependencies required to render
-math:
-
-```=mathtex
-\begin{aligned}
-f(t) &= \int_{-\infty}^\infty F(\omega) \cdot (-1)^{2 \omega t} \mathrm{d}\omega \\
-F(\omega) &= \int_{-\infty}^\infty f(t) \div (-1)^{2 \omega t} \mathrm{d}t \\
-\end{aligned}
-```
-
-This: [`(-1)^x = \cos(\pi x) + i\sin(\pi x)`]($mathtex) is an inline equation
-instead!