new article on fresh nixos installs
All checks were successful
Build OCI Image / docker (push) Successful in 1m35s

This commit is contained in:
2025-10-28 12:44:25 +08:00
parent 736d6e9d3b
commit 67171c05d9
2 changed files with 131 additions and 1 deletions

View File

@@ -4,7 +4,7 @@
.author = "Sample Author",
.layout = "post.shtml",
.draft = false,
.tags = ["k3s", "gitea", "postgres", "kubernetes", "infranut"],
.tags = ["k3s", "gitea", "postgres", "kubernetes", "nutinfra"],
---
# The problem
[Gitea](https://git.yadunut.dev) running on my k3s cluster recently died on me. Surprisingly, it wasn't an issue with the cluster (which is usually the case due to my tinkering), but the postgres and redis containers failing to start. Looking into the k8s logs, I found out that it was due bitnami removing their repositories for postgres and valkey.

View File

@@ -0,0 +1,130 @@
---
.title = "NixOS on new systems",
.date = @date("2025-10-28"),
.author = "Yadunand Prem",
.layout = "post.shtml",
.draft = false,
.tags = ["k3s", "kubernetes", "nutinfra", "nix", "nixOS"],
---
# Fresh installing NixOS on a new system
So greencloudVPS just had its 12th birthday sale, and with greed, i bought their server bundle... 220 USD/3 years for a 4 core, 12GB ram server with 4 TB storage was a server deal I could not give up on... So with 4TB, I wanted it as my offsite backup server + another node for my kubernetes cluster. And now with 2 public IPs, I could do fun stuff like floating IP for more reliability.
Another thing I want to do is slowly migrate away from my 1 \$5/month digital ocean server, which mainly runs headscale, to one of these servers instead.
So the question is, do I really need headscale, or can I just use plain wireguard and with with nix, is it much easier to push my changes to a cluster now with tools like colmena? My [homelab](https://github.com/yadunut/homelab) is using colmena right now, but I'm slowly migrating to [this repo](https://git.yadunut.dev/yadunut/nix) for all my nix needs.
Another thing to look into is completely restructuring my current infra setup, now that I have more than 1 cloud node.
But I digress, this article is on installing nixOS on this system.
## The tools
- [nixos anywhere](https://github.com/nix-community/nixos-anywhere)
NixOS anywhere, as its name suggests, is a tool to install nixOS on any linux system supporting kexec over ssh.
- [disko](https://github.com/nix-community/disko)
Nix tool for declarative disk partitioning
## Setup
asd
### Figuring out partitioning with disko
The nut-gc2 server has 2 drives, vda at 60GB, and vdb, at 4TB. I want the vda to be the primary partition, storing the root file system, and VDB just being used for longhorn.
So lets get that out in disko. I initially used UEFI but i was having issues with it booting up, so I'm switching to BIOS.
```nix
{
disko.devices = {
disk = {
main = {
type = "disk";
device = "/dev/vda";
content = {
type = "gpt";
partitions = {
boot = {
size = "1M";
type = "EF02";
};
ESP = {
size = "512M";
type = "EF00";
content = {
type = "filesystem";
format = "vfat";
mountpoint = "/boot";
mountOptions = [ "umask=0077" ];
};
};
root = {
end = "-4G";
content = {
type = "filesystem";
format = "ext4";
mountpoint = "/";
mountOptions = [ "noatime" ];
};
};
swap = {
size = "100%";
content = {
type = "swap";
discardPolicy = "both";
resumeDevice = false;
};
};
};
};
};
vdb = {
type = "disk";
device = "/dev/vdb";
content = {
type = "gpt";
partitions = {
root = {
size = "100%";
content = {
type = "filesystem";
format = "ext4";
mountpoint = "/srv";
mountOptions = [ "noatime" ];
};
};
};
};
};
};
};
}
```
A simple configuration creating 3 partitions on vda, boot, root and swap, and 1 partition for data mounted to srv. I'll be serving longhorn from the /srv/longhorn directory and garage for s3 from the /srv/garage directory.
Next would be the general system configuration. Since this is long and very custom, I'm just gonna link it here, and you can view this in [the repo](https://git.yadunut.dev/yadunut/nix/src/branch/main/systems/x86_64-linux/nut-gc2/default.nix).
Now that the configuration is done, I have secrets(such as my k3s token) that I want to transfer over to the new system. To cleanly do this, I can generate the host keys on this system, use that to rekey the secret, transfer over the host keys and then erase them from my system when complete.
```sh
tmp_dir=$(mktemp -d)
mkdir -p "$tmp_dir/etc/ssh"
MACHINE_NAME="nut-gc2"
ssh-keygen -A -f "$tmp_dir"
for key in "$tmp_dir"/etc/ssh/ssh_host_*_key; do
[ -f "$key" ] || continue
echo "Updating comment on $key ..."
ssh-keygen -q -c -C "root@$MACHINE_NAME" -f "$key"
done
echo "ssh key: $(cat $tmp_dir/etc/ssh/ssh_host_ed25519_key.pub)"
```
Now with the ssh key generated, we can copy that over into our secrets.nix, and use age to rekey the secret
```sh
agenix --rekey
```
With this rekeyed, we can get to infecting the system
```sh
nix run github:nix-community/nixos-anywhere -- --generate-hardware-config nixos-generate-config ./systems/x86_64-linux/nut-gc2/hardware-configuration.nix --flake ".#${MACHINE_NAME}" --extra-files "${tmp_dir}" --print-build-logs --build-on remote "root@${MACHINE_IP}"
```
Now with the server infected, you'll need to update your `~/.ssh/known_hosts` and you can then ssh into your machine