Yadunand Prem
67171c05d9
All checks were successful
Build OCI Image / docker (push) Successful in 1m35s
131 lines
5.1 KiB
Plaintext
131 lines
5.1 KiB
Plaintext
---
|
|
.title = "NixOS on new systems",
|
|
.date = @date("2025-10-28"),
|
|
.author = "Yadunand Prem",
|
|
.layout = "post.shtml",
|
|
.draft = false,
|
|
.tags = ["k3s", "kubernetes", "nutinfra", "nix", "nixOS"],
|
|
---
|
|
# Fresh installing NixOS on a new system
|
|
|
|
So greencloudVPS just had its 12th birthday sale, and with greed, i bought their server bundle... 220 USD/3 years for a 4 core, 12GB ram server with 4 TB storage was a server deal I could not give up on... So with 4TB, I wanted it as my offsite backup server + another node for my kubernetes cluster. And now with 2 public IPs, I could do fun stuff like floating IP for more reliability.
|
|
|
|
Another thing I want to do is slowly migrate away from my 1 \$5/month digital ocean server, which mainly runs headscale, to one of these servers instead.
|
|
|
|
So the question is, do I really need headscale, or can I just use plain wireguard and with with nix, is it much easier to push my changes to a cluster now with tools like colmena? My [homelab](https://github.com/yadunut/homelab) is using colmena right now, but I'm slowly migrating to [this repo](https://git.yadunut.dev/yadunut/nix) for all my nix needs.
|
|
|
|
Another thing to look into is completely restructuring my current infra setup, now that I have more than 1 cloud node.
|
|
|
|
But I digress, this article is on installing nixOS on this system.
|
|
|
|
## The tools
|
|
- [nixos anywhere](https://github.com/nix-community/nixos-anywhere)
|
|
NixOS anywhere, as its name suggests, is a tool to install nixOS on any linux system supporting kexec over ssh.
|
|
- [disko](https://github.com/nix-community/disko)
|
|
Nix tool for declarative disk partitioning
|
|
|
|
## Setup
|
|
asd
|
|
|
|
### Figuring out partitioning with disko
|
|
The nut-gc2 server has 2 drives, vda at 60GB, and vdb, at 4TB. I want the vda to be the primary partition, storing the root file system, and VDB just being used for longhorn.
|
|
|
|
So lets get that out in disko. I initially used UEFI but i was having issues with it booting up, so I'm switching to BIOS.
|
|
```nix
|
|
{
|
|
disko.devices = {
|
|
disk = {
|
|
main = {
|
|
type = "disk";
|
|
device = "/dev/vda";
|
|
content = {
|
|
type = "gpt";
|
|
partitions = {
|
|
boot = {
|
|
size = "1M";
|
|
type = "EF02";
|
|
};
|
|
ESP = {
|
|
size = "512M";
|
|
type = "EF00";
|
|
content = {
|
|
type = "filesystem";
|
|
format = "vfat";
|
|
mountpoint = "/boot";
|
|
mountOptions = [ "umask=0077" ];
|
|
};
|
|
};
|
|
root = {
|
|
end = "-4G";
|
|
content = {
|
|
type = "filesystem";
|
|
format = "ext4";
|
|
mountpoint = "/";
|
|
mountOptions = [ "noatime" ];
|
|
};
|
|
};
|
|
swap = {
|
|
size = "100%";
|
|
content = {
|
|
type = "swap";
|
|
discardPolicy = "both";
|
|
resumeDevice = false;
|
|
};
|
|
};
|
|
};
|
|
};
|
|
};
|
|
vdb = {
|
|
type = "disk";
|
|
device = "/dev/vdb";
|
|
content = {
|
|
type = "gpt";
|
|
partitions = {
|
|
root = {
|
|
size = "100%";
|
|
content = {
|
|
type = "filesystem";
|
|
format = "ext4";
|
|
mountpoint = "/srv";
|
|
mountOptions = [ "noatime" ];
|
|
};
|
|
};
|
|
};
|
|
};
|
|
};
|
|
};
|
|
};
|
|
}
|
|
```
|
|
A simple configuration creating 3 partitions on vda, boot, root and swap, and 1 partition for data mounted to srv. I'll be serving longhorn from the /srv/longhorn directory and garage for s3 from the /srv/garage directory.
|
|
|
|
Next would be the general system configuration. Since this is long and very custom, I'm just gonna link it here, and you can view this in [the repo](https://git.yadunut.dev/yadunut/nix/src/branch/main/systems/x86_64-linux/nut-gc2/default.nix).
|
|
|
|
Now that the configuration is done, I have secrets(such as my k3s token) that I want to transfer over to the new system. To cleanly do this, I can generate the host keys on this system, use that to rekey the secret, transfer over the host keys and then erase them from my system when complete.
|
|
|
|
```sh
|
|
tmp_dir=$(mktemp -d)
|
|
mkdir -p "$tmp_dir/etc/ssh"
|
|
MACHINE_NAME="nut-gc2"
|
|
|
|
ssh-keygen -A -f "$tmp_dir"
|
|
for key in "$tmp_dir"/etc/ssh/ssh_host_*_key; do
|
|
[ -f "$key" ] || continue
|
|
echo "Updating comment on $key ..."
|
|
ssh-keygen -q -c -C "root@$MACHINE_NAME" -f "$key"
|
|
done
|
|
echo "ssh key: $(cat $tmp_dir/etc/ssh/ssh_host_ed25519_key.pub)"
|
|
```
|
|
|
|
Now with the ssh key generated, we can copy that over into our secrets.nix, and use age to rekey the secret
|
|
```sh
|
|
agenix --rekey
|
|
```
|
|
|
|
With this rekeyed, we can get to infecting the system
|
|
```sh
|
|
nix run github:nix-community/nixos-anywhere -- --generate-hardware-config nixos-generate-config ./systems/x86_64-linux/nut-gc2/hardware-configuration.nix --flake ".#${MACHINE_NAME}" --extra-files "${tmp_dir}" --print-build-logs --build-on remote "root@${MACHINE_IP}"
|
|
```
|
|
|
|
Now with the server infected, you'll need to update your `~/.ssh/known_hosts` and you can then ssh into your machine
|