docs: Add debugging.md and repo-layout.md guides

CONTRIBUTING.md

@@ -1,23 +1,3 @@
 # Contributing to Clan
 
-## Live-reloading documentation
-
-Enter the `docs` directory:
-
-```shell-session
-cd docs
-```
-
-Enter the development shell or enable `direnv`:
-
-```shell-session
-direnv allow
-```
-
-Run a local server:
-
-```shell-session
-mkdocs serve
-```
-
-Open http://localhost:8000/ in your browser.
+Go to the Contributing guide at https://docs.clan.lol/manual/contribute/

docs/CONTRIBUTING.md

@@ -24,7 +24,7 @@ Let's get your development environment up and running:
 
       - To automatically setup a devshell on entering the directory
         ```bash
-        nix profile install nixpkgs#nix-direnv-flakes
+        nix profile install nixpkgs#nix-direnv-flakes nixpkgs#direnv
         ```
 
 3. **Add direnv to your shell**:
@@ -36,6 +36,9 @@ Let's get your development environment up and running:
       echo 'eval "$(direnv hook zsh)"' >> ~/.zshrc && echo 'eval "$(direnv hook bash)"' >> ~/.bashrc && eval "$SHELL"
       ```
 
+3. **Allow the devshell**
+      - Go to `clan-core/pkgs/clan-cli` and do a `direnv allow` to setup the necessary development environment to execute the `clan` command
+
 4. **Create a Gitea Account**:
       - Register an account on https://git.clan.lol
       - Fork the [clan-core](https://git.clan.lol/clan/clan-core) repository
@@ -91,109 +94,11 @@ Let's get your development environment up and running:
       merge-after-ci --reviewers Mic92 Lassulus Qubasa
       ```
 
-# Debugging
 
-Here are some methods for debugging and testing the clan-cli:
 
-## See all possible packages and tests
+### Whats Next?
 
-To quickly show all possible packages and tests execute:
-
-```bash
-nix flake show --system no-eval
-```
-
-Under `checks` you will find all tests that are executed in our CI. Under `packages` you find all our projects.
-
-```
-git+file:///home/lhebendanz/Projects/clan-core
-├───apps
-│   └───x86_64-linux
-│       ├───install-vm: app
-│       └───install-vm-nogui: app
-├───checks
-│   └───x86_64-linux
-│       ├───borgbackup omitted (use '--all-systems' to show)
-│       ├───check-for-breakpoints omitted (use '--all-systems' to show)
-│       ├───clan-dep-age omitted (use '--all-systems' to show)
-│       ├───clan-dep-bash omitted (use '--all-systems' to show)
-│       ├───clan-dep-e2fsprogs omitted (use '--all-systems' to show)
-│       ├───clan-dep-fakeroot omitted (use '--all-systems' to show)
-│       ├───clan-dep-git omitted (use '--all-systems' to show)
-│       ├───clan-dep-nix omitted (use '--all-systems' to show)
-│       ├───clan-dep-openssh omitted (use '--all-systems' to show)
-│       ├───"clan-dep-python3.11-mypy" omitted (use '--all-systems' to show)
-├───packages
-│   └───x86_64-linux
-│       ├───clan-cli omitted (use '--all-systems' to show)
-│       ├───clan-cli-docs omitted (use '--all-systems' to show)
-│       ├───clan-ts-api omitted (use '--all-systems' to show)
-│       ├───clan-app omitted (use '--all-systems' to show)
-│       ├───default omitted (use '--all-systems' to show)
-│       ├───deploy-docs omitted (use '--all-systems' to show)
-│       ├───docs omitted (use '--all-systems' to show)
-│       ├───editor omitted (use '--all-systems' to show)
-└───templates
-    ├───default: template: Initialize a new clan flake
-    └───new-clan: template: Initialize a new clan flake
-```
-
-You can execute every test separately by following the tree path `nix build .#checks.x86_64-linux.clan-pytest` for example.
-
-## Test Locally in Devshell with Breakpoints
-
-To test the cli locally in a development environment and set breakpoints for debugging, follow these steps:
-
-1. Run the following command to execute your tests and allow for debugging with breakpoints:
-   ```bash
-   cd ./pkgs/clan-cli
-   pytest -n0 -s --maxfail=1 ./tests/test_nameofthetest.py
-   ```
-   You can place `breakpoint()` in your Python code where you want to trigger a breakpoint for debugging.
-
-## Test Locally in a Nix Sandbox
-
-To run tests in a Nix sandbox, you have two options depending on whether your test functions have been marked as impure or not:
-
-### Running Tests Marked as Impure
-
-If your test functions need to execute `nix build` and have been marked as impure because you can't execute `nix build` inside a Nix sandbox, use the following command:
-
-```bash
-nix run .#impure-checks
-```
-
-This command will run the impure test functions.
-
-### Running Pure Tests
-
-For test functions that have not been marked as impure and don't require executing `nix build`, you can use the following command:
-
-```bash
-nix build .#checks.x86_64-linux.clan-pytest --rebuild
-```
-
-This command will run all pure test functions.
-
-### Inspecting the Nix Sandbox
-
-If you need to inspect the Nix sandbox while running tests, follow these steps:
-
-1. Insert an endless sleep into your test code where you want to pause the execution. For example:
-
-   ```python
-   import time
-   time.sleep(3600)  # Sleep for one hour
-   ```
-
-2. Use `cntr` and `psgrep` to attach to the Nix sandbox. This allows you to interactively debug your code while it's paused. For example:
-
-   ```bash
-   psgrep -a -x your_python_process_name
-   cntr attach <container id, container name or process id>
-   ```
-
-Or you can also use the [nix breakpoint hook](https://nixos.org/manual/nixpkgs/stable/#breakpointhook)
+Please look into the [debugging](./debugging.md) guide next!
 
 
 # Standards

docs/mkdocs.yml

@@ -62,6 +62,8 @@ nav:
           - Modules: clanmodules/index.md
           - Disk Templates: manual/disk-templates.md
       - Contribute: manual/contribute.md
+      - Debugging: manual/debugging.md
+      - Repo Layout: manual/repo-layout.md
   # - Concepts:
   #     - Overview: concepts/index.md
   - Reference:

docs/site/getting-started/disk-encryption.md

@@ -14,7 +14,7 @@ lsblk --output NAME,ID-LINK,FSTYPE,SIZE,MOUNTPOINT
 
 === "**Single Disk**"
     Below is the configuration for `disko.nix`
-    ```nix hl_lines="17 48"
+    ```nix hl_lines="13 53"
       --8<-- "docs/code-examples/disko-single-disk.nix"
     ```
 
@@ -22,7 +22,7 @@ lsblk --output NAME,ID-LINK,FSTYPE,SIZE,MOUNTPOINT
 
 === "**Raid 1**"
     Below is the configuration for `disko.nix`
-    ```nix hl_lines="17 48 49"
+    ```nix hl_lines="13 53 54"
       --8<-- "docs/code-examples/disko-raid.nix"
     ```
 
@@ -172,7 +172,7 @@ ssh -p 7172 root@192.168.178.141
 2. Run the `systemd-tty-ask-password-agent` utility to query a password:
 
 ```bash
-systemd-tty-ask-password-agent --query
+systemd-tty-ask-password-agent
 ```
 
 After completing these steps, your NixOS should be successfully installed and ready for use.

docs/site/getting-started/index.md

@@ -1,6 +1,9 @@
 # :material-clock-fast: Getting Started
 
-Create your own clan with these initial steps and manage a fleet of machines with one single testable git repository!
+Ready to create your own clan and manage a fleet of machines? Follow these simple steps to get started.
+
+By the end of this guide, you'll have a fresh NixOS configuration ready to push to one or more machines. You'll create a new git repository and a flake, and all you need is at least one machine to push to. This is the easiest way to begin, and we recommend you to copy your existing configuration into this new setup!
+
 
 ### Prerequisites
 

docs/site/manual/debugging.md

@@ -0,0 +1,157 @@
+# Debugging
+
+Here are some methods for debugging and testing the clan-cli
+
+## Using a Development Branch
+
+To streamline your development process, I suggest not installing `clan-cli`. Instead, clone the `clan-core` repository and add `clan-core/pkgs/clan-cli/bin` to your PATH to use the checked-out version directly.
+
+!!! Note
+    After cloning, navigate to `clan-core/pkgs/clan-cli` and execute `direnv allow` to activate the devshell. This will set up a symlink to nixpkgs at a specific location; without it, `clan-cli` won't function correctly.
+
+With this setup, you can easily use [breakpoint()](https://docs.python.org/3/library/pdb.html) to inspect the application's internal state as needed.
+
+This approach is feasible because `clan-cli` only requires a Python interpreter and has no other dependencies.
+
+```nix
+pkgs.mkShell {
+    packages = [
+        pkgs.python3
+    ];
+    shellHook = ''
+            export GIT_ROOT="$(git rev-parse --show-toplevel)"
+            export PATH=$PATH:~/Projects/clan-core/pkgs/clan-cli/bin
+    '';
+}
+```
+
+## The Debug Flag
+
+You can enhance your debugging process with the `--debug` flag in the `clan` command. When you add this flag to any command, it displays all subprocess commands initiated by `clan` in a readable format, along with the source code position that triggered them. This feature makes it easier to understand and trace what's happening under the hood.
+
+```bash
+$ clan machines list --debug                                                                          1 ↵
+Debug log activated
+nix \
+    --extra-experimental-features 'nix-command flakes' \
+    eval \
+    --show-trace --json \
+    --print-build-logs '/home/qubasa/Projects/qubasas-clan#clanInternals.machines.x86_64-linux' \
+    --apply builtins.attrNames \
+    --json
+Caller: ~/Projects/clan-core/pkgs/clan-cli/clan_cli/machines/list.py:96::list_nixos_machines
+
+warning: Git tree '/home/qubasa/Projects/qubasas-clan' is dirty
+demo
+gchq-local
+wintux
+
+```
+
+## VSCode
+
+If you're using VSCode, it has a handy feature that makes paths to source code files clickable in the integrated terminal. Combined with the previously mentioned techniques, this allows you to open a Clan in VSCode, execute a command like `clan machines list --debug`, and receive a printed path to the code that initiates the subprocess. With the `Ctrl` key (or `Cmd` on macOS) and a mouse click, you can jump directly to the corresponding line in the code file and add a `breakpoint()` function to it, to inspect the internal state.
+
+## See all possible packages and tests
+
+To quickly show all possible packages and tests execute:
+
+```bash
+nix flake show
+```
+
+Under `checks` you will find all tests that are executed in our CI. Under `packages` you find all our projects.
+
+```
+git+file:///home/lhebendanz/Projects/clan-core
+├───apps
+│   └───x86_64-linux
+│       ├───install-vm: app
+│       └───install-vm-nogui: app
+├───checks
+│   └───x86_64-linux
+│       ├───borgbackup omitted (use '--all-systems' to show)
+│       ├───check-for-breakpoints omitted (use '--all-systems' to show)
+│       ├───clan-dep-age omitted (use '--all-systems' to show)
+│       ├───clan-dep-bash omitted (use '--all-systems' to show)
+│       ├───clan-dep-e2fsprogs omitted (use '--all-systems' to show)
+│       ├───clan-dep-fakeroot omitted (use '--all-systems' to show)
+│       ├───clan-dep-git omitted (use '--all-systems' to show)
+│       ├───clan-dep-nix omitted (use '--all-systems' to show)
+│       ├───clan-dep-openssh omitted (use '--all-systems' to show)
+│       ├───"clan-dep-python3.11-mypy" omitted (use '--all-systems' to show)
+├───packages
+│   └───x86_64-linux
+│       ├───clan-cli omitted (use '--all-systems' to show)
+│       ├───clan-cli-docs omitted (use '--all-systems' to show)
+│       ├───clan-ts-api omitted (use '--all-systems' to show)
+│       ├───clan-app omitted (use '--all-systems' to show)
+│       ├───default omitted (use '--all-systems' to show)
+│       ├───deploy-docs omitted (use '--all-systems' to show)
+│       ├───docs omitted (use '--all-systems' to show)
+│       ├───editor omitted (use '--all-systems' to show)
+└───templates
+    ├───default: template: Initialize a new clan flake
+    └───new-clan: template: Initialize a new clan flake
+```
+
+You can execute every test separately by following the tree path `nix run .#checks.x86_64-linux.clan-pytest -L` for example.
+
+## Test Locally in Devshell with Breakpoints
+
+To test the cli locally in a development environment and set breakpoints for debugging, follow these steps:
+
+1. Run the following command to execute your tests and allow for debugging with breakpoints:
+   ```bash
+   cd ./pkgs/clan-cli
+   pytest -n0 -s --maxfail=1 ./tests/test_nameofthetest.py
+   ```
+   You can place `breakpoint()` in your Python code where you want to trigger a breakpoint for debugging.
+
+## Test Locally in a Nix Sandbox
+
+To run tests in a Nix sandbox, you have two options depending on whether your test functions have been marked as impure or not:
+
+### Running Tests Marked as Impure
+
+If your test functions need to execute `nix build` and have been marked as impure because you can't execute `nix build` inside a Nix sandbox, use the following command:
+
+```bash
+nix run .#impure-checks -L
+```
+
+This command will run the impure test functions.
+
+### Running Pure Tests
+
+For test functions that have not been marked as impure and don't require executing `nix build`, you can use the following command:
+
+```bash
+nix build .#checks.x86_64-linux.clan-pytest --rebuild
+```
+
+This command will run all pure test functions.
+
+### Inspecting the Nix Sandbox
+
+If you need to inspect the Nix sandbox while running tests, follow these steps:
+
+1. Insert an endless sleep into your test code where you want to pause the execution. For example:
+
+   ```python
+   import time
+   time.sleep(3600)  # Sleep for one hour
+   ```
+
+2. Use `cntr` and `psgrep` to attach to the Nix sandbox. This allows you to interactively debug your code while it's paused. For example:
+
+   ```bash
+   psgrep <your_python_process_name>
+   cntr attach <container id, container name or process id>
+   ```
+
+Or you can also use the [nix breakpoint hook](https://nixos.org/manual/nixpkgs/stable/#breakpointhook)
+
+### What's next?
+
+Please look into the [repo layout](./repo-layout.md) guide next!

docs/site/manual/repo-layout.md

@@ -0,0 +1,60 @@
+# Repo Layout Guide
+
+This guide will help you navigate the codebase and locate key files:
+
+```bash
+$ tree -L 1
+.
+├── checks          # Contains NixOS and VM tests
+├── clanModules     # Clan modules available for end-user import
+├── docs            # Source files for docs.clan.lol, generated with MkDocs
+├── flakeModules
+├── lib             # User-exposed Clan Nix functions like buildClan and inventory
+├── machines
+├── nixosModules    # Internal Clan Nix functions, e.g., clanCore
+├── pkgs            # Clan applications and packaged dependencies
+├── formatter.nix   # Configuration for nix-treefmt, manages `nix fmt`
+├── scripts
+├── sops
+├── templates       # Template files for creating a new Clan
+└── vars
+```
+
+## Getting Started with Infrastructure
+
+To dive into infrastructure, check out our clan infra repo: [clan-infra](https://git.clan.lol/clan/clan-infra). Please provide us with your public SOPS key so we can add you as an admin.
+
+## Related Projects
+
+- **Data Mesher**: [dm](https://git.clan.lol/clan/dm)
+- **Nixos Facter**: [nixos-facter](https://github.com/nix-community/nixos-facter)
+- **Nixos Anywhere**: [nixos-anywhere](https://github.com/nix-community/nixos-anywhere)
+- **Disko**: [disko](https://github.com/nix-community/disko)
+
+## Fixing Bugs or Adding Features in Clan-CLI
+
+If you have a bug fix or feature that involves a related project, clone the relevant repository and replace its invocation in your local setup.
+
+For instance, if you need to update `nixos-anywhere` in clan-cli, find its usage:
+
+```python
+run(
+    nix_shell(
+        ["nixpkgs#nixos-anywhere"],
+        cmd,
+    ),
+    RunOpts(log=Log.BOTH, prefix=machine.name, needs_user_terminal=True),
+)
+```
+
+You can replace `"nixpkgs#nixos-anywhere"` with your local path:
+
+```python
+run(
+    nix_shell(
+        ["<path_to_local_src>#nixos-anywhere"],
+        cmd,
+    ),
+    RunOpts(log=Log.BOTH, prefix=machine.name, needs_user_terminal=True),
+)
+```

pkgs/clan-cli/clan_cli/machines/list.py

@@ -92,6 +92,7 @@ def list_nixos_machines(flake_url: str | Path) -> list[str]:
             "--json",
         ]
     )
+
     proc = run_no_stdout(cmd)
 
     try: