Commit Graph

589 Commits

Author SHA1 Message Date
Kenji Berthold
5d0ca5aff8 Merge pull request 'pkgs/clan/lib: Fix clan template creation when already in a flake' (#4501) from kenji/ke-clan-flakes-create-existing-flake-fix into main
Reviewed-on: https://git.clan.lol/clan/clan-core/pulls/4501
2025-07-29 11:19:04 +00:00
DavHau
19a8101e98 Merge pull request 'pkgs/cli/create: Show less output by default' (#4499) from kenji/ke-create-show-less-output into main
Reviewed-on: https://git.clan.lol/clan/clan-core/pulls/4499
2025-07-29 10:45:16 +00:00
DavHau
cb89fb97f1 clan machines update: add --fetch-local feature
Motivation: updating a machine fails, if it depends on a private github repo, as the remote will likely not be authenticated.

This adds a new flag `--fetch-local` to `clan machines update` which fetches all flake inputs prior to building, then uploads them to the build-host.

This also adds a new error message, when flake inputs could not fetched, to hint the user to use `--fetch-local`
2025-07-28 17:01:42 +07:00
Johannes Kirschbauer
98a0b9600b api/writability: add docstring 2025-07-27 12:47:37 +02:00
Johannes Kirschbauer
abeb517a22 api/writability: add unit test 2025-07-27 00:03:05 +02:00
Johannes Kirschbauer
fbdbcfa6d5 InventoryStore: factor write into _write for actual disk interaction 2025-07-27 00:02:47 +02:00
Johannes Kirschbauer
303af9af6b api: init get_machine_writeability 2025-07-27 00:01:51 +02:00
Johannes Kirschbauer
414e412e7e persist/writeability: expose is writeable key helper 2025-07-27 00:01:36 +02:00
Johannes Kirschbauer
c2e84f11af persist/util: add field helper 2025-07-27 00:01:07 +02:00
Johannes Kirschbauer
bf2eb000d5 api/set_machine: add unit tests 2025-07-26 23:59:51 +02:00
a-kenji
0a43721a45 pkgs/clan/lib: Fix clan template creation when already in a flake
Fix clan template creation when already in a flake.
Currently we already fail with very clear and descriptive error when
trying to evaluate the template of the flake we are in:
```
Failed to select template 'flake-parts' from flake '/tmp/superclan' (via attribute path: /tmp/superclan#clanInternals.templates.clan."flake-parts")
```

This is undesired behavior.
When we are trying to create a clan with `clan flakes create`.
We can't rely on the fact that the flake we are currently in exports flake templates.

Now we *try* to evaluate the flake we are in upon creation.
If there are no clan templates available, we now will fall back to
builtin templates.

Closes: #4472
2025-07-26 15:21:44 +02:00
a-kenji
51eb7bd0b5 pkgs/cli/create: Show less output by default
This masks the output of `nix flake update`, since it is quite verbose
and takes attention away to usually more interesting and pressing
information.

Example:
```
unpacking 'https://git.clan.lol/clan/clan-core/archive/main.tar.gz' into the Git cache...
warning: creating lock file "/tmp/hoowowo/clan/flake.lock":
• Added input 'clan-core':
    '1d8ac7b1b5.tar.gz?narHash=sha256-eBxi0ZMwaALfMsP70N0FRMlOSq0qePv%2BjebVBHXlOqk%3D' (2025-07-25)
• Added input 'clan-core/data-mesher':
    '18dfd42bdb.tar.gz?narHash=sha256-jyoEbaXa8/MwVQ%2BPajUdT63y3gYhgD9o7snO/SLaikw%3D' (2025-07-21)
• Added input 'clan-core/data-mesher/flake-parts':
    follows 'clan-core/flake-parts'
• Added input 'clan-core/data-mesher/nixpkgs':
    follows 'clan-core/nixpkgs'
• Added input 'clan-core/data-mesher/treefmt-nix':
    follows 'clan-core/treefmt-nix'
• Added input 'clan-core/disko':
    'github:nix-community/disko/545aba02960caa78a31bd9a8709a0ad4b6320a5c?narHash=sha256-7lrVrE0jSvZHrxEzvnfHFE/Wkk9DDqb%2BmYCodI5uuB8%3D' (2025-07-21)
• Added input 'clan-core/disko/nixpkgs':
    follows 'clan-core/nixpkgs'
• Added input 'clan-core/flake-parts':
    'github:hercules-ci/flake-parts/644e0fc48951a860279da645ba77fe4a6e814c5e?narHash=sha256-TVcTNvOeWWk1DXljFxVRp%2BE0tzG1LhrVjOGGoMHuXio%3D' (2025-07-21)
• Added input 'clan-core/flake-parts/nixpkgs-lib':
    follows 'clan-core/nixpkgs'
• Added input 'clan-core/nix-darwin':
    'github:nix-darwin/nix-darwin/e04a388232d9a6ba56967ce5b53a8a6f713cdfcf?narHash=sha256-HsJM3XLa43WpG%2B665aGEh8iS8AfEwOIQWk3Mke3e7nk%3D' (2025-06-30)
• Added input 'clan-core/nix-darwin/nixpkgs':
    follows 'clan-core/nixpkgs'
• Added input 'clan-core/nix-select':
    '69d8bf5961.tar.gz?narHash=sha256-IVaoOGDIvAa/8I0sdiiZuKptDldrkDWUNf/%2BezIRhyc%3D' (2025-04-18)
• Added input 'clan-core/nixos-facter-modules':
    'github:nix-community/nixos-facter-modules/14df13c84552a7d1f33c1cd18336128fbc43f920?narHash=sha256-uP9Xxw5XcFwjX9lNoYRpybOnIIe1BHfZu5vJnnPg3Jc%3D' (2025-06-20)
• Added input 'clan-core/nixpkgs':
    'https://releases.nixos.org/nixpkgs/nixpkgs-25.11pre827262.be9e214982e2/nixexprs.tar.xz?narHash=sha256-lUi%2BsPH7Kuh9uP3PyfgbENcJGReUM8Ffk9GxGBFbSN8%3D' (1980-01-01)
• Added input 'clan-core/sops-nix':
    'github:Mic92/sops-nix/2c8def626f54708a9c38a5861866660395bb3461?narHash=sha256-GllP7cmQu7zLZTs9z0J2gIL42IZHa9CBEXwBY9szT0U%3D' (2025-07-15)
• Added input 'clan-core/sops-nix/nixpkgs':
    follows 'clan-core/nixpkgs'
• Added input 'clan-core/systems':
    'github:nix-systems/default/da67096a3b9bf56a91d16901293e51ba5b49a27e?narHash=sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768%3D' (2023-04-09)
• Added input 'clan-core/treefmt-nix':
    'github:numtide/treefmt-nix/421b56313c65a0815a52b424777f55acf0b56ddf?narHash=sha256-tzbhc4XttkyEhswByk5R38l%2BztN9UDbnj0cTcP6Hp9A%3D' (2025-07-20)
• Added input 'clan-core/treefmt-nix/nixpkgs':
    follows 'clan-core/nixpkgs'
• Added input 'flake-parts':
    'github:hercules-ci/flake-parts/644e0fc48951a860279da645ba77fe4a6e814c5e?narHash=sha256-TVcTNvOeWWk1DXljFxVRp%2BE0tzG1LhrVjOGGoMHuXio%3D' (2025-07-21)
• Added input 'flake-parts/nixpkgs-lib':
    follows 'clan-core/nixpkgs'
• Added input 'nixpkgs':
    follows 'clan-core/nixpkgs'
```

Those are 48 lines that seldom carry actual useful information.
This can be shown on running `clan flakes create` with the `--debug`
flag.

Closes: #4496
2025-07-26 14:11:56 +02:00
Kenji Berthold
1d8ac7b1b5 Merge pull request 'pkgs/cli/lib: Allow clan templates list to function outside a clan' (#4490) from kenji/ke-templates-list-without-clan into main
Reviewed-on: https://git.clan.lol/clan/clan-core/pulls/4490
Reviewed-by: hsjobeki <hsjobeki@gmail.com>
2025-07-25 14:32:15 +00:00
a-kenji
5b5f1975c5 pkgs/cli/lib: Allow clan templates list to function outside a clan
Allow `clan templates list` to function outside a clan.
Currently when bootstrapping a clan and trying to list the templates
it fails as follows:

```
Traceback (most recent call last):
  File "/nix/store/pkrsr8zr90bps1fwrl8n74zbb9g038b8-clan-cli/bin/.clan-wrapped", line 9, in <module>
    sys.exit(main())
             ~~~~^^
  File "/nix/store/pkrsr8zr90bps1fwrl8n74zbb9g038b8-clan-cli/lib/python3.13/site-packages/clan_cli/cli.py", line 516, in main
    args.func(args)
    ~~~~~~~~~^^^^^^
  File "/nix/store/pkrsr8zr90bps1fwrl8n74zbb9g038b8-clan-cli/lib/python3.13/site-packages/clan_cli/templates/list.py", line 11, in list_command
    templates = list_templates(args.flake)
  File "/nix/store/pkrsr8zr90bps1fwrl8n74zbb9g038b8-clan-cli/lib/python3.13/site-packages/clan_lib/templates/__init__.py", line 20, in list_templates
    custom_templates = flake.select("clanInternals.inventoryClass.templatesPerSource")
                       ^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'select'
```

With the change we get the following output:
```
Available 'clan' templates
├── <builtin>
│   ├── default: Initialize a new clan flake
│   ├── flake-parts: Flake-parts
│   └── minimal: for clans managed via (G)UI
Available 'disko' templates
├── <builtin>
│   └── single-disk: A simple ext4 disk with a single partition
Available 'machine' templates
├── <builtin>
│   ├── flash-installer: Initialize a new flash-installer machine
│   └── new-machine: Initialize a new machine
```

Allowing to check for available templates without needing to have a
clan, which improves the bootstrapping experience.
2025-07-25 16:14:43 +02:00
lassulus
1a5b77d47a refactor: generalize Tor support to SOCKS5 proxy in network module
- Replace Tor-specific implementation with generic SOCKS5 proxy support
- Change `tor_socks` boolean to `socks_port` and `socks_wrapper` parameters
- Move Tor functionality to clan_lib.network.tor submodule
- Add connection context managers to NetworkTechnologyBase
- Improve network abstraction with proper remote() and connection() methods
- Update all callers to use new SOCKS5 proxy interface
- Fix network ping command to properly handle connection contexts

This allows for more flexible proxy configurations beyond just Tor,
while maintaining backward compatibility for Tor usage.
2025-07-24 22:26:44 +02:00
lassulus
9e85c64139 clan-cli flake: show cache file location 2025-07-24 22:24:34 +02:00
Luis Hebendanz
9668c318dc Merge pull request 'fix flake select logging' (#4483) from Qubasa/clan-core:fix_flake_select_logging into main
Reviewed-on: https://git.clan.lol/clan/clan-core/pulls/4483
2025-07-24 11:44:21 +00:00
Qubasa
7146c97362 clan_lib: Fix flake.select logging, now we log the first time select queries a path for the first time, it doesn't matter if it is cached or not. 2025-07-24 18:27:40 +07:00
Qubasa
b74aa31b87 clan-lib: Fix missing logging for flake.select execution 2025-07-24 17:29:09 +07:00
Qubasa
94662b722d clan-lib: Remove injected "op_key" argument from all functions and do it over the threadcontext instead. Remove double threading in http server 2025-07-24 14:25:20 +07:00
hsjobeki
1e51439414 Merge pull request 'pytest: add simple clan_flake function' (#4453) from api-fixes into main
Reviewed-on: https://git.clan.lol/clan/clan-core/pulls/4453
2025-07-23 10:27:05 +00:00
Johannes Kirschbauer
29c764773f pytest: clan_flake allow usage of plain dicts 2025-07-23 12:15:54 +02:00
DavHau
c94330ee9c clan create: fix failure when path was single word
This should better be fixed with types. It should be possible to initialize a flake from a Path, making it very clear that a path `foo` is meant and not a remote flake called `foo`
2025-07-23 12:33:57 +07:00
Johannes Kirschbauer
1dbaff7b61 pytest: add simple clan_flake function
Takes a clan nix expression as a string
Is empty by default and does nothing
Expensive lockfile patching is done once per session
2025-07-22 23:24:40 +02:00
hsjobeki
41a0138c16 Merge pull request 'clan/create: api fixes and unit tests' (#4449) from api-fixes into main
Reviewed-on: https://git.clan.lol/clan/clan-core/pulls/4449
2025-07-22 17:28:37 +00:00
Johannes Kirschbauer
f1be729206 create/clan: unit tests init 2025-07-22 19:17:41 +02:00
lassulus
d5e0f7e505 Merge pull request 'fix: handle arbitrary store paths references in flake cache' (#4441) from fix-flake-caching into main
Reviewed-on: https://git.clan.lol/clan/clan-core/pulls/4441
2025-07-22 16:18:15 +00:00
Johannes Kirschbauer
b9e5cf1220 clan/create: use post_processing hook 2025-07-22 18:14:56 +02:00
Johannes Kirschbauer
f4eb59c373 create/clan: add validation to create arguments 2025-07-22 18:13:56 +02:00
Johannes Kirschbauer
06257d044a test_create: fix duplicated variables 2025-07-22 18:12:51 +02:00
Johannes Kirschbauer
34ca7a4a7b create/clan: add abstraction for initial meta
Directly passing through persisted data is bad
2025-07-22 18:12:17 +02:00
Johannes Kirschbauer
a34ec8ed22 templates: add post_process hook for tests or other extensions 2025-07-22 18:07:26 +02:00
Johannes Kirschbauer
4597b207e7 pytest: fixtures offline_flake_hook init 2025-07-22 18:06:47 +02:00
Johannes Kirschbauer
9257cb02ee validator: hostname init 2025-07-22 18:06:27 +02:00
Jörg Thalheim
bd1451ce18 fix: handle arbitrary store paths references in flake cache
Previously, paths like /nix/store/hash-file.nix:123 were incorrectly
treated as pure store paths and wrapped in {"outPath": ...}, breaking
the cache. This fix:

- Adds helper functions to properly detect and handle store references
- Distinguishes between pure store paths and paths with metadata (line numbers)
- Supports multiple store references in a single string
- Handles custom NIX_STORE_DIR correctly
- Ensures existence checks work for all store references

Also fixes test_cache_gc to delete NIX_REMOTE for proper local store testing.
2025-07-22 17:13:04 +02:00
Johannes Kirschbauer
0043870882 merge_objects: add unit tests 2025-07-22 15:01:00 +02:00
Johannes Kirschbauer
0ea42ae541 store: move merge_objects into persistence helpers 2025-07-22 15:01:00 +02:00
a-kenji
3a2be243c0 pkgs/cli: Add machine to output while uploading sources
Add the `machine` to it's output while uploading sources.
2025-07-21 22:15:43 +02:00
Qubasa
4055508588 clan-lib: Add object_name to ClassSource and don't override __repr__ from NetworkTechnologyBase instead overwrite it in ClassSource 2025-07-21 14:25:01 +07:00
Qubasa
ff65dfc883 clanServices: change tor service to have "client" and "server" roles instead of just "default"
also improve error message when user forgot to update machine in clan
networking command
2025-07-21 14:25:01 +07:00
Qubasa
1f5ef04a61 clan-lib: Fix network.py missing vars generation and use import_with_source for better trace ability 2025-07-21 12:40:49 +07:00
Qubasa
89f0e90910 clan-lib: Init import_utils to add debug information to dynamically imported modules 2025-07-21 12:40:49 +07:00
Qubasa
137aa71529 clan-lib: Fix is_running of tor.py 2025-07-21 12:40:49 +07:00
lassulus
60768cc537 Add networking module
This adds a (for now hidden) clan network command that exposes list,
ping, overview subcommands to get informations about configured
networks.
ClanServices can now use the exports to define network specific
information.

This is not the complete feature yet, as we are lacking more tests and
documentation, but merging this now makes it easier to iterate.
2025-07-17 15:23:08 +02:00
Mic92
d036f98cd4 Merge pull request 'clan-cli: Move flash.py to clan_lib/flash' (#4374) from Qubasa/clan-core:move_flash into main
Reviewed-on: https://git.clan.lol/clan/clan-core/pulls/4374
2025-07-16 09:30:04 +00:00
Qubasa
a90cb56886 clan-cli: Move flash.py to clan_lib/flash 2025-07-16 15:29:18 +07:00
Qubasa
cfba97eee5 clan-cli: Reference HostKeyCheck literal instead of duplicating the list everywhere 2025-07-16 13:12:48 +07:00
Qubasa
fb4ccd1f63 clan-lib: Remove duplicate fields from installOptions and instead use them from Remote 2025-07-16 13:05:05 +07:00
Qubasa
2c4e688b0a clan-lib: Change BuildOn enum to Literal type. Literals can be translated better to typescript 2025-07-16 12:48:04 +07:00
Qubasa
f8a0943fbd clan-cli: Fix incorrect ipv6 check in check_machine_ssh_reachable 2025-07-16 12:34:30 +07:00