Commit Graph

64 Commits

Author SHA1 Message Date
Jörg Thalheim
64346bacc8 only compute final_script and validation hashes once 2025-04-04 18:31:12 +02:00
Jonathan Thiessen
eb35e6ea21 Make Generator's validation dynamic
* Switch `Generator`'s `validation` from a regular property to
  an `@property` annotated method backed by `Machine`'s `eval_nix()`.
* Ensure that `Machine`'s flake cache is flushed after each
  effectful generator execution (rather than only after all
  generators have been executed).
2025-03-30 04:33:30 +00:00
Jörg Thalheim
93cbe62765 always resolve symlinks for TemporaryDirectory
On macOS mktemp returns a temporary directory in a symlink.
Nix has a bug where it won't accept path:// located in a symlink.
This avoid this issue by always resolving symlinks as returned by
TemporaryDirectory.
2025-03-19 16:47:18 +01:00
DavHau
9797ef792a vars+facts: use bwrap only if supported 2025-03-09 13:52:15 +07:00
a-kenji
ce148c23a5 Fix various typos 2025-01-22 13:19:28 +01:00
Johannes Kirschbauer
c6fe4f2625 API/vars: use string based interfaces to get and set vars to avoid state mutations 2025-01-10 12:06:01 +00:00
Jörg Thalheim
bc48ec738b vars: commit validation hashes 2024-12-25 21:02:52 +01:00
Jörg Thalheim
14fdf2debd vars/fact: isolate secret generation better from the system 2024-12-25 19:21:51 +00:00
Michael Hoang
34b2b40222 vars: fix running generators on macOS 2024-12-19 00:16:22 +11:00
lassulus
74cec1d102 cli vars generate: quote generate name 2024-12-15 13:23:11 +01:00
lassulus
e0f95a7a16 cli vars generate: mount test_store into bwrap 2024-12-14 17:05:27 +01:00
lassulus
c888f01823 vars: eval finalScript lazy 2024-12-14 13:38:51 +01:00
Qubasa
ec54a6a978 clan-cli: Replace log.info to machine.info if applicable 2024-12-12 15:36:17 +01:00
lassulus
91118745de vars: remove intermediate classes 2024-12-10 14:04:31 +01:00
lassulus
3337d7bce6 vars: move ensure_consistent_state into health_check, move into store classes 2024-12-10 11:54:52 +00:00
Qubasa
3cc7511764 clan-cli: Replace HostGroup and MachineGroup with generic AsyncRuntime class. Propagate cmd prefix over thread local. Close threads on CTRL+C 2024-12-09 18:07:23 +01:00
lassulus
ceaeb2f1f6 vars migration: raise error on incomplete migration, commit migrated files 2024-12-06 11:25:17 +01:00
clan-bot
1511b14b3c Merge pull request 'fix vars migration prompts. add secretsForUsers to vars interface and implement that for pass' (#2551) from lassulus/clan-core:vars-stuff into main 2024-12-04 09:03:24 +00:00
DavHau
f5226ec7ff vars: Improve logging for migration 2024-12-04 12:42:03 +07:00
lassulus
c6a713d393 cli vars generate: log in global context what is global 2024-12-03 22:29:25 +01:00
lassulus
2916798e84 vars generate: show prompts only if not migrating 2024-12-03 22:25:16 +01:00
DavHau
681c671391 vars/migration: remove useless check 2024-11-29 17:23:31 +07:00
DavHau
66ce80c096 vars: rename: invalidation -> validation 2024-11-29 17:23:31 +07:00
DavHau
03bcf6bba3 vars: generate docs for cli and module 2024-11-29 17:23:31 +07:00
Qubasa
d2719f3179 clan-cli: cmd.run now has its options extracted to a dataclass 2024-11-28 15:26:37 +01:00
Jörg Thalheim
45dfbf54db vars: make interface more type-safe 2024-11-26 17:08:26 +01:00
DavHau
173436632d vars: fix migration - secrets end up in public store 2024-11-26 17:02:11 +07:00
Qubasa
41a84f5970 docs: Fix nix flake check problem with diskId 2024-11-25 18:39:16 +01:00
Qubasa
8866a85765 clan-cli: Refactor ssh part 2, Refactor custom_logger 2024-11-22 22:08:50 +01:00
DavHau
3f62e143ec vars: implement invalidation mechanism
This adds options `invalidationData` to generators.

`invalidationData` can be used by an author of a generator to signal if a re-generation is required after updating the logic.

Whenever a generator with invalidation data is executed, a hash of that data is stored by the respective public and/or secret backends.

The stored hashes will be checked on future deployments, and a re-generation is triggered whenever a hash doesn't match what's defined in nix.
2024-11-20 16:27:22 +07:00
Jörg Thalheim
a4e03a85eb vars: don't print stack trace if generator fails 2024-11-19 09:46:14 +00:00
Jörg Thalheim
9c6e04fa3f vars: introduce ensure_machine_has_access method for sops
this should help avoiding overriding existing shared secrets by not
triggering vars regeneration if a machine has no access.

wip
2024-11-19 09:46:14 +00:00
Jörg Thalheim
c98055c781 vars: introduce ensure_machine_has_access method for sops
this should help avoiding overriding existing shared secrets by not
triggering vars regeneration if a machine has no access.

wip
2024-11-14 15:37:55 +00:00
DavHau
54b8f5904e vars: allow re-encrypting secrets when recipient keys were added.
When the users of a secret change, when for example a new admin user is added, an error will be thrown when generating vars, prompting the user to pass --fix to re-encrypt the secrets
2024-11-13 18:49:30 +07:00
Jörg Thalheim
34410c6e17 vars: update message if vars are up-to-date 2024-11-08 15:43:10 +01:00
DavHau
7208c6dc82 vars: fix shared dependency was not resolved correctly 2024-10-23 20:43:33 +07:00
DavHau
cc43a46c53 vars: show full var_id when prompting 2024-10-16 13:13:41 +07:00
Jörg Thalheim
36573589ef {vars,facts}/generate: ensure that args.flake is passed 2024-10-08 17:51:55 +00:00
Jörg Thalheim
237327ead3 start to name temporary directories in more places 2024-10-01 18:57:59 +02:00
Jörg Thalheim
716c6c9139 generate: don't set type=bool in argparse.BooleanOptionalAction 2024-10-01 18:48:54 +02:00
DavHau
501ade7de7 vars: implement migration
Migrating generated files from the facts subsystem to the vars subsystem is now possible.

HowTo:
1. declare `clan.core.vars.generators.<generator>.migrateFact = my_service` where `my_service` refers to a service from `clan.core.facts.services`
2. run `clan vers generate your_machine` or `clan machines update your_machine`

Vars will only be migrated for a generator if:
1. The facts service specified via `migrateFact` does exist
2. None of the vars to generate exist yet
3. All public var names exist in the public facts store
4. All secret var names exist in the secret fact store

If the migration is deemed possible, the generator script will not be executed. Instead the files from the public or secret facts store are read and stored into the corresponding vars store
2024-09-19 17:57:03 +02:00
DavHau
89d39186ee vars/generate: improve output when vars are updated
fixes #2076

- print old and new value if possible
- also inform the user if something hasn't changed
2024-09-18 16:57:01 +02:00
DavHau
cf47c1d51a vars: generate proper commit messages
fixes #2126
2024-09-17 20:22:18 +02:00
DavHau
b3dd36bdb6 vars: improve generator pipeline
Compute the whole closure of to-be-executed generators upfront before executing anything

Properly compute closures for the 4 different scenarios:

1. full_closure: run all generators for a selected machine in topological order
2. all_missing_closure: run just the missing generators including their dependents
3. requested_closure: run only a selected list of generators including their missing dependencies and their dependents
4. minimal_closure: Run just enough to ensure that the list of selected generators are in a consistent state. Don't execute anything if nothing is missing.
2024-09-06 21:32:17 +02:00
DavHau
f5e65b5d0e vars: refactor - ask prompts before running any generators 2024-09-06 16:08:46 +02:00
DavHau
f161c7ca58 vars: improve generator execution pipeline
- ensure all dependents are re-generated as well
- refactor: separate out computation of generator update closure
2024-09-06 14:30:49 +02:00
DavHau
1254b80a52 vars: add api endpoint set_prompts 2024-09-04 18:06:11 +02:00
clan-bot
87c5ded6a2 Merge pull request 'apply TRY lint' (#2035) from joerg-ci into main 2024-09-04 13:20:11 +00:00
DavHau
570aadf065 vars: set vars via cli; improve getting vars via cli; 2024-09-04 14:48:42 +02:00
Jörg Thalheim
403b9cf2cc apply TRY lint 2024-09-03 18:13:46 +02:00