Commit Graph

2373 Commits

Author SHA1 Message Date
Lourenço Vales
ae68dd79cb lang: core: iter: Add a range function
This commit implements a range function that mimicks python's range
built-in by having a start, stop, and range argument. There's also
a few examples and tests to mimick Python's examples to guarantee
we're consistent with their behaviour.
2025-04-22 02:37:35 -04:00
James Shubin
de970ee557 engine: resources: Add symlink param to the file res
This adds initial symlink support to the file resource, and while it is
hopefully correct, there are always sneaky edge cases around symlinks
and security, so review and tests are highly encouraged!
2025-04-22 02:21:58 -04:00
James Shubin
60a3d7c65e lang: interfaces: Add more information about graph semantics
Sam thoughts.
2025-04-19 13:02:51 -04:00
James Shubin
9c1c587f7b lang: parse, core: world: Add a collect package
Some checks failed
/ Test (basic) on ubuntu-latest with golang 1.23 (push) Has been cancelled
/ Test (race) on ubuntu-latest with golang 1.23 (push) Has been cancelled
/ Test (shell) on ubuntu-latest with golang 1.23 (push) Has been cancelled
This lets us look at the available resource data for collection, and to
filter it so we can decide what we want to collect on our machine.

Other types of collect functions could be added in the future.
2025-04-05 17:00:53 -04:00
James Shubin
af04d364d0 lang: core: fmt: Make printf handle more cases
Until we make a clean determination about what this should print, this
should handle things for now.
2025-04-05 16:14:11 -04:00
James Shubin
748f05732a engine, etcd: Watch on star pattern for all hostnames
We forgot to watch on star hostname matches.
2025-04-05 15:45:44 -04:00
James Shubin
148bd50e9f engine, etcd: Prevent engine thrashing
These two small bugs would allow thrashing to occur since we'd
constantly delete and re-add exports, and constantly think that a noop
etcd operation made a change.
2025-04-05 15:28:54 -04:00
James Shubin
6c1c08ceda engine: resources: Test to make sure metaparams are preserved
We should ensure these get preserved across encoding/decoding. We rely
on this behaviour.
2025-04-05 12:45:23 -04:00
James Shubin
045b29291e engine, lang: Modern exported resources
I've been waiting to write this patch for a long time. I firmly believe
that the idea of "exported resources" was truly a brilliant one, but
which was never even properly understood by its original inventors! This
patch set aims to show how it should have been done.

The main differences are:

* Real-time modelling, since "once per run" makes no sense.
* Filter with code/functions not language syntax.
* Directed exporting to limit the intended recipients.

The next step is to add more "World" reading and filtering functions to
make it easy and expressive to make your selection of resources to
collect!
2025-04-05 12:45:23 -04:00
James Shubin
955112f64f engine: Let others use the ResUID struct
It's a useful key in maps.
2025-04-05 12:45:23 -04:00
James Shubin
7f341cee84 engine: resources: Improving logging even more
Messages should happen after the event on success. The error scenario
has its own pathway to report.
2025-04-05 12:45:23 -04:00
James Shubin
f71e623931 engine: resources: Print a message on empty file creation
We don't see this event happening which is confusing. There might be
other cases we didn't handle cleanly.
2025-04-05 12:45:23 -04:00
James Shubin
8ff187b4e9 lang: core, funcs: Rename things for consistency
Seems we had different patterns going on. This makes those all
consistent now.
2025-04-05 12:45:23 -04:00
James Shubin
30aca74089 engine, yamlgraph: Disable the old exported resources stuff
These were really just stubs so that I could prove out the reactive
model very early, and I don't think they're really used anywhere.

I'm also not really using the yamlgraph frontend. If someone wants any
of that code, step up, or it will rot even more.
2025-04-05 12:45:23 -04:00
James Shubin
3dfca97f86 engine: Add a method to determine if a res kind is valid 2025-04-05 12:45:23 -04:00
James Shubin
0d4c6e272d lib: Add timing for topological sort
At least for consistency with everyone else...
2025-04-05 12:45:23 -04:00
James Shubin
fce250b8af cli, etcd, lib: Fixup golint issues with SSH
This stuff is arbitrary and stupid.
2025-04-05 12:45:23 -04:00
James Shubin
f6a8404f9f modules: virtualization: Qcow2 should be the default
Snapshots and so much more is only possible with qcow2. A long time ago
it had performance issues, but things seem okay now.
2025-03-28 04:44:56 -04:00
Karpfen
c50a578426 git: Add vendor/ to gitignore 2025-03-22 14:56:16 -04:00
Karpfen
7e8ced534f misc: Use /usr/bin/env for a more generic shebang
Use path based SHELL in Makefiles. It was suggested that this is a
better solution for make for cases when there is no /usr/bin/env.

See: https://github.com/purpleidea/mgmt/pull/694#discussion_r1015596204
2025-03-22 14:53:21 -04:00
Lourenço Vales
f2d9219218 lang: core: os: Add is_virtual function
This is a basic implementation of a detection method for whether mgmt is
running in a virtualized environment. We achieve this by doing two types
of checks: on one hand, we check if the CPU flags can confirm the
presence of a virtualized env; on the other, we check if the presence
of known files related with DMI (and their contents) can confirm whether
 we're inside a virt env. Either of these situations will cause the
function to return true, with the default case being false. All of these
checks are relatively naive and can be improved by looking at the main
inspiration for this implementation, which was systemd's own check for
the presence of virtualization.
2025-03-21 14:18:55 -04:00
James Shubin
f269096eb9 cli, etcd, lib: Remove the etcd client from main
We are slowly getting rid of more cruft and abstracting it nicely. More
to go!
2025-03-19 06:01:42 -04:00
James Shubin
5665259784 cli, engine, etcd, lib: Move the hostname value to the API
Every world implementation needs a unique UUID, might as well move this
to the API.
2025-03-19 05:41:04 -04:00
James Shubin
02fca6409a cli, etcd, lib: Add an etcd client over ssh world backend
This provides a new kind of "world" backend, one that runs etcd over an
SSH connection. This is useful for situations where you want to run an
etcd cluster somewhere for clients across the net, but where you don't
want to expose the ports publicly.

If SSH authentication is setup correctly (using public keys) this will
tunnel over SSH for etcd to connect.

This patch does not yet support deploys over SSH, but that should be
fixed in the future as the world code gets cleaned up more.
2025-03-19 05:33:07 -04:00
James Shubin
a7a5237b07 cli, engine, etcd, lib: Pass in init args
Improve the API and make it more general.
2025-03-18 04:54:13 -04:00
James Shubin
7ad54fe3e8 cli, engine, etcd, lib: Split out the deployer into world
This should hopefully make the refactor into a clean world API a bit
better. Still more to do though!
2025-03-18 04:54:13 -04:00
James Shubin
1a35ab61ca engine: Split out the world filesystem interface 2025-03-18 03:32:42 -04:00
James Shubin
59c33a354c engine, lang: core: world: Split out the scheduler interface 2025-03-18 03:32:42 -04:00
James Shubin
c853e24ded engine: Split out the str world interface
This is a core API and it should really be cleaned up if possible.
2025-03-18 03:32:42 -04:00
James Shubin
692db084e4 engine: Split off resource world interface 2025-03-18 03:32:42 -04:00
James Shubin
1edff3b3f5 engine: Move another interface method 2025-03-18 03:32:42 -04:00
James Shubin
b173d9f8ef engine: Split out the etcd cluster size options
This is clean up work so that it's easier to generalize the world
backends.
2025-03-18 03:32:42 -04:00
James Shubin
a697add8d0 modules: dhcp: Add more package versions 2025-03-17 22:01:43 -04:00
James Shubin
c83e2cb877 git: Add a small ignore entry 2025-03-17 21:51:45 -04:00
James Shubin
642c6b952f lang: core, funcs: Port some functions to CallableFunc API
Some modern features of our function engine and language might require
this new API, so port what we can and figure out the rest later.
2025-03-16 23:23:57 -04:00
James Shubin
f313380480 engine: resources: Container stopped should be valid for no container 2025-03-13 01:03:11 -04:00
James Shubin
f8a4751290 engine: resources: Don't prematurely error docker watches
A subtlety about the engine is that while it guarantees CheckApply
happens in the listed edge-based dependency order, it doesn't stop
Watch from starting up in whatever order it wants to. As a result, we
can prematurely error since the docker service isn't running yet. It may
in fact be in the process of getting installed and started by mgmt
before we then try and use this resource! As a result, let it error once
for free and wait for CheckApply to get going before we start again.

Keep in mind, Watch has to use the .Running() method once to tell
CheckApply to do its initial event. So this concurrency is complex!

It's unclear if this is a bug in mgmt or not, but I'm leaning towards
not, particularly since there isn't an obvious way to fix it.
2025-03-12 06:14:38 -04:00
James Shubin
3ca1aa9cb1 engine: resources: Fix backwards docker ports
This wasn't setup properly, now it's fixed. Woops.
2025-03-12 05:45:27 -04:00
James Shubin
37308b950b cli, gapi: Add more information that deploy is running
There can be a non-obvious pause, so give some hint here...
2025-03-12 05:45:26 -04:00
James Shubin
05306e3729 engine: resources: Modernize the docker resources
They made the assumption that there would be a based docker service
installed at Init which could not be guaranteed. Also use the internal
metaparameter timeout feature instead of private counters.
2025-03-12 05:45:26 -04:00
James Shubin
a6057319a9 lang: Make scope error messages be more consistent 2025-03-12 03:33:08 -04:00
James Shubin
87d8533bd0 lib: Patch out the mess when using our magic option 2025-03-11 04:53:08 -04:00
James Shubin
dce83efa96 etcd: Add a special magic option hack
Workaround some legacy code for now.
2025-03-11 04:53:08 -04:00
James Shubin
1cb9648b08 etcd: Workaround possible rare deadlock
This code is terrible, but maybe this is good enough for now.
2025-03-11 04:18:03 -04:00
James Shubin
17b859d0d7 cli, gapi, lang, lib: Add a flag to skip autoedges
The GAPI API is a bit of a mess, but I think this seems to work for
standalone run and also deploys. Hopefully I didn't add any unnecessary
extra dead code here, but that's archaeology for another day.
2025-03-11 04:18:03 -04:00
James Shubin
8d34910b9b modules: prometheus: Fix title of service template 2025-03-11 04:18:02 -04:00
James Shubin
5667fec410 modules: prometheus: Remove erroneous tmpl extension 2025-03-11 04:18:02 -04:00
James Shubin
46035fee83 engine: resources: Add simple configuration steps to virt builder
This adds some simplistic configuration management / provisioning
functionality to this virt:builder resource which makes it easier to
kick off special functionality that we might want to build.
2025-03-11 04:18:02 -04:00
James Shubin
219d25b330 engine: resources, modules: virtualization: Add a seeds option
This makes it easier to configure the machine by giving it an automatic
initial setup of an mgmt client.
2025-03-11 04:18:02 -04:00
James Shubin
181aab9c81 engine: resources: Fix small cmp typo in virt builder res 2025-03-10 19:01:05 -04:00