Commit Graph

132 Commits

Author SHA1 Message Date
James Shubin
654e958d3f engine: resources: Add the proper prefix to grouped http resources
Resources that can be grouped into the http:server resource must have
that prefix. Grouping is basically hierarchical, and without that common
prefix, it means we'd have to special-case our grouping algorithm.
2025-05-25 01:40:25 -04:00
James Shubin
0e34f13cce engine: resources: Add a line resource
Simple enough for the common cases. It just needs some tests.
2025-05-06 02:22:39 -04:00
Lourenço Vales
f2a6a6769f engine: resources: Add a WatchFiles field to exec
This adds a field that takes a list of files for exec to watch for
events on.
2025-05-05 23:54:33 -04:00
James Shubin
774d408e13 engine: Fix up some send/recv corner cases
Initially I wasn't 100% clear or decided on the send/recv semantics.
After some experimenting, I think this is much closer to what we want.
Nothing should break or regress here, this only enables more
possibilities.
2025-05-05 23:53:37 -04:00
James Shubin
dad15f6adc examples: lang: Add missing folder 2025-05-04 13:54:45 -04:00
James Shubin
6ec707aea7 examples: lang: Simplify a common example 2025-05-02 03:12:18 -04:00
James Shubin
807c4b3430 engine: resources: Add an http ui resource
Many years ago I built and demoed a prototype of a simple web ui with a
slider, and as you moved it left and right, it started up or shutdown
some number of virtual machines.

The webui was standalone code, but the rough idea of having events from
a high-level overview flow into mgmt, was what I wanted to test out. At
this stage, I didn't even have the language built yet. This prototype
helped convince me of the way a web ui would fit into everything.

Years later, I build an autogrouping prototype which looks quite similar
to what we have today. I recently picked it back up to polish it a bit
more. It's certainly not perfect, and might even be buggy, but it's
useful enough that it's worth sharing.

If I had more cycles, I'd probably consider removing the "store" mode,
and replace it with the normal "value" system, but we would need the
resource "mutate" API if we wanted this. This would allow us to directly
change the "value" field, without triggering a graph swap, which would
be a lot less clunky than the "store" situation.

Of course I'd love to see a GTK version of this concept, but I figured
it would be more practical to have a web ui over HTTP.

One notable missing feature, is that if the "web ui" changes (rather
than just a value changing) we need to offer to the user to reload it.
It currently doesn't get an event for that, and so don't confuse your
users. We also need to be better at validating "untrusted" input here.

There's also no major reason to use the "gin" framework, we should
probably redo this with the standard library alone, but it was easier
for me to push out something quick this way. We can optimize that later.

Lastly, this is all quite ugly since I'm not a very good web dev, so if
you want to make this polished, please do! The wasm code is also quite
terrible due to limitations in the compiler, and maybe one day when that
works better and doesn't constantly deadlock, we can improve it.
2025-05-02 02:14:14 -04:00
Lourenço Vales
ae68dd79cb lang: core: iter: Add a range function
This commit implements a range function that mimicks python's range
built-in by having a start, stop, and range argument. There's also
a few examples and tests to mimick Python's examples to guarantee
we're consistent with their behaviour.
2025-04-22 02:37:35 -04:00
James Shubin
de970ee557 engine: resources: Add symlink param to the file res
This adds initial symlink support to the file resource, and while it is
hopefully correct, there are always sneaky edge cases around symlinks
and security, so review and tests are highly encouraged!
2025-04-22 02:21:58 -04:00
James Shubin
9c1c587f7b lang: parse, core: world: Add a collect package
Some checks failed
/ Test (basic) on ubuntu-latest with golang 1.23 (push) Has been cancelled
/ Test (race) on ubuntu-latest with golang 1.23 (push) Has been cancelled
/ Test (shell) on ubuntu-latest with golang 1.23 (push) Has been cancelled
This lets us look at the available resource data for collection, and to
filter it so we can decide what we want to collect on our machine.

Other types of collect functions could be added in the future.
2025-04-05 17:00:53 -04:00
James Shubin
045b29291e engine, lang: Modern exported resources
I've been waiting to write this patch for a long time. I firmly believe
that the idea of "exported resources" was truly a brilliant one, but
which was never even properly understood by its original inventors! This
patch set aims to show how it should have been done.

The main differences are:

* Real-time modelling, since "once per run" makes no sense.
* Filter with code/functions not language syntax.
* Directed exporting to limit the intended recipients.

The next step is to add more "World" reading and filtering functions to
make it easy and expressive to make your selection of resources to
collect!
2025-04-05 12:45:23 -04:00
Lourenço Vales
f2d9219218 lang: core: os: Add is_virtual function
This is a basic implementation of a detection method for whether mgmt is
running in a virtualized environment. We achieve this by doing two types
of checks: on one hand, we check if the CPU flags can confirm the
presence of a virtualized env; on the other, we check if the presence
of known files related with DMI (and their contents) can confirm whether
 we're inside a virt env. Either of these situations will cause the
function to return true, with the default case being false. All of these
checks are relatively naive and can be improved by looking at the main
inspiration for this implementation, which was systemd's own check for
the presence of virtualization.
2025-03-21 14:18:55 -04:00
James Shubin
3ca1aa9cb1 engine: resources: Fix backwards docker ports
This wasn't setup properly, now it's fixed. Woops.
2025-03-12 05:45:27 -04:00
James Shubin
2899bc234a lang: Add a forkv loop statement for iterating over a map
This adds a forkv statement which is used to iterate over a map with a
body of statements. This is an important data transformation tool which
should be used sparingly, but is important to have.

An import statement inside of a forkv loop is not currently supported.
We have a simple hack to detect the obvious cases, but more deeply
nested scenarios probably won't be caught, and you'll get an obscure
error message if you try to do this.

This was incredibly challenging to get right, and it's all thanks to Sam
for his brilliance.

Note, I couldn't think of a better keyword that "forkv" but suggestions
are welcome if you think you have a better idea. Other ideas were formap
and foreach, but neither got me very excited.
2025-03-08 17:45:29 -05:00
James Shubin
cf7e73bbf6 lang: Add a for loop statement for iterating over a list
This adds a for statement which is used to iterate over a list with a
body of statements. This is an important data transformation tool which
should be used sparingly, but is important to have.

An import statement inside of a for loop is not currently supported. We
have a simple hack to detect the obvious cases, but more deeply nested
scenarios probably won't be caught, and you'll get an obscure error
message if you try to do this.

This was incredibly challenging to get right, and it's all thanks to Sam
for his brilliance.

Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
2025-03-08 17:45:29 -05:00
James Shubin
d56896cb0d lang: core: Simplify list and map lookup default functions 2025-02-26 19:59:47 -05:00
James Shubin
d579787bcd lang: core: Simplify list and map lookup functions 2025-02-26 19:59:47 -05:00
James Shubin
82489c3fe0 engine: resources: Add shell field to user resource 2025-02-07 18:08:25 -05:00
James Shubin
134e2f1cd9 examples: lang: Add a new env example 2025-02-06 07:49:35 -05:00
James Shubin
e6d614f4dd engine: resources: Add a bmc resource
This resource manages bmc devices in servers or elsewhere. This also
integrates with the provisioner code.
2025-01-03 18:19:31 -05:00
James Shubin
59a20f53eb lang: core: sys, engine: resource: Update hostname functionality
We didn't have a solid resource and sys.hostname() didn't have events!
2024-11-06 21:09:50 -05:00
James Shubin
be7a5399e3 lang: core: util: Add hostname mapper function
This adds a new util package with some useful functionality which could
be implemented as pure mcl, but instead we add it here as a good place
to help with code reuse.
2024-11-05 14:55:58 -05:00
James Shubin
422719c345 lang: core: map: Add functions to extract keys and values
Simple stuff, but now it's done!
2024-10-30 00:58:10 -04:00
James Shubin
10f82c6566 lang: core: list: Add a concat function
It works with arbitrary numbers of arguments too!
2024-10-18 10:02:29 -04:00
James Shubin
d24149518c util: distro: Refactor family and distro code
I hate writing abstraction code like this, but I'm hoping it will be
useful.
2024-10-15 20:36:50 -04:00
James Shubin
1f12150d8f engine: resources, examples: lang: Expand on future deploy ideas
This is one idea for reference. But I doubt this is needed anytime soon
since we have a good working solution in the examples.
2024-10-13 20:47:47 -04:00
James Shubin
d3a7cefcc6 engine: resources: Add an archiving, deploy resource
This makes a bundle out of the code in the current deploy. Hopefully
this is useful for handoff!
2024-10-13 16:44:23 -04:00
James Shubin
c937280664 engine: local, lang: core: local: Add a pool function
This adds a new local API for pool allocation, and with it a
corresponding function in the "local" import.
2024-10-04 22:37:39 -04:00
James Shubin
898b58e3e7 lang: core: strings: Add a substring function named substr 2024-10-03 13:49:08 -04:00
James Shubin
6419f931ee engine: resources: Add a virt-builder resource
This wraps the excellent virt-builder utility which makes setting up new
virtual machines a breeze.
2024-09-26 11:47:40 -04:00
James Shubin
a4f368fc9f engine: resources: Add a sysctl resource
Very useful since these are easy to forget!
2024-09-15 23:07:03 -04:00
James Shubin
e7b57a32fd engine: resources: Add a tar resource
This makes tar archives from a list of files and/or directories. It
correctly includes empty directories as well. The code structure is
similar to the gzip resource. While this resource is arguably more
useful than gzip, it was invaluable to write the gzip resource first
since that made writing this one much easier.
2024-09-13 20:04:53 -04:00
James Shubin
29eebd0d07 lang: core: Move template to golang namespace
I don't think this template function should be in any way authoritative,
so let's namespace it.
2024-09-13 15:51:24 -04:00
James Shubin
5bbc06d8bc engine: resources: Add new gzip resource
This may have lots of uses, particularly for bootstrapping and handoff
if we want to compress payloads. It is also a good model resource for
how to implement such a resource to avoid re-computing the result on
every CheckApply call. Of course if the computation is cheaper than the
hashing of the data this isn't the optimal approach.
2024-09-13 03:32:10 -04:00
James Shubin
2e774215e4 lang: core: deploy: Add a function to get the binary path
Useful for bootstrapping new machines.
2024-09-10 23:22:53 -04:00
James Shubin
118f266211 lang: core: local: Add a new vardir function
This gives us a function to return a created vardir folder. It is not
locally namespaced, and a future function will have to namespace one to
each scope.
2024-09-09 18:04:18 -04:00
James Shubin
3d0660559e examples: lang: Add a join example
Good reminder that a lot of the golang stdlib functions are available.
2024-09-09 14:54:45 -04:00
James Shubin
d0ed004b24 examples: lang: Test that each of the mcl examples compiles
We let these rot, so fixup the issues and test them!
2024-07-31 17:29:42 -04:00
James Shubin
6de7d8b254 lang: funcs: Catch non-specific type build error
If you had ambiguous code, and specified an invalid type, this could
sneak through and become a runtime error, instead of a compile-time
error. We fix this and add a test.
2024-07-31 17:29:42 -04:00
Omar Al-Shuha
ccb0e55d5a examples: lang: Fix env0 example
Change function calls to the correct
one, remove extra argument in getenv
call, and fix typo.
2024-07-08 02:22:57 +02:00
James Shubin
aa03b5ce2f lang: core: iter: Add filter iterator function
This was fun to write and adds a new core iterator function.
2024-07-03 21:25:19 -04:00
James Shubin
e747e12002 examples: lang: Fixup a few examples
We might change unification to allow naked single strings with fancier
unification, but let's leave it as is for now and see how often it comes
up.
2024-07-02 23:50:24 -04:00
James Shubin
d1753c592a lang: core: iter: Misc formatting fixes
Also fix up the examples.
2024-07-02 23:49:56 -04:00
James Shubin
388d08e245 engine: resources: Add a dhcp range resource
This adds the ability to offer a dhcp lease to someone when we don't
know their mac address in advance.

This also uses the extended autogrouping API to keep the internal API
simpler.
2024-03-20 17:45:06 -04:00
James Shubin
849de648f0 engine: resources: Add a creates field for exec
This adds a standard gate that prevents execution if a file exists. Of
note, this also adds a watch on it, so we can have a proper watched exec
resource without a watch cmd.
2024-03-16 01:21:32 -04:00
James Shubin
c5e3e0ee70 engine: resources: Add firewalld resource
This is a simple firewalld resource to make the seamless opening of
firewall ports in standalone laptop (and other) environments easy.
2024-03-16 00:36:06 -04:00
James Shubin
589a5f9aeb cli, lib, lang: Port to new cli library
The new version of the urfave/cli library is moving to generics, and
it's completely unclear to me why this is an improvement. Their new API
is very complicated to understand, which for me, defeats the purpose of
golang.

In parallel, I needed to do some upcoming cli API refactoring, so this
was a good time to look into new libraries. After a review of the
landscape, I found the alexflint/go-arg library which has a delightfully
elegant API. It does have a few rough edges, but it's otherwise very
usable, and I think it would be straightforward to add features and fix
issues.

Thanks Alex!
2024-03-01 21:02:55 -05:00
James Shubin
a05b6a927e lang: funcs: core: os: Add an args function to read argv 2024-02-22 13:15:39 -05:00
James Shubin
26bce5ddc0 engine: resources: Add a new http:proxy resource
This adds a new caching http proxy resource that can be autogrouped into
the core http:server resource, and which caches and serves files that it
received from a different http server. It first pulls them down when
they are initially requested, which makes it possible for us to use this
for provisioning a Linux installation without having to pre-rsync the
entire package repository.
2024-02-22 13:15:39 -05:00
James Shubin
fc49888ba2 examples: lang: Add a simpler map-iterator example 2024-01-31 16:02:47 -05:00