Commit Graph

398 Commits

Author SHA1 Message Date
James Shubin
936cf7dd9d engine: graph: autogroup: Ensure we sort correctly
This should ensure this sort order of longer thing first, when running
the algorithm. We want to autogroup two or more http:server:flag
resources together first, before the whole grouped resource gets pulled
into the http:server one.
2025-05-25 04:21:44 -04:00
James Shubin
fd5bc63293 engine: traits: Allow self-kind with hierarchical grouping
Not sure why I was blocking this previously, perhaps there was no use
case, and I was trying to ensure there wasn't accidental, unwanted
grouping? Perhaps I wanted to improve performance? In any case, let's
turn this off for now, and check for bugs.
2025-05-25 04:19:38 -04:00
James Shubin
be4cb6658e engine: graph: Simplify the Send/Recv code
This code might be slightly redundant, so simplify it. If something went
wrong, this will be the commit that did it, but it seems relatively
safe.
2025-05-25 04:18:26 -04:00
James Shubin
efff84bbd4 engine: graph: autogroup: Print these errors when debugging
This gives important clues as to why something isn't grouping as
expected. Show them if needed.
2025-05-25 03:48:57 -04:00
James Shubin
74f36c5d73 engine: resources: Add some compile time checks for groupers
These can "break" silently and not autogroup if we change the resource
and it no longer fulfills the interface. Add this compile time check to
prevent that.
2025-05-25 03:47:47 -04:00
James Shubin
b868a60f69 engine: resources: Simplify the Watch loop
I had some legacy unnecessary boolean for sending everywhere. Not sure
why I never re-read it, it's so easy to just copy and paste and carry
on.
2025-05-25 02:12:14 -04:00
James Shubin
f73127ec23 engine: resources: Make error not ambiguous
The same text exists elsewhere.
2025-05-25 01:42:41 -04:00
James Shubin
654e958d3f engine: resources: Add the proper prefix to grouped http resources
Resources that can be grouped into the http:server resource must have
that prefix. Grouping is basically hierarchical, and without that common
prefix, it means we'd have to special-case our grouping algorithm.
2025-05-25 01:40:25 -04:00
James Shubin
1f54253f95 engine: resources: Add a trim field to line resource 2025-05-25 01:40:21 -04:00
James Shubin
861ba50f9c engine: resources: Add a ui redirect
I always forget the /index.html part so make it easier!
2025-05-15 02:52:57 -04:00
James Shubin
287504cfa8 engine: resources: Add missing struct tags to kv 2025-05-15 01:46:03 -04:00
James Shubin
e86d66b906 engine: resources: Avoid double slash on error
Errors will include a second slash if this ends with one. Might as well
clean it to avoid the semblance of a bug.
2025-05-09 00:00:46 -04:00
James Shubin
9a63fadfbd engine: resources: Rename var
So it doesn't conflict with "path" import.
2025-05-08 23:16:57 -04:00
James Shubin
7afa372765 engine: resources: Let the user race me
If a user is racing the file resource, don't error permanently, just
skip the file that vanished, and move on with your life.
2025-05-08 23:11:51 -04:00
James Shubin
fddebb2474 engine, lang: core: Match exported resources properly
I inverted the logic for complex setups and forgot to handle the zero
cases. I also didn't notice my loop continue error. This cleans all this
up so that we can have proper exported resource matching.
2025-05-08 22:29:03 -04:00
James Shubin
ad0dd44130 engine: Don't force validation for hidden resources
I think this is what I want in most scenarios, is there a reason to do
otherwise? This is because we may wish to export incomplete resources,
where the remaining necessary fields for validation happens on collect.
2025-05-06 03:36:01 -04:00
James Shubin
2ee403bab9 git: Update gitignore files
We were overly matching in some cases by not starting with a slash. This
updates a few other cases too.
2025-05-06 02:52:26 -04:00
James Shubin
0e34f13cce engine: resources: Add a line resource
Simple enough for the common cases. It just needs some tests.
2025-05-06 02:22:39 -04:00
Lourenço Vales
f2a6a6769f engine: resources: Add a WatchFiles field to exec
This adds a field that takes a list of files for exec to watch for
events on.
2025-05-05 23:54:33 -04:00
James Shubin
774d408e13 engine: Fix up some send/recv corner cases
Initially I wasn't 100% clear or decided on the send/recv semantics.
After some experimenting, I think this is much closer to what we want.
Nothing should break or regress here, this only enables more
possibilities.
2025-05-05 23:53:37 -04:00
James Shubin
ae1d9b94d4 engine: util: Add a debug utility
This is useful for some patches. Let's see if I can remember to use and
improve it!
2025-05-05 22:30:31 -04:00
James Shubin
267bcc144b engine: util: Clean up error messages 2025-05-05 22:29:10 -04:00
James Shubin
fd40c3b64f engine: util: Fix grammar typo 2025-05-05 20:21:29 -04:00
James Shubin
e2b6da01d8 engine: graph: Fix messy imports 2025-05-05 20:21:29 -04:00
James Shubin
807c4b3430 engine: resources: Add an http ui resource
Many years ago I built and demoed a prototype of a simple web ui with a
slider, and as you moved it left and right, it started up or shutdown
some number of virtual machines.

The webui was standalone code, but the rough idea of having events from
a high-level overview flow into mgmt, was what I wanted to test out. At
this stage, I didn't even have the language built yet. This prototype
helped convince me of the way a web ui would fit into everything.

Years later, I build an autogrouping prototype which looks quite similar
to what we have today. I recently picked it back up to polish it a bit
more. It's certainly not perfect, and might even be buggy, but it's
useful enough that it's worth sharing.

If I had more cycles, I'd probably consider removing the "store" mode,
and replace it with the normal "value" system, but we would need the
resource "mutate" API if we wanted this. This would allow us to directly
change the "value" field, without triggering a graph swap, which would
be a lot less clunky than the "store" situation.

Of course I'd love to see a GTK version of this concept, but I figured
it would be more practical to have a web ui over HTTP.

One notable missing feature, is that if the "web ui" changes (rather
than just a value changing) we need to offer to the user to reload it.
It currently doesn't get an event for that, and so don't confuse your
users. We also need to be better at validating "untrusted" input here.

There's also no major reason to use the "gin" framework, we should
probably redo this with the standard library alone, but it was easier
for me to push out something quick this way. We can optimize that later.

Lastly, this is all quite ugly since I'm not a very good web dev, so if
you want to make this polished, please do! The wasm code is also quite
terrible due to limitations in the compiler, and maybe one day when that
works better and doesn't constantly deadlock, we can improve it.
2025-05-02 02:14:14 -04:00
James Shubin
412e480b44 engine: local: Get the logic right
I think we were not benefitting from the cache and sending unnecessary
events. It would be great to have tests for this, but commit this fix
for now, and be embarrassed in the future if I got this code wrong.
2025-05-02 00:04:00 -04:00
James Shubin
cc2a235fbb engine: resources: Add a reminder about events
I might want to do this some day, it could be important. Look into it.
2025-05-02 00:04:00 -04:00
James Shubin
7c77efec1d engine: resources: Cleanup this old code
This is equivalent and cleaner.
2025-05-02 00:04:00 -04:00
James Shubin
de970ee557 engine: resources: Add symlink param to the file res
This adds initial symlink support to the file resource, and while it is
hopefully correct, there are always sneaky edge cases around symlinks
and security, so review and tests are highly encouraged!
2025-04-22 02:21:58 -04:00
James Shubin
748f05732a engine, etcd: Watch on star pattern for all hostnames
We forgot to watch on star hostname matches.
2025-04-05 15:45:44 -04:00
James Shubin
148bd50e9f engine, etcd: Prevent engine thrashing
These two small bugs would allow thrashing to occur since we'd
constantly delete and re-add exports, and constantly think that a noop
etcd operation made a change.
2025-04-05 15:28:54 -04:00
James Shubin
6c1c08ceda engine: resources: Test to make sure metaparams are preserved
We should ensure these get preserved across encoding/decoding. We rely
on this behaviour.
2025-04-05 12:45:23 -04:00
James Shubin
045b29291e engine, lang: Modern exported resources
I've been waiting to write this patch for a long time. I firmly believe
that the idea of "exported resources" was truly a brilliant one, but
which was never even properly understood by its original inventors! This
patch set aims to show how it should have been done.

The main differences are:

* Real-time modelling, since "once per run" makes no sense.
* Filter with code/functions not language syntax.
* Directed exporting to limit the intended recipients.

The next step is to add more "World" reading and filtering functions to
make it easy and expressive to make your selection of resources to
collect!
2025-04-05 12:45:23 -04:00
James Shubin
955112f64f engine: Let others use the ResUID struct
It's a useful key in maps.
2025-04-05 12:45:23 -04:00
James Shubin
7f341cee84 engine: resources: Improving logging even more
Messages should happen after the event on success. The error scenario
has its own pathway to report.
2025-04-05 12:45:23 -04:00
James Shubin
f71e623931 engine: resources: Print a message on empty file creation
We don't see this event happening which is confusing. There might be
other cases we didn't handle cleanly.
2025-04-05 12:45:23 -04:00
James Shubin
30aca74089 engine, yamlgraph: Disable the old exported resources stuff
These were really just stubs so that I could prove out the reactive
model very early, and I don't think they're really used anywhere.

I'm also not really using the yamlgraph frontend. If someone wants any
of that code, step up, or it will rot even more.
2025-04-05 12:45:23 -04:00
James Shubin
3dfca97f86 engine: Add a method to determine if a res kind is valid 2025-04-05 12:45:23 -04:00
James Shubin
5665259784 cli, engine, etcd, lib: Move the hostname value to the API
Every world implementation needs a unique UUID, might as well move this
to the API.
2025-03-19 05:41:04 -04:00
James Shubin
a7a5237b07 cli, engine, etcd, lib: Pass in init args
Improve the API and make it more general.
2025-03-18 04:54:13 -04:00
James Shubin
7ad54fe3e8 cli, engine, etcd, lib: Split out the deployer into world
This should hopefully make the refactor into a clean world API a bit
better. Still more to do though!
2025-03-18 04:54:13 -04:00
James Shubin
1a35ab61ca engine: Split out the world filesystem interface 2025-03-18 03:32:42 -04:00
James Shubin
59c33a354c engine, lang: core: world: Split out the scheduler interface 2025-03-18 03:32:42 -04:00
James Shubin
c853e24ded engine: Split out the str world interface
This is a core API and it should really be cleaned up if possible.
2025-03-18 03:32:42 -04:00
James Shubin
692db084e4 engine: Split off resource world interface 2025-03-18 03:32:42 -04:00
James Shubin
1edff3b3f5 engine: Move another interface method 2025-03-18 03:32:42 -04:00
James Shubin
b173d9f8ef engine: Split out the etcd cluster size options
This is clean up work so that it's easier to generalize the world
backends.
2025-03-18 03:32:42 -04:00
James Shubin
f313380480 engine: resources: Container stopped should be valid for no container 2025-03-13 01:03:11 -04:00
James Shubin
f8a4751290 engine: resources: Don't prematurely error docker watches
A subtlety about the engine is that while it guarantees CheckApply
happens in the listed edge-based dependency order, it doesn't stop
Watch from starting up in whatever order it wants to. As a result, we
can prematurely error since the docker service isn't running yet. It may
in fact be in the process of getting installed and started by mgmt
before we then try and use this resource! As a result, let it error once
for free and wait for CheckApply to get going before we start again.

Keep in mind, Watch has to use the .Running() method once to tell
CheckApply to do its initial event. So this concurrency is complex!

It's unclear if this is a bug in mgmt or not, but I'm leaning towards
not, particularly since there isn't an obvious way to fix it.
2025-03-12 06:14:38 -04:00
James Shubin
3ca1aa9cb1 engine: resources: Fix backwards docker ports
This wasn't setup properly, now it's fixed. Woops.
2025-03-12 05:45:27 -04:00