Commit Graph

2358 Commits

Author SHA1 Message Date
James Shubin
e86d66b906 engine: resources: Avoid double slash on error
Errors will include a second slash if this ends with one. Might as well
clean it to avoid the semblance of a bug.
2025-05-09 00:00:46 -04:00
James Shubin
9a63fadfbd engine: resources: Rename var
So it doesn't conflict with "path" import.
2025-05-08 23:16:57 -04:00
James Shubin
7afa372765 engine: resources: Let the user race me
If a user is racing the file resource, don't error permanently, just
skip the file that vanished, and move on with your life.
2025-05-08 23:11:51 -04:00
James Shubin
fddebb2474 engine, lang: core: Match exported resources properly
I inverted the logic for complex setups and forgot to handle the zero
cases. I also didn't notice my loop continue error. This cleans all this
up so that we can have proper exported resource matching.
2025-05-08 22:29:03 -04:00
James Shubin
ad0dd44130 engine: Don't force validation for hidden resources
I think this is what I want in most scenarios, is there a reason to do
otherwise? This is because we may wish to export incomplete resources,
where the remaining necessary fields for validation happens on collect.
2025-05-06 03:36:01 -04:00
James Shubin
2ee403bab9 git: Update gitignore files
We were overly matching in some cases by not starting with a slash. This
updates a few other cases too.
2025-05-06 02:52:26 -04:00
James Shubin
0e34f13cce engine: resources: Add a line resource
Simple enough for the common cases. It just needs some tests.
2025-05-06 02:22:39 -04:00
Lourenço Vales
f2a6a6769f engine: resources: Add a WatchFiles field to exec
This adds a field that takes a list of files for exec to watch for
events on.
2025-05-05 23:54:33 -04:00
James Shubin
4903995052 lang: core: os: Add readfilewait which won't error easily
Just a useful helper function which we may want for a while.
2025-05-05 23:54:28 -04:00
James Shubin
774d408e13 engine: Fix up some send/recv corner cases
Initially I wasn't 100% clear or decided on the send/recv semantics.
After some experimenting, I think this is much closer to what we want.
Nothing should break or regress here, this only enables more
possibilities.
2025-05-05 23:53:37 -04:00
James Shubin
ae1d9b94d4 engine: util: Add a debug utility
This is useful for some patches. Let's see if I can remember to use and
improve it!
2025-05-05 22:30:31 -04:00
James Shubin
267bcc144b engine: util: Clean up error messages 2025-05-05 22:29:10 -04:00
James Shubin
fd40c3b64f engine: util: Fix grammar typo 2025-05-05 20:21:29 -04:00
James Shubin
e2b6da01d8 engine: graph: Fix messy imports 2025-05-05 20:21:29 -04:00
James Shubin
dad15f6adc examples: lang: Add missing folder 2025-05-04 13:54:45 -04:00
James Shubin
6ec707aea7 examples: lang: Simplify a common example 2025-05-02 03:12:18 -04:00
James Shubin
807c4b3430 engine: resources: Add an http ui resource
Many years ago I built and demoed a prototype of a simple web ui with a
slider, and as you moved it left and right, it started up or shutdown
some number of virtual machines.

The webui was standalone code, but the rough idea of having events from
a high-level overview flow into mgmt, was what I wanted to test out. At
this stage, I didn't even have the language built yet. This prototype
helped convince me of the way a web ui would fit into everything.

Years later, I build an autogrouping prototype which looks quite similar
to what we have today. I recently picked it back up to polish it a bit
more. It's certainly not perfect, and might even be buggy, but it's
useful enough that it's worth sharing.

If I had more cycles, I'd probably consider removing the "store" mode,
and replace it with the normal "value" system, but we would need the
resource "mutate" API if we wanted this. This would allow us to directly
change the "value" field, without triggering a graph swap, which would
be a lot less clunky than the "store" situation.

Of course I'd love to see a GTK version of this concept, but I figured
it would be more practical to have a web ui over HTTP.

One notable missing feature, is that if the "web ui" changes (rather
than just a value changing) we need to offer to the user to reload it.
It currently doesn't get an event for that, and so don't confuse your
users. We also need to be better at validating "untrusted" input here.

There's also no major reason to use the "gin" framework, we should
probably redo this with the standard library alone, but it was easier
for me to push out something quick this way. We can optimize that later.

Lastly, this is all quite ugly since I'm not a very good web dev, so if
you want to make this polished, please do! The wasm code is also quite
terrible due to limitations in the compiler, and maybe one day when that
works better and doesn't constantly deadlock, we can improve it.
2025-05-02 02:14:14 -04:00
James Shubin
6b10477ebc lang: core: convert: Add a simple str to int function 2025-05-02 00:24:21 -04:00
James Shubin
412e480b44 engine: local: Get the logic right
I think we were not benefitting from the cache and sending unnecessary
events. It would be great to have tests for this, but commit this fix
for now, and be embarrassed in the future if I got this code wrong.
2025-05-02 00:04:00 -04:00
James Shubin
cc2a235fbb engine: resources: Add a reminder about events
I might want to do this some day, it could be important. Look into it.
2025-05-02 00:04:00 -04:00
James Shubin
7c77efec1d engine: resources: Cleanup this old code
This is equivalent and cleaner.
2025-05-02 00:04:00 -04:00
James Shubin
4b1548488d lib: It is called mcl officially for a while now 2025-04-28 00:31:14 -04:00
James Shubin
47aecd25c3 lang: funcs: structs: Pass through the type
Not sure why this wasn't done or if it should be, but seems plausible
for now.
2025-04-27 22:23:42 -04:00
James Shubin
fb6eae184a lang: ast: Refactor to unindent slightly 2025-04-27 22:19:14 -04:00
James Shubin
16d3e3063c lang: funcs: facts: Do not reuse fact pointers
In my carelessness, I was re-using pointers when a fact was used twice!
This could cause disastrous consequences like a double close panic on a
datetime.now() fact for example.

In other news, we should consider if it's possible to get more clever
about graph shape optimization so that we don't need more than once
instance of certain functions like datetime.now() in our graph.
2025-04-27 22:14:51 -04:00
James Shubin
37bb67dffd lang: Improve graph shape with speculative execution
Most of the time, we don't need to have a dynamic call sub graph, since
the actual function call could be represented statically as it
originally was before lambda functions were implemented. Simplifying the
graph shape has important performance benefits in terms of both keep the
graph smaller (memory, etc) and in avoiding the need to run transactions
at runtime (speed) to reshape the graph.

Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
2025-04-27 22:14:51 -04:00
James Shubin
9c9f2f558a lang: Move out this legacy execution function
Hasn't been used in a while, but it's fine if we want to use it for
tests.
2025-04-22 03:24:23 -04:00
James Shubin
1a81e57410 lang: interfaces: Update stale comments 2025-04-22 03:24:23 -04:00
James Shubin
7096293885 lang: funcs: dage: Return better errors
Helps a lot with debugging.
2025-04-22 03:24:23 -04:00
James Shubin
1536a94026 lang: Functions that build should be copyable
It's not entirely clear if this is required, but it's probably a good
idea. We should consider making it a requirement of the BuildableFunc
interface.
2025-04-22 03:24:23 -04:00
James Shubin
1bb1e056c4 lang: funcs: structs: Add some extra safety checks
Not sure if these are even needed.
2025-04-22 03:24:23 -04:00
James Shubin
e71b11f843 lang: funcs: facts: Check if a fact is callable 2025-04-22 03:24:23 -04:00
James Shubin
b4769eefd9 lang: funcs: facts: Add a separate callable interface
Add some symmetry to our interfaces for now, even though I'd love to
drop the idea of "facts" altogether.
2025-04-22 03:24:23 -04:00
James Shubin
d4a24d4c9d lang: funcs: wrapped: Simplify the implementation 2025-04-22 03:24:23 -04:00
Ahmad Abuziad
c5d7fdb0a3 util: Add a bunch of tests
This improves our test coverage significantly.
2025-04-22 03:18:49 -04:00
Lourenço Vales
ae68dd79cb lang: core: iter: Add a range function
This commit implements a range function that mimicks python's range
built-in by having a start, stop, and range argument. There's also
a few examples and tests to mimick Python's examples to guarantee
we're consistent with their behaviour.
2025-04-22 02:37:35 -04:00
James Shubin
de970ee557 engine: resources: Add symlink param to the file res
This adds initial symlink support to the file resource, and while it is
hopefully correct, there are always sneaky edge cases around symlinks
and security, so review and tests are highly encouraged!
2025-04-22 02:21:58 -04:00
James Shubin
60a3d7c65e lang: interfaces: Add more information about graph semantics
Sam thoughts.
2025-04-19 13:02:51 -04:00
James Shubin
9c1c587f7b lang: parse, core: world: Add a collect package
Some checks failed
/ Test (basic) on ubuntu-latest with golang 1.23 (push) Has been cancelled
/ Test (race) on ubuntu-latest with golang 1.23 (push) Has been cancelled
/ Test (shell) on ubuntu-latest with golang 1.23 (push) Has been cancelled
This lets us look at the available resource data for collection, and to
filter it so we can decide what we want to collect on our machine.

Other types of collect functions could be added in the future.
2025-04-05 17:00:53 -04:00
James Shubin
af04d364d0 lang: core: fmt: Make printf handle more cases
Until we make a clean determination about what this should print, this
should handle things for now.
2025-04-05 16:14:11 -04:00
James Shubin
748f05732a engine, etcd: Watch on star pattern for all hostnames
We forgot to watch on star hostname matches.
2025-04-05 15:45:44 -04:00
James Shubin
148bd50e9f engine, etcd: Prevent engine thrashing
These two small bugs would allow thrashing to occur since we'd
constantly delete and re-add exports, and constantly think that a noop
etcd operation made a change.
2025-04-05 15:28:54 -04:00
James Shubin
6c1c08ceda engine: resources: Test to make sure metaparams are preserved
We should ensure these get preserved across encoding/decoding. We rely
on this behaviour.
2025-04-05 12:45:23 -04:00
James Shubin
045b29291e engine, lang: Modern exported resources
I've been waiting to write this patch for a long time. I firmly believe
that the idea of "exported resources" was truly a brilliant one, but
which was never even properly understood by its original inventors! This
patch set aims to show how it should have been done.

The main differences are:

* Real-time modelling, since "once per run" makes no sense.
* Filter with code/functions not language syntax.
* Directed exporting to limit the intended recipients.

The next step is to add more "World" reading and filtering functions to
make it easy and expressive to make your selection of resources to
collect!
2025-04-05 12:45:23 -04:00
James Shubin
955112f64f engine: Let others use the ResUID struct
It's a useful key in maps.
2025-04-05 12:45:23 -04:00
James Shubin
7f341cee84 engine: resources: Improving logging even more
Messages should happen after the event on success. The error scenario
has its own pathway to report.
2025-04-05 12:45:23 -04:00
James Shubin
f71e623931 engine: resources: Print a message on empty file creation
We don't see this event happening which is confusing. There might be
other cases we didn't handle cleanly.
2025-04-05 12:45:23 -04:00
James Shubin
8ff187b4e9 lang: core, funcs: Rename things for consistency
Seems we had different patterns going on. This makes those all
consistent now.
2025-04-05 12:45:23 -04:00
James Shubin
30aca74089 engine, yamlgraph: Disable the old exported resources stuff
These were really just stubs so that I could prove out the reactive
model very early, and I don't think they're really used anywhere.

I'm also not really using the yamlgraph frontend. If someone wants any
of that code, step up, or it will rot even more.
2025-04-05 12:45:23 -04:00
James Shubin
3dfca97f86 engine: Add a method to determine if a res kind is valid 2025-04-05 12:45:23 -04:00