Commit Graph

89 Commits

Author SHA1 Message Date
Edward Toroshchyn
c25a2a257b misc: Fix typos and spelling errors 2025-02-24 16:01:46 +01:00
James Shubin
d30ff6cfae legal: Remove year
Instead of constantly making these updates, let's just remove the year
since things are stored in git anyways, and this is not an actual modern
legal risk anymore.
2025-01-26 16:24:51 -05:00
James Shubin
097efdd66a engine: graph: autoedge: Clean up redundant logs
They repeat themselves, this is cleaner.
2024-12-03 00:56:22 -05:00
James Shubin
4489e5ce6e engine: graph: autoedge: Quiet down the useless logs
These are so useful, let's silence them.
2024-09-18 21:38:29 -04:00
James Shubin
57b4a7efce lib, engine: graph: Let children directories be readable
We want to be able to put useful scripts in $vardir type places, but if
the perms at the higher levels block this, then that can't work. The
top-level should always be more permissive, and then it grows more
restricted as we descend.
2024-09-18 21:03:58 -04:00
James Shubin
80178422db engine: graph, resources: Clean up log messages
The idea is to have a better user experience in the terminal.
2024-08-06 15:12:10 -04:00
James Shubin
8a78907977 engine: autogroup, traits, graph: Extend autogroup API
This extends the autogrouping API so that a child can easily get a
reference to the parent that it is autogrouped in. This can simplify the
API for some resources when it makes sense to allow them access to the
parent handle. Use sparingly and intelligently!
2024-03-20 17:45:06 -04:00
James Shubin
90f6d4e563 legal: Update http to https 2024-03-05 01:05:50 -05:00
James Shubin
3e31ee9455 legal: Additional permission under GNU GPL version 3 section 7
With the recent merging of embedded package imports and the entry CLI
package, it is now possible for users to build in mcl code into a single
binary. This additional permission makes it explicitly clear that this
is permitted to make it easier for those users. The condition is phrased
so that the terms can be "patched" by the original author if it's
necessary for the project. For example, if the name of the language
(mcl) changes, has a differently named new version, someone finds a
phrasing improvement or a legal loophole, or for some other
reasonable circumstance. Now go write some beautiful embedded tools!
2024-03-05 01:04:09 -05:00
James Shubin
dd0e67540f all: Remove deprecated io/ioutil package
Porting everything to the newer imports was trivial except for one
instance which required a very small refactor.
2024-02-28 16:36:49 -05:00
James Shubin
8ea7d4cf84 engine: graph, util: Restore send/recv behaviour
A regression in 4b0cdf9123 caused the
basic send/recv functionality to break for simple scenarios. This was
due to inadequate testing, and a partial misunderstanding of the
situation.

New testing should hopefully catch more cases, but send/recv and
compile-time checks are still not as complete as is probably possible.
2024-01-31 22:59:58 -05:00
James Shubin
1630eafbe3 engine: graph: Allow Recv overriding for Send/Recv
This allows us to pass in an alternate implementation for Recv, if we
want to temporarily use a different data source.
2024-01-22 20:37:53 -05:00
James Shubin
091bf3a64c engine: graph: Add a function for printing Send/Recv logs 2024-01-22 20:37:53 -05:00
James Shubin
e5a189b8c6 engine: graph: Split SendRecv off from the engine
It can be used in more places if it's not tied to the engine struct.
This also changes the signature so that more information is returned.
This can be used for logging or other useful things. Of note, this
happens to be the same struct as already exists. It's used for
convenience since it happens to match up! Of course they're related.
2024-01-22 20:37:53 -05:00
James Shubin
e946b39960 engine: graph: Set closed flag when resume signal closes
We had this backwards. Woops...
2024-01-22 20:37:53 -05:00
James Shubin
9dd5dfdde2 engine: graph: autogroup: Improve the autogrouping algorithm
This improves the autogrouping algorithm to support hierarchical
autogrouping. It's not guaranteed to work if we replace the reachability
grouper with something more efficient, but it's good enough for now.
2024-01-22 20:37:53 -05:00
James Shubin
a8f194259b legal: Happy 2024 everyone...
Done with:

ack '2023+' -l | xargs sed -i -e 's/2023+/2024+/g'

Checked manually with:

git add -p

Hello to future James from 2025, and Happy Hacking!
2024-01-22 15:52:49 -05:00
James Shubin
bc63b7608e engine: graph: Use an atomic bool instead of a mutex
The isStateOK variable can be accessed concurrently as these are
supposed to be "benign" races. As such, they need to be labelled as such
so that we don't hit some undefined compiler behaviour.

Here are five good references relating to "benign" data races in golang.

1) https://web.archive.org/web/20181022150257/https://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong

2) https://go.dev/ref/mem - "Informal Overview" section.

3) https://docs.oracle.com/cd/E19205-01/820-0619/gecqt/index.html

4) https://www.usenix.org/legacy/event/hotpar11/tech/final_files/Boehm.pdf

5) https://go.dev/doc/articles/race_detector

TL;DR: wrap your benign races with sync/atomic or eliminate them.
2024-01-05 15:50:42 -05:00
James Shubin
c2f508e261 engine: graph: Use an rwmutex around read/write of vertex state
This fixes two small races we had while simultaneously reading from and
writing to the vertex timestamp, and simultaneously writing to the
vertex isStateOK (dirty) flag.

They were actually "safe" races, in that it doesn't matter if the
read/write race got the old or new value, or that the double write
happened. The time sequencing was correct (I believe) in both cases, but
this triggers the race detector now that we have tests for it.
2024-01-02 18:18:34 -05:00
James Shubin
18e1f08156 engine: graph: Allow send/recv to work with autogrouped resources
We've previously not received a value from within an autogrouped
resource. It turns out this would be quite useful, and so this patch
implements the additional plumbing and testing so that this works!

Testing that an autogrouped resource can still send values has not been
done at this time.
2023-12-08 18:18:17 -05:00
James Shubin
9d47b6843f engine, gapi, lang, lib: Plumb through new local API
This is a new API that is similar in spirit and plumbing to the World
API, but it intended for all local machine operations and will likely
only ever have one implementation.
2023-12-03 18:24:20 -05:00
James Shubin
7991b4ab25 engine, lang, lib: Re-order for consistency
Some small cleanups to make room for future patches.
2023-12-03 18:02:00 -05:00
James Shubin
ee88161808 engine: graph, resources: Reduce and clean up logging
Make the output more usable.
2023-11-22 22:37:35 -05:00
James Shubin
029cfaf1f8 engine, lib: Misc log and comment changes
Just misc polishing.
2023-11-22 20:21:40 -05:00
James Shubin
75d3895e84 engine: Improve logging for resources and exec
This makes things a bit clearer and fixes some silly formatting bugs.
2023-11-22 20:21:40 -05:00
James Shubin
4b0cdf9123 lang: types, engine: graph: Support pointer interfaces
If we have rare, but special *interface{} values in resource structs, we
should be able to handle them normally. It's really not recommended that
you use these unless you know exactly why they are useful.
2023-11-18 13:35:52 -05:00
James Shubin
aa001ed2dc engine: graph: Improve error message for rare case
Not sure if we ever hit this, but we should fix it up since I noticed
it.
2023-11-18 13:35:52 -05:00
James Shubin
3a3bc568b3 engine: Allow more send/recv cases
This allows the other part of this. This still needs to be improved, but
it unblocks some use-cases.
2023-11-12 17:50:07 -05:00
James Shubin
b048b2684b engine, lang: Allow resources with a field of type interface
This lets us add a resource that has an implementation with a field
whose type is determined at compile time. This let's us write more
flexible resources.

What's missing is additional type checking so that we guarantee that a
specific resource doesn't change types during run-time.
2023-11-12 17:25:03 -05:00
James Shubin
9a1a81925e engine: Improve error messages for ambiguity
This appears twice, make each one unique.
2023-11-12 17:09:50 -05:00
James Shubin
4fb356b19b engine: graph: Refactor the send/recv code to inline it
This is useful subsequently.
2023-11-12 16:13:04 -05:00
James Shubin
089267837d engine: graph: Add a lock around metas access
We forgot about the concurrent writes. This should fix that.
2023-09-19 13:46:20 -04:00
James Shubin
0d381e4c91 engine: graph: Handle the back poke differently
A back poke is the deferral or delay of a Process/CheckApply. This is
because we notice that we're not truly ready to CheckApply due to some
timestamp issue. When Process errors, we should accept that, but not
treat it as a success.
2023-09-04 15:22:14 -04:00
James Shubin
7ccda7e99b engine: Add a ctx to the CheckApply API
This is just a rough port, there are lots of optimizations to be done
and lots of timeout values that should be replaced by a new timeout meta
param!
2023-09-02 01:34:42 -04:00
James Shubin
cf49a9a6e7 engine: graph: Give retry channel its own signal
This just makes the copy+pasting less confusing.
2023-09-02 00:59:48 -04:00
James Shubin
1d10f85c28 engine: graph: Move misplaced comment 2023-09-01 22:13:34 -04:00
James Shubin
d3d84524f5 engine: graph: Improve limit output
Just show the milliseconds, and round it slightly.
2023-09-01 21:57:05 -04:00
James Shubin
cc04221516 engine: graph: Allow pause/resume while in retry or limit
The retry and limit "satellite" event loops didn't allow pausing or
resuming, and instead you needed to wait until either was done before
you could pause.

The downside of this patch is that for very fast graph transitions, we
wouldn't be really obeying the limits anymore, however now that we have
per resource kind+name uid, we can persist the limits across graph swaps
if we want to.

Most importantly, this allows us to exit entirely when we're stuck in
one of these satellite loops.
2023-09-01 21:57:05 -04:00
James Shubin
f9bc50e262 engine: Retry should be stateful and add RetryReset
Make the retry meta param a bit more sane now that we can persist it
between graph switches. This also unblocks us from pausing during retry
loops.
2023-09-01 21:57:05 -04:00
James Shubin
9545e409d4 engine: Create a resource kind+name specific stateful store
This adds a meta state store that is preserved between graph switches if
the kind and name match. This is useful so that rapid graph changes
don't necessarily reset their retry count if they've only changed one
resource field.
2023-09-01 21:57:05 -04:00
James Shubin
8299c04fc6 engine: graph, util: Clean up error printing
We should improve on this more, but at least as a quick fix, stop
splitting the error across two lines. This makes the logs really ugly.
2023-09-01 16:53:40 -04:00
James Shubin
0b1b0a3f80 engine: graph: Don't deadlock on error
This simplifies the pause mechanism and also avoids a deadlock on error.
If the Worker shuts down completely, but before we've been removed from
the graph, then an attempted pause would deadlock if we didn't have an
escape hatch here.

This removes the unnecessary ack mechanism now that we have a
synchronous channel send to represent the pausing, rather than an
asynchronous channel closing.
2023-09-01 16:53:40 -04:00
James Shubin
2773a621a2 engine: graph: Cleanup pause/resume code
There's always the fear that there is either a panic or a deadlock in
the highly concurrent engine resource code. I have not seen one recently
and I've been running some pretty concurrent tests. In the meantime, and
with my hopefully improved knowledge of concurrency, I decided to
rewrite some of the "uglier" parts of the engine. I think it is a lot
clearer now, and much less likely that there is a concurrency issue.

This has been tested by running the examples/lang/fastcount.mcl example.
2023-08-30 21:58:38 -04:00
James Shubin
c06cf44fd7 lib, engine: graph: Rename the Close method 2023-08-30 21:38:01 -04:00
James Shubin
7288e5d4a4 engine: graph: Improve documentation on concurrent use
Certain other methods should not be called concurrently, but this only
documents the most important cases.
2023-08-30 21:38:01 -04:00
James Shubin
b62f501745 engine: graph: Fix typo
We actually now cleanup instead of closing. It's semantically slightly
different, so be consistent with the error message.
2023-08-30 21:38:01 -04:00
James Shubin
514927c0b3 engine: resources, graph: Change the Close method to Cleanup
This also changes a few similarly named methods. Clearer what it's doing
in terms of cleanup vs. causing some action.
2023-08-08 01:11:29 -04:00
James Shubin
963393e3d9 engine: graph, resources: Change Watch to use ctx
This is a general port. There are many optimizations and cleanups we can
do now that we have a proper context passed in. That's for a future
patch.
2023-08-08 01:11:29 -04:00
James Shubin
53a878bf61 engine: resources, graph: Change the done channel into a ctx
This is part one of porting Watch to context.
2023-08-08 01:11:29 -04:00
James Shubin
c5efe7a17b lang, engine: Remove unneeded error wrapping
These situations basically never fail, and if they do, we certainly
don't need more context. This simplifies things a bit.
2023-04-20 18:02:40 -04:00