These test both graph shape consistency and single value outputs.
Eventually we want to make the graph shape tests more precise, and also
verify specific outputs how it used to be. For now, this is okay.
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
This adds a new implementation of the function engine that runs the DAG
function graph. This version is notable in that it can run a graph that
changes shape over time. To make changes to the same of the graph, you
must use the new transaction (Txn) system. This system implements a
simple garbage collector (GC) for scheduled removal of nodes that the
transaction system "reverses" out of the graph.
Special thanks to Samuel Gélineau <gelisam@gmail.com> for his help
hacking on and debugging so much of this concurrency work with me.
We should check this for safety. An error is better than a panic. If we
try to set an unexported field, this would panic. We should prevent
being able to even type unify that though!
This returns the type with the arg names we'll actually use. This is
helpful so we can pass values to the right places. We have named edges
so you can actually see what's going on.
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
This plumbs through the new Output method signature that accepts a table
of function pointers to values and relies on the previous storing of the
function pointers to be used for the lookup right now. This has the
elegant side-effect that Output generation could run in parallel with
the graph engine, as the engine only needs to pause to take a snapshot
of the current values tables.
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
The Graph signature changes are needed for future function work, and it
also fits in nicely with the need for storing the value pointer for each
function node. These are used to later extract values during the Output
stage.
Sam deserves all of the credit for realizing both of these points and
convincing me to make the change! It worked out great, cheers!
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
This makes some of the Graph sig changes to prepare the code for proper
functions. The remaining bits will happen later.
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
If our machine has that pipe busy, don't panic the test. (We do want it
to fail though!)
We're also more careful to check nil object just as a convenience to
help programmers.
A back poke is the deferral or delay of a Process/CheckApply. This is
because we notice that we're not truly ready to CheckApply due to some
timestamp issue. When Process errors, we should accept that, but not
treat it as a success.
The retry and limit "satellite" event loops didn't allow pausing or
resuming, and instead you needed to wait until either was done before
you could pause.
The downside of this patch is that for very fast graph transitions, we
wouldn't be really obeying the limits anymore, however now that we have
per resource kind+name uid, we can persist the limits across graph swaps
if we want to.
Most importantly, this allows us to exit entirely when we're stuck in
one of these satellite loops.
This adds a meta state store that is preserved between graph switches if
the kind and name match. This is useful so that rapid graph changes
don't necessarily reset their retry count if they've only changed one
resource field.
This simplifies the pause mechanism and also avoids a deadlock on error.
If the Worker shuts down completely, but before we've been removed from
the graph, then an attempted pause would deadlock if we didn't have an
escape hatch here.
This removes the unnecessary ack mechanism now that we have a
synchronous channel send to represent the pausing, rather than an
asynchronous channel closing.
There's always the fear that there is either a panic or a deadlock in
the highly concurrent engine resource code. I have not seen one recently
and I've been running some pretty concurrent tests. In the meantime, and
with my hopefully improved knowledge of concurrency, I decided to
rewrite some of the "uglier" parts of the engine. I think it is a lot
clearer now, and much less likely that there is a concurrency issue.
This has been tested by running the examples/lang/fastcount.mcl example.
This adds a new test functions package and also a new "fastcount"
function which counts up from zero as fast as is possible. You probably
don't want to use this in production, but it is useful for performance
and deadlock testing the resource and function engines.
This ports the integration tests to use a standalone etcd server instead
of depending on the flaky elastic etcd clustering. Hopefully we will
polish and/or reimplement that at some point in the future, but at least
for now let's make things reliable.
Standalone etcd is useful for when we don't want to use the embedded
version to make it easier to deploy somewhere or for testing.
This pulls in about the same amount of code since we already embedded
etcd previously. Since the embedded etcd feature of mgmt is not very
stable, we'll add this for now.
When mgmt is in etcd-client-only mode and using an external etcd server,
we don't want to unset our only known endpoint since this would deadlock
our etcd client since it can't connect to anyone. This could have
happened because a plain etcd server didn't set any endpoints to follow,
and as a result we noticed it was empty and decided to use that instead.
To workaround this issue on an earlier version of mgmt, you would have
had to run:
etcdctl put /_mgmt/endpoints/etcd http://localhost:2379
to set this magic key on the initial etcd server.
There are many reasonable cases where we might want to allow a dynamic
format string. Support that situation by adding the new invariants that
are needed for those cases.
We were skipping over being fully consistent with all of the generator
invariants when running the solver. This allowed us to miss some of the
conditions that a generator might impose. Usually this caused us to be
"solved" when in fact we had an invalid program.
As per https://go.dev/blog/rebuild if we include -trimpath then we'll
not store full build paths in the build artifacts.
We still have some CGO dependencies, but we'll look into those
separately.
There were some bugs about setting resource fields that were structs
with various fields. This makes things more strict and correct. Now we
check for duplicate field names earlier (duplicates due to identical
aliases) and we also don't try and set private fields, or incorrectly
set partial structs.
Most interestingly, this also cleans up all of the resources and ensures
that each one has nicer docs and a clear struct tag for fields that we
want to use in mcl. These are mandatory now, and if you're missing the
tag, then we will ignore the field.
