This was really not ever tested properly, and I worry it will deadlock.
It definitely kicks off falls positives that don't even do any harm as
far as we can tell.
There was and still is a bunch of terrible mess in this code. This does
some initial cleanup, and also fixes an important bug!
If you're provisioning a vmhost from scratch, then the function engine
might do some work to get the libvirt related services running before
the virt resource is used to build a vm. Since we had connection code in
Init() it would fail if it wasn't up already, meaning we'd have to write
fancy mcl code to avoid this, or we could do this refactor and keep
things more logical.
These are expected from our engine. We do care about timeout's and so
on. This allows os to return ctx.Err() whenever a <-ctx.Done() happens,
which is more idiomatic for what we really want, but which we weren't
thorough with before.
This mega patch primarily introduces a new function engine. The main
reasons for this new engine are:
1) Massively improved performance with lock-contended graphs.
Certain large function graphs could have very high lock-contention which
turned out to be much slower than I would have liked. This new algorithm
happens to be basically lock-free, so that's another helpful
improvement.
2) Glitch-free function graphs.
The function graphs could "glitch" (an FRP term) which could be
undesirable in theory. In practice this was never really an issue, and
I've not explicitly guaranteed that the new graphs are provably
glitch-free, but in practice things are a lot more consistent.
3) Simpler graph shape.
The new graphs don't require the private channels. This makes
understanding the graphs a lot easier.
4) Branched graphs only run half.
Previously we would run two pure side of an if statement, and while this
was mostly meant as an early experiment, it stayed in for far too long
and now was the right time to remove this. This also means our graphs
are much smaller and more efficient too.
Note that this changed the function API slightly. Everything has been
ported. It's possible that we introduce a new API in the future, but it
is unexpected to cause removal of the two current APIs.
In addition, we finally split out the "schedule" aspect from
world.schedule(). The "pick me" aspects now happen in a separate
resource, rather than as a yucky side-effect in the function. This also
lets us more precisely choose when we're scheduled, and we can observe
without being chosen too.
As usual many thanks to Sam for helping through some of the algorithmic
graph shape issues!
Clear out lots of cruft and mistakes in this old API. The language GAPI
isn't updated with this commit, and as a result, this won't build, but
the fixes for it are coming shortly. We could have merged the two
commits, but it's easier to show them separately for clarity.
The nil type is being added, but only slightly. It is not meant for real
use in the language. It will not work in all situations, and it isn't
implemented for many methods.
It is being added only for being a "dummy" placeholder value in the
engine when we have an unused value being propagated along an edge.
This is a bit tricky, and we should nuke and redo some of this API. The
sneaky bit has to do with whether we've already added the namespace
magic into our etcd client or not.
We want the pattern to be key:0, val:0, key:1, val:1, and so on... This
was previously using 0,1,2,3...
When we use Call directly, we need to fix this. Previously this was dead
code which is why the bug wasn't caught.
There's no reason we need to remake these two channels, when we can just
use one. We should probably rewrite this code entirely, but at least we
get rid of this race for now.
This removes the secret channel from the call function. Having it made
it more complicated to write new function engines, and it's not clear
why it was even needed in the first place. It seems that even the
current generation of function engines work just fine without it.
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
It's common in many concurrent engines to have a situation where we
collect errors on shutdown. Errors can either because a context closed,
or because some engine error happened. The latter, can also cause the
former, leading to a list of returned errors. In these scenarios, we
want to filter out all the secondary context errors, unless that's all
that's there. This provides a helper function to do so.
We should consider if it's possible to avoid all of this transforming
entirely, but at least for now, do it all in one place by having an
available helper.
This started because it was possible, not because it was very useful.
The overhead of using the full function API, is lessened by the function
API helpers, and the upcoming improvements in the function API.
It's much easier to have one fewer API's to manage and so on.
It's also a stark reminder of how weak tools like "puppet" are which
only really have data collection systems that don't take arguments.
If the user is logged in, and we try to change from /home/james to
/home/james/ we'll get the error:
usermod: user james is currently used by process ????
and furthermore, it makes no sense to try and make this change since the
usermod function won't do anything if you run:
usermod --gid james --groups wheel --home /home/james/ james
when /etc/passwd has /home/james as the string.
The provisioner should be able to encrypt things. We should use an empty
passphrase so that the choosing of the actual passphrase can be done at
first boot.
