Sorry for the size of this patch, I was busy hacking and plumbing away
and it got out of hand! I'm allowing this because there doesn't seem to
be anyone hacking away on parts of the code that this would break, since
the resource code is fairly stable in this change. In particular, it
revisits and refreshes some areas of the code that didn't see anything
new or innovative since the project first started. I've gotten rid of a
lot of cruft, and in particular cleaned up some things that I didn't
know how to do better before! Here's hoping I'll continue to learn and
have more to improve upon in the future! (Well let's not hope _too_ hard
though!)
The logical goal of this patch was to make logical grouping of resources
possible. For example, it might be more efficient to group three package
installations into a single transaction, instead of having to run three
separate transactions. This is because a package installation typically
has an initial one-time per run cost which shouldn't need to be
repeated.
Another future goal would be to group file resources sharing a common
base path under a common recursive fanotify watcher. Since this depends
on fanotify capabilities first, this hasn't been implemented yet, but
could be a useful method of reducing the number of separate watches
needed, since there is a finite limit.
It's worth mentioning that grouping resources typically _reduces_ the
parallel execution capability of a particular graph, but depending on
the cost/benefit tradeoff, this might be preferential. I'd submit it's
almost universally beneficial for pkg resources.
This monster patch includes:
* the autogroup feature
* the grouping interface
* a placeholder algorithm
* an extensive test case infrastructure to test grouping algorithms
* a move of some base resource methods into pgraph refactoring
* some config/compile clean ups to remove code duplication
* b64 encoding/decoding improvements
* a rename of the yaml "res" entries to "kind" (more logical)
* some docs
* small fixes
* and more!
This used to be GetType(), but since now things are "resources", we want
to know what "kind" they are, since asking what "type" they are is
confusing, and makes less logical sense than "Kind".
Naming the resources "type" was a stupid mistake, and is a huge source
of confusion when also talking about real types. Fix this before it gets
out of hand.
Every graph needs each vertex to have a change to run initially (after
it has started up) so that initial state detection can be applied to
fix anything that happened while the program was not running. We used to
poke every vertex which was unnecessary, when in fact we only need to
poke the set of vertices that are the minimum set of ultimate
pre-requisites for every other vertex in the graph. That way, you're
either poked directly, or poked by someone who was, etc...
It turns out we don't need Dilworth's theorem, and that looking at
vertices with an indegree of 0 is enough (I think it is a special case
when we have a DAG).
This also fixes a goroutine start scheduling race by ensuring the
initial pokes are received!
This required a change in the event system to add an "activity" field.
This is meant to be generic in the case that there is more than one need
for it, but at the moment, allows a poke to tell that it is a poke in
response to an apply that just finished, instead of a regular poke or
backpoke in which all that matters is timestamp updates, because there
wasn't any actual work done (since that state was okay).
* Fixup graph state readability
* Rename original SetState() to SetConvergedState() and friends...
* Add type state management for proper BackPoke() commands...
* Add better DEBUG logging
This is an important optimization that prevents running a BackPoke on a
parent which is in the process of running and will most certainly poke
the caller back in a moment. This avoids unnecessary roundtrips.
Unfortunately, there are still other algorithms required so that races
can't cause the graph to run for longer than necessary.
* Add exec type
* Switch erroneous use of fmt to log instead
* Check for edge existence for safety before using
* Avoid recalling etcd channel maker
* Clean up logging output
This is the third main feature of this system. The code needs a bunch of
polish, but it actually all works :)
I've tested this briefly with N <= 3.
Currently you have to build your own etcd cluster. It's quite easy, just
run `etcd` and it will be ready. I usually run it in a throw away /tmp/
dir so that I can blow away the stored data easily.
This requires graphviz to be installed on your machine. If you run the
command with sudo, it will create the files with the original user
ownership to make it easier to remove them without root.
This is still a dirty prototype, so please excuse the mess. Please
excuse the fact that this is a mega patch. Once things settle down this
won't happen any more.
Some of the changes squashed into here include:
* Merge vertex loop with type loop
(The file watcher seems to cache events anyways)
* Improve pgraph library
* Add indegree, outdegree, and topological sort with tests
* Add reverse function for vertex list
* Tons of additional cleanup!
Amazingly, on my first successful compile, this seemed to run!
A special thanks to Ira Cooper who helped me talk through some of the
algorithmic decisions and for his help in finding better ones!
This is a prototype that i'm attempting to "release early". Expect a lot
of changes! It is intended to be a config management tool that will:
* be event based
* execute actions in parallel
* function as a distributed system
There are a bunch more design ideas going into this, please stay tuned!
