The test.sh script aborts as soon as a test fails. This can save time
on the local machine, but is inconvenient in CI where an early minor
failure can mask more severe errors that will be found later.
Some users like to have their project in ~/code/mgmt/ for example, but
this is not compatible with a $GOPATH which is elsewhere. This isn't a
problem unless you need vendored directories in ~/code/mgmt/vendor/
which doesn't work because ~/code/mgmt/ isn't in $GOPATH. With this
symlink and the provided ~/bin/go wrapper, vendored directories work
exactly as expected, and we also get a local $GOPATH pointing to the
same thing. Since $GOPATH must have a src/ dir, and vendor/ must NOT,
then we symlink the two together accordingly.
An important part of this is that those who like to put everything in
$GOPATH won't be affected by any of this!
The old converged detection was hacked in code, instead of something
with a nice interface. This cleans it up, splits it into a separate
file, and removes a race condition that happened with the old code.
We also take the time to get rid of the ugly Set* methods and replace
them all with a single AssociateData method. This might be unnecessary
if we can pass in the Converger method at Resource construction.
Lastly, and most interesting, we suspend the individual timeout callers
when they've already converged, thus reducing unnecessary traffic, and
avoiding fast (eg: < 5 second) timers triggering more than once if they
stay converged!
A quick note on theory for any future readers... What happens if we have
--converged-timeout=0 ? Well, for this and any other positive value,
it's important to realize that deciding if something is converged is
actually a race between if the converged timer will fire and if some
random new event will get triggered. This is because there is nothing
that can actually predict if or when a new event will happen (eg the
user modifying a file). As a result, a race is always inherent, and
actually not a negative or "incorrect" algorithm.
A future improvement could be to add a global lock to each resource, and
to lock all resources when computing if we are converged or not. In
practice, this hasn't been necessary. The worst case scenario would be
(in theory, because this hasn't been tested) if an event happens
*during* the converged calculation, and starts running, the exit command
then runs, and the event finishes, but it doesn't get a chance to notify
some service to restart. A lock could probably fix this theoretical
case.
The error seen is:
req.Cancel undefined
(type *http.Request has no field or method Cancel)
And is possibly the result of something etcd folks changed. Sadly, the
fast building 1.4.x won't work at the moment :(
This might not be fully correct, but it seems to be accurate so far. Of
particular note, the vertex order needs to be deterministic for this
algorithm, which isn't provided by a map, since golang intentionally
randomizes it. As a result, this also adds a sorted version of
GetVertices called GetVerticesSorted.
Sorry for the size of this patch, I was busy hacking and plumbing away
and it got out of hand! I'm allowing this because there doesn't seem to
be anyone hacking away on parts of the code that this would break, since
the resource code is fairly stable in this change. In particular, it
revisits and refreshes some areas of the code that didn't see anything
new or innovative since the project first started. I've gotten rid of a
lot of cruft, and in particular cleaned up some things that I didn't
know how to do better before! Here's hoping I'll continue to learn and
have more to improve upon in the future! (Well let's not hope _too_ hard
though!)
The logical goal of this patch was to make logical grouping of resources
possible. For example, it might be more efficient to group three package
installations into a single transaction, instead of having to run three
separate transactions. This is because a package installation typically
has an initial one-time per run cost which shouldn't need to be
repeated.
Another future goal would be to group file resources sharing a common
base path under a common recursive fanotify watcher. Since this depends
on fanotify capabilities first, this hasn't been implemented yet, but
could be a useful method of reducing the number of separate watches
needed, since there is a finite limit.
It's worth mentioning that grouping resources typically _reduces_ the
parallel execution capability of a particular graph, but depending on
the cost/benefit tradeoff, this might be preferential. I'd submit it's
almost universally beneficial for pkg resources.
This monster patch includes:
* the autogroup feature
* the grouping interface
* a placeholder algorithm
* an extensive test case infrastructure to test grouping algorithms
* a move of some base resource methods into pgraph refactoring
* some config/compile clean ups to remove code duplication
* b64 encoding/decoding improvements
* a rename of the yaml "res" entries to "kind" (more logical)
* some docs
* small fixes
* and more!
I didn't know this was possible until I was browsing through some golang
docs recently. This should hopefully make it clearer which the common
methods to all resources are (which don't need to be reimplemented each
time) and which ones are unique and need to be created for each
resource.
This lets some golint errors in, but fails if you're over a certain
threshold. The current threshold of 15% (of LOC) is arbitrary and
subject to change. The algorithm should be extended to check a range of
commits, although it's unclear how to detect what range of commits make
up a patch set.
This is a monster patch that finally gets the iterative pkg auto edges
working the way they should. For each file, as soon as one matches, we
don't want to keep add dependencies on other file objects under that
tree structure. This reduces the number of necessary edges considerably,
and allows the graph to run more concurrently.
This avoids code duplication since we need to reuse this logic
elsewhere, and as well this adds support for resolving the mapping for
multiple numbers of packages at the same time.
Note that on occasion, InstallPackages has hung waiting for Finished or
Destroy signals which never came! Increasing the PkBufferSize seemed to
help, but it's not clear if this is a permanent fix, or if more advanced
debugging and/or work arounds are needed. Is it possible that not
*waiting* to receive the destroy signals is causing this?
By adding the "kind" to the base resource, it is still identifiable even
when the resource specific elements are gone in calls that are only
defined in the base resource. This is also more logical for building
resources!
This also switches resources to use an Init() method. This will be
useful for when resources have more complex initialization to do.
This allows for resources to automatically add necessary edges to the
graph so that the event system doesn't have to work overtime due to
sub-optimal execution order.
I added a regression to the file resource. This was caused by two
different bugs that I added when I switched the API to use checkapply. I
would have caught these issues, except my test cases *also* had a bug! I
think I've fixed all three issues now.
Lastly, when running on travis, the tests behave very differently! Some
of the tests actually fail, and it's not clear why. As a result, we had
to disable them! I guess you get what you pay for.
This used to be GetType(), but since now things are "resources", we want
to know what "kind" they are, since asking what "type" they are is
confusing, and makes less logical sense than "Kind".
This is based on PackageKit, which means events, *and* we automatically
get support for any of the backends that PackageKit supports. This means
dpkg, and rpm are both first class citizens! Many other backends will
surely work, although thorough testing is left as an exercise to the
reader, or to someone who would like to write more test cases!
Unfortunately at the moment, there are a few upstream PackageKit bugs
which cause us issues, but those have been apparently resolved upstream.
If you experience issues with an old version of PackageKit, test if it
is working correctly before blaming mgmt :)
In parallel, mgmt might increase the testing surface for PackageKit, so
hopefully this makes it more robust for everyone involved!
Lastly, I'd like to point out that many great things that are typically
used for servers do start in the GNOME desktop world. Help support your
GNOME GNU/Linux desktop today!
This simplifies the API, and reduces code duplication for most
resources. It became obvious when writing the pkg resource, that the two
operations should really be one. Hopefully this will last! Comments
welcome.
This is very important work that is doubly hard because the API isn't
stable yet. If you see felix, buy him a beverage.
PS: Sorry felix that I just broke the api. I'll send you the patch to
fix it!
