Commit Graph

272 Commits

Author SHA1 Message Date
Jonathan Gold
62ca12608d cli: Add license flag
This patch adds the option to print the license with a cli flag. It
uses go-bindata to store the license file. The file is generated by
running `make bindata` and the result is stored in the bindata
directory.
2017-12-08 00:57:58 -05:00
Guillaume Herail
ac629404f4 test: Switch to goimports instead of gofmt
see https://github.com/purpleidea/mgmt/pull/256#issuecomment-346360414
2017-11-25 06:49:00 -05:00
Guillaume Herail
3575d597f7 resources: Add User/Group to ExecRes 2017-11-24 10:38:16 -05:00
James Shubin
846c5f8762 test: Add another check for off-by-one-error commit tags 2017-11-24 09:46:32 -05:00
James Shubin
52fd1ae73e test: Add check for common doc vs docs ambiguity 2017-11-23 08:20:44 -05:00
James Shubin
51ce83f20b test: Add extra commit message tests for some common mistakes
Feel free to add more if we identify them.
2017-11-21 11:05:20 -05:00
James Shubin
81daf10157 test: Fix linter issues
These are some linter issues that were found in a new version of the
linter. Let's fix them now before that linter hits our test suite.
2017-09-26 19:38:53 -04:00
James Shubin
0df4824a56 test: Increase timeouts for slow travis
Should prevent more intermittent failures.
2017-09-09 15:31:06 -04:00
James Shubin
653c76709a test: Fix another intermittent failure
Some of the tests had very precise timeouts, which weren't very
important. Here's another one that timed out early.
2017-09-04 16:39:01 -04:00
James Shubin
6c8588c019 test: Increase timeouts because travis is slow
Should hopefully prevent some intermittent failures.
2017-09-04 13:02:05 -04:00
James Shubin
0edba74091 etcd: Bump to version 3.2.6 and update all the grpc deps
Note: When go-grpc-prometheus was in the main $gopath (even at this
version) and everyone else was where they always were in vendor/ this
didn't build! It gave errors like:

	have SendHeader("github.com/purpleidea/mgmt/vendor/google.golang.org/grpc/metadata".MD) error
	want SendHeader("google.golang.org/grpc/metadata".MD) error

and I got frustrated. Putting it "next" to the other vendored deps seems
to have fixed this. Where are the golang docs that explain this
phenomenon?

This also requires golang 1.8+ as that is a requirement for etcd. It's
probably a reasonable thing for us too.

Note the older versions of etcd had some bugs with the concurrency
package and other things, so this is a necessary bump.
2017-08-30 14:16:02 -04:00
James Shubin
0dadf3d78a resources: Add NewNamedResource helper
This makes the common pattern of NewResource, SetName, easier. It also
makes it less likely for you to forget to use SetName.
2017-06-17 18:09:49 -04:00
James Shubin
a8bbb22fe8 resources: Fix golint issues
Including a trick to get the golinter to allow our compact code!
2017-06-08 04:38:25 -04:00
James Shubin
f1db088af4 test: Don't be noisy when running cd during testing 2017-06-08 01:05:58 -04:00
James Shubin
9f5057eac7 resources: Do not panic on autogrouped graph switches
Graph changes from autogrouped -> not autogrouped or vice versa cause a
panic (or I assume a leak) because we compared the auto grouped graph to
the ungrouped one, which would cause an Exit on an unstarted Vertex.
This includes a test that seems to reliably reproduces the issue.
2017-06-08 01:05:58 -04:00
James Shubin
b8ff6938df resources: Unify resource creation and kind setting
This removes the duplication of the kind string and cleans up things for
resource creation.
2017-06-07 03:07:02 -04:00
James Shubin
28a6430778 test: Add gometalinter to our test suite
Add a bunch of new linters to our tests! We can uncomment each sub
linter as we fix up the few remaining issues.
2017-06-03 02:04:10 -04:00
James Shubin
6e4157da35 test: Remove debugging echo from go vet test
I accidentally left it in which totally defeats the point of tests!
2017-06-03 01:34:02 -04:00
James Shubin
9cbaa892d3 gapi: Allow the GAPI implementer to specify fast and exit
This allows the implementer of the GAPI to specify three parameters for
every Next message sent on the channel. The Fast parameter tells the
agent if it should do the pause quickly or if it should finish the
sequence. A quick pause means that it will cause a pause immediately
after the currently running resources finish, where as a slow (default)
pause will allow the wave of execution to finish. This is usually
preferred in scenarios where complex graphs are used where we want each
step to complete. The Exit parameter tells the engine to exit, and the
Err parameter tells the engine that an error occurred.
2017-06-02 04:03:10 -04:00
James Shubin
9531465410 test: Make sure our examples build
Since there are occasional API changes, I'd like to at least remember to
keep the examples building, so we now have a test to remind us!
2017-06-02 03:32:53 -04:00
James Shubin
0545c4167b pgraph: Remove NewVertex and NewEdge methods and fix examples
Since the pgraph graph can store arbitrary pointers, we don't need a
special method to create the vertices or edges as long as they implement
the String() string method. This cleans up the library and some of the
examples which I let rot previously.
2017-05-31 18:04:58 -04:00
James Shubin
1fbe72b52d test: Run go vet across whole packages not individual files
The golang tooling is quite deficient, in that it makes it quite
difficult to get the tools to do_the_right_thing, without ample wrapping
of bash scripting. Go vet was finding issues because it didn't have the
full context available. Hopefully this package level context is
sufficient for now. It still lacks inter-package context though.
2017-05-29 15:43:50 -04:00
James Shubin
f4bb066737 test: Run go vet with -source flag in newer releases
This should hopefully eliminate some false positives.
https://github.com/golang/go/issues/20514
2017-05-29 15:43:50 -04:00
James Shubin
d11854f4e8 pgraph: Clean up pgraph module to get ready for clean lib status
The graph of dependencies in golang is a DAG, and as such doesn't allow
cycles. Clean up this lib so that it eventually doesn't import our
resources module or anything else which might want to import it.

This patch makes adjacency private, and adds a generalized key store to
the graph struct.
2017-05-13 13:28:41 -04:00
James Shubin
a4858be967 lib, gapi: Next method of GAPI should generate first event
This puts the generation of the initial event into the Next method of
the GAPI. If it does not happen, then we will never get a graph. This is
important because this notifies the GAPI when we're actually ready to
try and generate a graph, rather than blocking on the Graph method if we
have a long compile for example.

This is also required for the etcd watch cleanup.
2017-04-10 03:20:58 -04:00
James Shubin
66d9c7091c lib: examples: Update to most recent API
At some point in the past the API changed. Fixed now.
2017-04-10 03:20:58 -04:00
Julien Pivotto
33d20ac6d8 prometheus: Add detailed metrics
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2017-03-16 14:18:46 +01:00
James Shubin
cd5e2e1148 pgraph: Add fast pausing and exiting of graphs
This causes a graph to actually stop processing part way through, even
if there are poke's that want to continue on. This is so that the user
experience of pressing ^C actually causes a shutdown without finishing
the graph execution. It might be preferred to have this be a user
defined setting at some point in the future, such as if the user presses
^C twice. As well, we might want to implement an interrupt API so that
individual resource execution can be asked to bail out early if
requested. This could happen on a third ^C press.
2017-03-13 07:54:03 -04:00
James Shubin
91af528ff8 pgraph: Move the quiesce done indicator to avoid deadlock
This avoids a deadlock on resource failure when retry==0. Without this
we would never exit. This adds a test in too!
2017-03-12 13:52:35 -04:00
James Shubin
8ff048d055 test: Disable prometheus-3.sh test temporarily
It seems to be failing, and I'm not sure where the regression is, or if
there is a race. Sorry roidelapluie.
2017-03-09 11:46:21 -05:00
James Shubin
22b48e296a resources, yamlgraph: Drop the kind capitalization
This stopped making sense now that we have a resource with two primary
capitals. It was just a silly formatting hack anyways. Welcome kv!
2017-03-09 02:50:55 -05:00
James Shubin
5f060d60a7 test: Avoid matching three X's
This helps my "WIP" detector script avoid false positives. It is a
simple script which helps me find release critical problems.
2017-03-01 22:37:08 -05:00
James Shubin
d8e19cd79a semaphore: Create a semaphore metaparam
This adds a P/V style semaphore mechanism to the resource graph. This
enables the user to specify a number of "id:count" tags associated with
each resource which will reduce the parallelism of the CheckApply
operation to that maximum count.

This is particularly interesting because (assuming I'm not mistaken) the
implementation is dead-lock free assuming that no individual resource
permanently ever blocks during execution! I don't have a formal proof of
this, but I was able to convince myself on paper that it was the case.

An actual proof that N P/V counting semaphores in a DAG won't ever
dead-lock would be particularly welcome! Hint: the trick is to acquire
them in alphabetical order while respecting the DAG flow. Disclaimer,
this assumes that the lock count is always > 0 of course.
2017-02-27 02:57:06 -05:00
James Shubin
757cb0cf23 test: Small fixups to t4 and a rename 2017-02-26 20:54:22 -05:00
Julien Pivotto
7d92ab335a prometheus: Add mgmt_pgraph_start_time_seconds metric
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2017-02-26 15:28:43 +01:00
James Shubin
8be09eadd4 test: Fix up probable timeout failures due to slow ci
We had *occasional* failures most likely due to slow ci and compounded
by low entropy. We also weren't pointing at the right test!
2017-02-22 23:08:09 -05:00
James Shubin
98bc96c911 golint: Fixup issues found in the report
This also increases the max allowed to 5% -- I'm happy to make this
lower if someone asks.
2017-02-22 22:18:55 -05:00
James Shubin
3bd37a7906 test: Don't block on graph transitions
Improvements in the engine have uncovered some annoying race conditions
which would cause the engine to block between transitions. This is a
test which catches the most obvious file based ones.

This requires inotify to work in the test environment.
2017-02-22 17:45:16 -05:00
James Shubin
18ea05c837 pgraph, resources: Add proper start/stop signals
We need to perform some operations in lock step between graph
transitions. This should help with that!
2017-02-21 18:48:27 -05:00
James Shubin
bec7f1726f resources: virt: Allow hotplugging
This allows hot (un)plugging of CPU's! It also includes some general
cleanups which were necessary to support this as well as some other
features to the virt resource. Hotunplug requires Fedora 25.

It also comes with a mini shell script to help demo this capability.

Many thanks to pkrempa for his help with the libvirt API!
2017-02-21 18:42:07 -05:00
James Shubin
74dfb9d88d test: Make test status more clear 2017-02-21 18:40:31 -05:00
James Shubin
02dddfc227 test: Fix yamlfmt test
Last chance before we kill this entirely.
2017-02-21 16:16:41 -05:00
James Shubin
a601115650 test: Fix false negative on go vet
This was my fault, now it is fixed :) It passed locally due to a bug.
2017-02-20 18:12:30 -05:00
James Shubin
ac142694f5 test: Improve go vet so that it is less noisy 2017-02-20 17:08:48 -05:00
James Shubin
69b0913315 test: Fix tests by hooking up go test properly
The internal golang tests broke when we turned everything into packages.
This resurrects them with the hopes that we'll add more!
2017-02-20 16:40:40 -05:00
Mildred Ki'Lya
d0d62892c8 resources: file: Allow creation of empty directories 2017-02-16 20:41:59 -05:00
Julien Pivotto
be5040e7a8 prometheus: Remove mgmt_process_start_time_seconds metric
That metric is useless as by default the prometheus golang client
provides the `process_start_time_seconds` metric.

This reverts commit 25e2af7c89.
2017-02-14 22:56:12 +01:00
Julien Pivotto
25e2af7c89 prometheus: Add mgmt_process_start_time_seconds metric 2017-02-14 22:14:59 +01:00
James Shubin
e9adbf18d3 test: prometheus: Fix up test 2017-02-14 12:10:54 -05:00
Sean Jones
610202097a test: Enable macOS shell testing
* Check and install libvirt with Homebrew

  macOS does not have apt, dnf or yum. Add checking for homebrew for
  installing libvirt.

* Use platform timeout for tests
    * Add timeout detection to test/util.sh
    * Use $timeout for shell test requiring timeout
2017-02-14 11:59:44 -05:00