lourenco/mgmt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: lourenco:0.0.17
Branches Tags
lourenco:master lourenco:feat/cloudflaredns
lourenco:0.0.27 lourenco:0.0.26 lourenco:0.0.25 lourenco:0.0.24 lourenco:0.0.23 lourenco:0.0.22 lourenco:0.0.21 lourenco:0.0.20 lourenco:0.0.19 lourenco:0.0.18 lourenco:0.0.17 lourenco:0.0.16 lourenco:0.0.15 lourenco:0.0.14 lourenco:0.0.13 lourenco:0.0.12 lourenco:0.0.11 lourenco:0.0.10 lourenco:0.0.9 lourenco:0.0.8 lourenco:0.0.7 lourenco:0.0.6 lourenco:0.0.5 lourenco:0.0.4 lourenco:0.0.3 lourenco:0.0.2 lourenco:0.0.1
...
compare: lourenco:0.0.21
Branches Tags
lourenco:feat/cloudflaredns lourenco:master
lourenco:0.0.27 lourenco:0.0.26 lourenco:0.0.25 lourenco:0.0.24 lourenco:0.0.23 lourenco:0.0.22 lourenco:0.0.21 lourenco:0.0.20 lourenco:0.0.19 lourenco:0.0.18 lourenco:0.0.17 lourenco:0.0.16 lourenco:0.0.15 lourenco:0.0.14 lourenco:0.0.13 lourenco:0.0.12 lourenco:0.0.11 lourenco:0.0.10 lourenco:0.0.9 lourenco:0.0.8 lourenco:0.0.7 lourenco:0.0.6 lourenco:0.0.5 lourenco:0.0.4 lourenco:0.0.3 lourenco:0.0.2 lourenco:0.0.1

243 Commits
0.0.17 ... 0.0.21

Author SHA1 Message Date
James Shubin
6a7d904fae misc: Improve tagging script 
This way we can push the tag *after* all the builds succeed. If
something goes wrong, we can always delete our local tag and try again.
 2019-10-04 06:49:04 -04:00
James Shubin
d4043d3f86 misc, make: Add full file path into fpm script 
This is needed for our fancier, unique file names.
 2019-10-04 06:43:17 -04:00
James Shubin
b4902a4f58 make: Add a unique token to the package file name 
This unique token is necessary so that storing the files in the same dir
(basically a GitHub release) or in the SHA1SUMS file doesn't cause a
conflict.
 2019-10-04 06:06:44 -04:00
James Shubin
ffe402f201 misc: Add fedora-30 mkosi+fpm build environment 
Good example of how to add a new distro or version.
 2019-10-04 06:02:08 -04:00
James Shubin
09cc7da282 misc: Add proper archlinux prefix in build script 2019-10-04 06:01:23 -04:00
James Shubin
2d2dad41f4 todo: Update the TODO file so that it has a sane purpose 
We stored some stuff in GitHub, and some stuff here. We can keep using
this, but let's do it for the stuff that hasn't changed in a while.
 2019-10-04 04:11:26 -04:00
bjanssens
5f7c0a86dd art: Add the requested art 
Signed-off-by: bjanssens <bjanssens@inuits.eu>
 2019-10-04 09:23:20 +02:00
Donald Bakong
fc1c631c98 engine: resources: Change Res API from Compare to Cmp 
This will be done by refactoring the current method, to return an error
message instead of a boolean value. This will also update a typo on the
user res.
 2019-09-27 18:10:58 -04:00
James Shubin
89bdafacb8 misc: Refactor Makefile slightly 
We could make this even better in the future with lists.
 2019-09-23 06:57:26 -04:00
James Shubin
73b6b3f129 misc: Remove old image building cruft 2019-09-23 06:50:26 -04:00
James Shubin
b2a495f593 misc: Add mkosi target for ubuntu bionic 
The name of these is pretty weird.
 2019-09-23 06:50:00 -04:00
James Shubin
65ee904377 misc: Work around old golang in ubuntu 
Hopefully this helps.
 2019-09-23 06:48:55 -04:00
James Shubin
13f59230b5 misc: Split Makefile PHONY target into multiple lines 
AIUI this is valid make. Please correct me if I'm wrong.
 2019-09-23 06:48:55 -04:00
James Shubin
36d2a0de1e misc: Make mkosi building suitable for different distro versions 
We'd like to be able to build both Fedora N and N-1 at the same time if
possible. This makes it more generally applicable for this scenario, as
well as for other distros.
 2019-09-23 06:48:55 -04:00
James Shubin
a4db9fc8e5 misc: Add mkosi based package building with fpm 
Building distro packages is great, however if they aren't built in the
correct environment with the associated dependencies, then they won't
work properly on those distros.

This patch adds an `mkosi` based image building environment that builds
the packages in their respective distros, and then copies them out into
our releases directory.

You'll now want to `make tag && make mkosi && make release` to get a new
release out. We use a small hack to trick the `make release` portion to
not re-build the distro packages if they're already present in the
releases/ directory for that version.

This commit depends on a very recent version of mkosi (it was tested
with git master) and also depends on two currently unmerged patches:
https://github.com/systemd/mkosi/pull/363 and
https://github.com/systemd/mkosi/pull/365
 2019-09-20 12:32:41 -04:00
James Shubin
9dae5ef83b engine: resources: Improve the file res and add strict state 
This might be slightly controversial, in that you must specify the state
if a file would need to be created to perform the action. We no longer
implicitly assume that just specifying content is enough. As it turns
out, I believe this is safer and more correct. The code to implement
this turns out to be much more logical and simplified, and does this
removes an ambiguous corner case from the reversed resource code.

Some discussion in: https://github.com/purpleidea/mgmt/issues/540

This patch also does a bit of related cleanup.
 2019-09-14 16:07:53 -04:00
James Shubin
e8842a740c lang: Remove duplicate log message 
Looks like we had two copies of the same message by accident.
 2019-09-11 04:26:15 -04:00
James Shubin
0d3807ad09 lang, test: Fix copy paste error with log message 
This changes this to the correct error message.
 2019-09-11 04:26:15 -04:00
James Shubin
5c27a249b7 engine: resources: Add reversible API and file resource 
This adds the first reversible resource (file) and the necessary engine
API hooks to make it all work. This allows a special "reversed" resource
to be added to the subsequent graph in the stream when an earlier
version "disappears". This disappearance can happen if it was previously
in an if statement that then becomes false.

It might be wise to combine the use of this meta parameter with the use
of the `realize` meta parameter to ensure that your reversed resource
actually runs at least once, if there's a chance that it might be gone
for a while.

This patch also adds a new test harness for testing resources. It
doesn't test the "live" aspect of resources, as it doesn't run Watch,
but it was designed to ensure CheckApply works as intended, and it runs
very quickly with a simplified timeline of happenings.
 2019-09-11 03:40:22 -04:00
James Shubin
7e41860b28 docs: Add missing docs on the rewatch and realize meta params 
Sometimes it's hard to keep this in sync.
 2019-09-11 03:40:22 -04:00
James Shubin
43ff92bbe7 engine: resources: Clean up test log message 2019-09-11 03:40:22 -04:00
James Shubin
28adc7e563 engine: resource: Refactor helper functions 
Maybe we can use them in other tests too.
 2019-09-11 03:40:22 -04:00
James Shubin
9788411995 engine: resources: Add another validation check 
This simple check should prevent some silly mistakes and make the logic
easier for other parts of the code that won't have to worry about this
pattern.
 2019-09-11 03:40:22 -04:00
James Shubin
0c9e8cc50e engine: resources: Change the default file state 
The default file state should be undefined. This is important because if
a reverse scenario that doesn't specify the state gets given this
default, it will be as if it was specified explicitly, which wouldn't
necessarily be what we want. Instead, an undefined state should
implicitly cause a file to get created if there's a reason to do so,
such as if content or another attribute is specified.

Hopefully this change doesn't introduce any bugs in the CheckApply code,
if it does, then it was due to a lack of implicit file creation.
 2019-09-11 03:16:57 -04:00
James Shubin
34d572c523 engine: Improve the way we make a unique res path token 
This is needed in the state directory.
 2019-09-11 03:16:57 -04:00
James Shubin
011b496b3f engine: resources: Ensure the Kind and Name methods work 
Triple check these work after decoding, by adding a test.
 2019-09-11 03:16:57 -04:00
James Shubin
12b906eac6 engine: Refactor state dir into a separate function 
This lets us re-use it, and know the path is fixed.
 2019-09-11 03:16:57 -04:00
James Shubin
20937d05c3 engine: resources: file: Add undefined file state and validate it 
We should consider using *string instead of the empty string, but let's
keep the diff smaller for now.
 2019-09-11 03:16:57 -04:00
James Shubin
4943d37ccf engine: resources: file: Use constants for state values 
More robustness is yay!
 2019-09-11 03:16:57 -04:00
James Shubin
3a8fd215de engine: resources: file: Add Copy method to file res 
This lets us implement the CopyableRes interface.
 2019-09-11 03:16:57 -04:00
James Shubin
87572e8922 test: Catch capitalized error messages in tests 2019-09-06 03:28:49 -04:00
James Shubin
f1eedc7a01 lang: Clarify error message about missing field 
User probably just mistyped a field name. Make that clear.
 2019-09-06 03:28:49 -04:00
Donald Bakong
b79e48dd77 docs: Fix typo on quick-start-guide.md 2019-08-25 22:53:43 -04:00
James Shubin
18872194af misc: Warn users with weird computers 
A user seemed to experience a weird golang issue when they had deps from
both package managers installed. I won't block or fail their install,
but we can print a warning message so that someone sees it in their
logs.
 2019-08-23 22:10:34 -04:00
James Shubin
bafd7ba282 misc: Use apt install of apt-get where possible 
The future is now!
 2019-08-23 22:10:12 -04:00
Donald Bakong
b186481181 pgraph: Add a test for FindEdge() function 2019-08-08 00:43:26 -04:00
James Shubin
09ca6d11ad lang: funcs: Module name should be public 
For consistency with the rest of the core functions.
 2019-07-29 11:17:43 -04:00
James Shubin
e68e4e786d docs: Add newly recorded talks and blog post 2019-07-26 06:52:01 -04:00
James Shubin
ee638254c3 lang: Remove the specialized info structs 
Since this was an early form of the modern data struct, remove those and
pass in the correct data. This is also important in case we have
something more complex inside our string interpolation!
 2019-07-26 04:20:04 -04:00
James Shubin
1e678905c4 util: Fix typo 2019-07-26 04:20:04 -04:00
James Shubin
10804c4b25 lang: Improve the gapi copying 
We hit a weird bug where dirs would not get copied properly. I thought
the solution might be to add the missing dirs so they'd get a proper
mkdir, but in the end that didn't work well, so we just use `mkdirall`
and that seems to work. Let's leave it like this for now. Some of the
previous work for that is in the previous commit.
 2019-07-26 04:20:04 -04:00
James Shubin
4bf9b4d41b util: Add some path helper functions 
In the end, I'm not sure how useful these will be, but we'll keep them
in for now.
 2019-07-26 04:20:04 -04:00
James Shubin
1161872324 etcd: fs: Errors should start with lower case 2019-07-26 04:20:04 -04:00
James Shubin
98cb570896 util: Add new mkdirall variants for the copy functions 
This adds versions that recursively `mkdir` and all don't error as
easily. This works around some bugs we were having with file copying.
 2019-07-26 04:20:04 -04:00
James Shubin
ed4ee3b58e lang: funcs: Add deploy package with readfile related functions 
This adds a readfile function to actually access files from our deploy.
A fun side effect is that we can even access our own code! In general,
it's a good reminder that you should only run trusted code on your own
infrastructure. This also includes a fancy new test case.
 2019-07-26 03:38:26 -04:00
James Shubin
066048f4de lang: Pass through the Fs and the FsURI 
This should give us options as to how a function should interact with an
FS. I feel like it's cleaner to go through the World API, and passing in
the FsURI lets us do that, but I passed in the Fs at the same time in
case it's useful for some reason. I think using it is a boundary
violation, but it's just a hunch. Does anything break when we move from
one deploy to the next?
 2019-07-26 03:07:08 -04:00
James Shubin
4b6b91c08b lang: Make sure to call Init for functions that arrive via import 
We weren't calling Init on some functions which should have had this
done. I'm not sure whether this is the right place, or if it should be
elsewhere as part of the scope building process. Good enough for now.
 2019-07-22 06:49:02 -04:00
James Shubin
2980523a5b lang: Add a new function interface to accept data 
Sometimes certain internal functions might want to get some data from
the AST or from something relating to the state of the language. This
adds a method to pass in that data. For now it's a very simple method,
but we could generalize it in the future if it becomes more useful.
 2019-07-22 06:46:04 -04:00
James Shubin
f2f9c043bf lang, gapi: Work around a copy bug in the deploy 
It seems when we had a files/ dir that we added to our deploy, it would
get copied into /files/files/whatever instead of /files/whatever where
it should be. Hopefully this works around the issue forever.
 2019-07-22 06:40:47 -04:00
James Shubin
5d59cfd2c9 util: Ensure the afero copy function is working as intended 
The destination should be a dir sometimes.
 2019-07-22 06:38:02 -04:00
James Shubin
f94474e24f lang: Add the world implementation to our test suite 
This allows our tests to actual run the World API in them.
 2019-07-22 06:36:37 -04:00
James Shubin
a63fc6d9ba util: Add a remove path suffix util function 
This pairs with a similar one we already had.
 2019-07-22 06:35:13 -04:00
James Shubin
076adeef80 lang: funcs: Fix a copypasta error with the not equals operator 
Woops, sorry!
 2019-07-22 06:08:37 -04:00
James Shubin
a0e756317c lang: Add tests for slow unification 
These used to be cases where our algorithm was unusably slow.

Thanks to foxxx0 for the report!
 2019-07-21 03:15:06 -04:00
James Shubin
252cb5f2f3 lang: Detect windows style CR and return a better error 
If you get a sneaky \r in your code, the error just looks like
whitespace, so this way we can warn you explicitly.
 2019-07-21 03:10:21 -04:00
James Shubin
64288b4914 lang, test: Inline some overly indented tests 
Sometimes you're busy hacking and it's nice for future you to fix up
your code!
 2019-07-21 01:19:15 -04:00
James Shubin
9ca6c6a315 test: Split up long tests into multiple sub tests again 
I think we need this for non --race tests too.
 2019-07-21 00:55:36 -04:00
James Shubin
3651ab5c0c lang: Add more tests for function 2019-07-20 22:27:21 -04:00
James Shubin
b3f15e1ddc lang: Add more tests for class and include 2019-07-20 01:33:42 -04:00
James Shubin
da2a5f72bd lib: Update dep for uuid 
Apparently the package has moved.
 2019-07-17 04:07:24 -04:00
James Shubin
591e6b68e0 test: Split up long tests into multiple sub tests 
Hopefully this avoids the timeouts running the lang package.
 2019-07-17 02:45:04 -04:00
James Shubin
0119abdcdd travis: Try to work around CI slowdowns 
I think they throttle strangely on their garbage machines.
 2019-07-17 01:21:29 -04:00
James Shubin
e57ca15330 lang: Avoid running graphviz in tests by default 
This will help travis actually run the tests faster.
 2019-07-17 01:21:29 -04:00
James Shubin
f53376cea1 lang: Add function values and lambdas 
This adds a giant missing piece of the language: proper function values!
It is lovely to now understand why early programming language designers
didn't implement these, but a joy to now reap the benefits of them. In
adding these, many other changes had to be made to get them to "fit"
correctly. This improved the code and fixed a number of bugs.
Unfortunately this touched many areas of the code, and since I was
learning how to do all of this for the first time, I've squashed most of
my work into a single commit. Some more information:

* This adds over 70 new tests to verify the new functionality.

* Functions, global variables, and classes can all be implemented
natively in mcl and built into core packages.

* A new compiler step called "Ordering" was added. It is called by the
SetScope step, and determines statement ordering and shadowing
precedence formally. It helped remove at least one bug and provided the
additional analysis required to properly capture variables when
implementing function generators and closures.

* The type unification code was improved to handle the new cases.

* Light copying of Node's allowed our function graphs to be more optimal
and share common vertices and edges. For example, if two different
closures capture a variable $x, they'll both use the same copy when
running the function, since the compiler can prove if they're identical.

* Some areas still need improvements, but this is ready for mainstream
testing and use!
 2019-07-17 00:27:09 -04:00
James Shubin
4f1c463bdd misc: Add graphviz deps for travis 
Some tests run graphviz.
 2019-07-17 00:27:09 -04:00
James Shubin
6643a3d937 lang: Add function types to the yacc type parser 
Hopefully our type unification algorithm will be sufficiently good that
you never need to actually specify the function type, but it's useful
for testing and completeness.
 2019-07-12 16:46:08 -04:00
James Shubin
da8cb40242 lang: If the test fails earlier than expected, exit early 
If a test failed in stage 2 (fail2) instead of an expected fail in stage
3 (fail3) then it would continue running, which was an undefined
behaviour in our API. IOW we should not run Unify if SetScope failed.
This patch adds these additional checks to ensure our tests are more
robust.
 2019-07-12 16:46:08 -04:00
James Shubin
4c6d304e60 lang: types: Improve ComplexCmp function 
This improves the ComplexCmp function so that it can compare partial
types to variant types. As a result of this improvement, it actually
ended up simplifying the code significantly. This also added a test
suite for this function. This fix was important for tricky type
unification problems.
 2019-07-12 16:46:08 -04:00
James Shubin
99d3ef42e9 lang: Name the expr call graph differently 
It was wrongly named func instead of call, although this doesn't
actually matter in terms of code execution.
 2019-07-12 16:46:08 -04:00
James Shubin
e2289dc2a0 lang: funcs: Add better logging to the function engine 2019-07-12 16:46:08 -04:00
James Shubin
9b4f50cde9 lang: Add the NamedArgs interface 
This lets you specify which args are being used in the general function
API, which can make code readability and debugability slightly better.

In an ideal world, we wouldn't need this at all, but I can't figure out
how to avoid it at the moment, so we'll include it for now, as it's
always easy to delete if we find a more elegant solution.
 2019-07-12 16:46:08 -04:00
James Shubin
fe64bd9dbb lang: Move type duplicates checker into a separate package 2019-07-12 16:46:08 -04:00
James Shubin
0991264c8c lang: funcs: Use the correct arg names when running a pure func 
We were using the default argnames when the actual list of names was
available. Use these instead, and validate that we have the correct
number of them.
 2019-07-12 16:46:08 -04:00
James Shubin
3b608ad544 lang: funcs: Verify that simple polyfunc list is not empty 
This could prevent some incorrect usage of the simplepoly API.
 2019-07-12 16:46:08 -04:00
James Shubin
3f1a379908 lang: Use the short flag to list subtests 
If you want to know which test to run, it's not always obvious, so by
adding the -short flag, we'll get a listing that we can use! You'll need
to add -v as well so that the output actually displays.
 2019-07-12 16:46:08 -04:00
Adam Sigal
61a67dae29 pgraph: Add a test for FindEdge() and HasVertex() functions 
Added a new test called TestFindEdge1(). In it, a 7-node
graph is created on which to test the aforementioned functions.
 2019-07-11 22:05:13 -04:00
John Hooks
609aefd808 lib: Support for systemd STATE_DIRECTORY or XDG cache dir 
If running mgmt from a systemd unit, this enables the
STATE_DIRECTORY environment variable to be used for creating the
cache directory defined by StateDirectory= in the unit file. It
also enables the XDG_CACHE_HOME environment variable to be used.
If the user isn't root and the environment variable isn't set,
it will use the default XDG_CACHE_HOME directory.
 2019-07-08 21:19:33 -04:00
Felix Frank
191a2495a5 engine: resources: mount: Fix the dbus call for reloading systemd 
The Reload method cannot just be invoked on the administrative DBus
object. Just like the method for reloading specific units, it needs
to be invoked on the proper DBus service, addressing the proper object
and using the right interface.

Added an additional constant for the systemd DBus service. Even though
it shares the same value as the interface base name, this is
happenstance and it's technically incorrect to open a connection to an
interface name. The connection needs a service name.

Fixes #509
 2019-06-04 23:44:57 +02:00
James Shubin
a235b760dc docs: Fix typo and grammar issue 2019-05-20 09:53:19 -04:00
James Shubin
e4eb3c23a2 lang: funcs: core: Allow nested system imports 
We were passing the wrong module name for system imports. This is now
fixed, includes an example, and some tests!
 2019-05-20 09:23:28 -04:00
James Shubin
12582e963d lang: funcs: core: Make module names public 
This is needed for when we have nested modules.
 2019-05-20 08:45:43 -04:00
James Shubin
d5074871c7 examples: lang: Add a unicode example 2019-05-15 04:13:20 -04:00
James Shubin
e0d024ac95 examples: lang: Update autoedges example 
The /dev/null thing isn't needed anymore. Also make it easier to change
noop value.
 2019-05-15 03:51:01 -04:00
James Shubin
7a756cacb9 engine: graph: Add a mutex around waits map access 
If you ran some extremely absurd code, it turns out you can cause a
race. This was found by roiedelapluie experimenting! In this case, it
would panic with: fatal error: concurrent map read and map write. This
patch adds the mutex to avoid this rare race.
 2019-05-14 10:53:36 -04:00
Jan Martens
3c1da423fa engine: resources: nspawn: Trim possible systemd version suffix 2019-05-14 16:00:26 +02:00
James Shubin
38dfaa1caa docs: Update FAQ to mention go mod 2019-05-14 06:18:53 -04:00
James Shubin
a050cff50f docs: Add build issue to FAQ 
Some new users might experience this if they setup their $GOPATH
incorrectly.
 2019-05-13 07:10:36 -04:00
James Shubin
93c1b37aab lang: funcs: Add a mod function to the math package 
This should make flip-flop functions easy to write.
 2019-05-13 06:30:15 -04:00
James Shubin
01d4226c4a docs, readme: Improve new user experience 
This hopefully improves some docs for new users, and makes releases more
easily available.
 2019-05-06 07:56:38 -04:00
James Shubin
fc6032d3b7 lang: funcs: Add a weekday function to the datetime package 
This returns the day of the week. It also includes a helpful example
demonstrating how this functionality can be fun!
 2019-05-06 06:59:50 -04:00
James Shubin
43839d1090 all: Switch the --lang syntax to use argv instead 
It was a bit awkward using `mgmt run lang --lang <input>` so instead, we
now drop the --lang, and read the positional argv for the input.

This also does the same for the --yaml frontend.
 2019-05-05 11:10:47 -04:00
James Shubin
b3632584c3 etcd: Move to etcd v3.3.13 
This makes a small jump to the new etcd stable release. This isn't a
major difference, but it includes an important patch in
7814718c73149e2bbb9517cd02edb8332b621d86 which caused mgmt users to
scratch their heads, since it wasn't obvious that etcd was doing a Fatal
instead of a Panic or an error.
 2019-05-05 09:32:04 -04:00
James Shubin
e9257580cd misc: Update to golang 1.11.x 
Bump to the newer version.
 2019-05-05 09:32:04 -04:00
James Shubin
e3cc6309ea lib: Fix gofmt regression in golang 1.11.x 
Golang has many exceptions to its "compatibility promise", including the
gofmt output. The fact that they change it arbitrarily for things like
this is absurd. (Remove the patch and run `gofmt` to see for yourself.)

This change re-worked the comment, since include the `gofmt` suggested
line break makes absolutely no sense, and is not convenient.
 2019-05-05 09:32:04 -04:00
James Shubin
17fd625f7f lang: types: Workaround stringer regression in golang 1.11 
Here's a fix for another golang regression in 1.11.x which wasn't needed
before! More info in: https://github.com/golang/go/issues/31843
 2019-05-05 09:32:04 -04:00
James Shubin
d1ecfd8657 test: Fix typos, these aren't cats 
Miau!
 2019-05-05 09:32:04 -04:00
James Shubin
4aa3cfad40 lang: Add var prefix to var expr to avoid ambiguity 2019-05-05 09:32:04 -04:00
James Shubin
3bcb697662 lang: funcs: structs: Make error message more precise 
This should prevent ambiguity with other similar or identical error
messages.
 2019-05-05 09:32:04 -04:00
James Shubin
88318b73e4 lang: Print the actual stream error on test failure 
This is useful for debugging.
 2019-05-05 09:32:03 -04:00
James Shubin
2f7e202f40 lang, lang: types: Add automatic stringer generation 
It's more useful if we know the string representation of Kind's.
 2019-05-05 09:32:03 -04:00
James Shubin
310239e707 lang: types: Remove unnecessary prefix in generated kinds 
This will make displayed errors cleaner.
 2019-05-05 09:30:13 -04:00
James Shubin
4de75373dd pgraph: Use pointers for unique vertex identifiers 
This will build more accurate graphs, since we could have duplicated
vertex names for distinct vertices. This now builds the correct
topology, even if the labels are duplicated.
 2019-04-29 16:08:36 -04:00
James Shubin
c0d329e6d8 pgraph: Quote graphviz strings properly 
If strings include quotes, this previously didn't work.
 2019-04-29 16:08:36 -04:00
Johan Bloemberg
8a0840d35b lang: funcs: Add uptime implementation for macOS 2019-04-29 16:07:21 -04:00
Ward Vandewege
f9bb9ef33e docs: Fix link to puppet guide 2019-04-29 06:11:13 -04:00
Christian Rebischke
acb2a5d2b0 lang: funcs: Add ArchLinux family detection 
Signed-off-by: Christian Rebischke <chris@nullday.de>
 2019-04-25 17:05:25 +02:00
James Shubin
63ef11c708 lang: Add a new type unification test 
I wanted to make sure that the type unification algorithm restricts the
implementation of the class when included, when one of the polymorphic
types is specified with a fixed type. It seems this works! I had the
idea for this test while walking around aimlessly.
 2019-04-24 03:46:21 -04:00
James Shubin
d70bbfb5d0 lang: unification: Improve type unification algorithm 
The simple type unification algorithm suffered from some serious
performance and memory problems when used with certain code bases. This
adds some crucial optimizations that improve performance drastically.
 2019-04-23 21:21:42 -04:00
James Shubin
97d60ac98d lang: Quote printed strings 
This quotes printed strings that contain special characters such as
newline. This changes the output of some tests, but makes future tests
that include a raw \n more appropriate.
 2019-04-23 21:03:02 -04:00
Jonathan Gold
8f1f5d33fd engine: resources: mount: Restart remote-fs target 2019-04-23 16:24:49 -04:00
Wouter Dullaert
d65c85c19f cli: Removed obsolete no-watch-config flag 
Having it around creates the expectation that by default mgmt will put a watch
on the config.
 2019-04-22 13:42:27 +02:00
James Shubin
22d893fc1e test: shell: Increase etcd timeouts for slow travis again 
Increase this one too...
 2019-04-21 20:11:38 -04:00
James Shubin
806d2f6a4a lang: Fix import scoping issue with classes 
When include-ing a class, we propagated the scope of the include into
the class instead of using the correct scope that existed when the class
was defined and instead propagating only the include arguments in.

This patch fixes the issue and adds a ton of tests as well. It also
propagates the scope into the include args, in case that is needed, and
adds a test for that as well.

Thanks to Nicolas Charles for the initial bug report.
 2019-04-21 19:49:38 -04:00
James Shubin
fc3baa28d6 lang: funcs: Add regexp package and match function 
This adds a simple regexp match function. This will be useful for
regexp based name classification, if you're into that sort of thing.
 2019-04-16 21:42:32 -04:00
James Shubin
eba45e6207 lib, gapi: Display deploy ID to add some clarity 
This should make it easier to understand exactly when a new deploy
starts.
 2019-04-16 18:11:32 -04:00
James Shubin
272fd3edc3 test: shell: Increase etcd timeouts for slow travis 
We need a real test environment that's not travis.
 2019-04-16 18:08:48 -04:00
James Shubin
5ad8b33aa7 etcd: Make error more specific 
This should clarify which member remove branch we're in if we error.
 2019-04-16 18:08:39 -04:00
James Shubin
cacd14fcf8 util: Make test more resistant to races 
This doesn't guarantee which print statement runs first, so the last
part of it can race. Adding a sleep makes this highly unlikely.
 2019-04-11 21:43:48 -04:00
James Shubin
859e4749ae lib: Clean up logging 
Since most of our logging goes through a single Logf command, we don't
need the file name information any more. Our hierarchial logging is
sufficient enough.

Eventually we will replace the top-level logger with a more visually
capable logging fixture.
 2019-04-11 21:43:48 -04:00
James Shubin
a5842a41b2 etcd: Rewrite embed etcd implementation 
This is a giant cleanup of the etcd code. The earlier version was
written when I was less experienced with golang.

This is still not perfect, and does contain some races, but at least
it's a decent base to start from. The automatic elastic clustering
should be considered an experimental feature. If you need a more
battle-tested cluster, then you should manage etcd manually and point
mgmt at your existing cluster.
 2019-04-11 21:43:48 -04:00
James Shubin
fb275d9537 test: Skip net test in travis 
Travis fails intermittently, and I have no idea what's wrong with their
infra or what's using this address, so skip it here.
 2019-04-10 22:55:02 -04:00
James Shubin
88f7b7e786 docs: Add faq entry about locked binary 
Doesn't happen very often, but add it in case someone is curious or uses
search engine foo to figure out the answer.
 2019-04-05 17:50:25 -04:00
James Shubin
30402effa9 util: Add context utility functions 
This adds a utility function to close a context via a closed signalling
channel, and also functions to wrap and unwrap a wait group into and out
of a context.
 2019-04-02 15:30:34 -04:00
James Shubin
7d96623f06 util: Add safe easy ack that allows multiple ack's 
Just another sync utility to make code more readable.
 2019-04-02 07:07:36 -04:00
James Shubin
398706246e util: Add subscribed signal primitive 
Add a little sync primitive to our utility library. This should
hopefully make some of the future code easier to deal with.
 2019-04-02 07:07:36 -04:00
James Shubin
6628fc02f2 util: Add context wait signal to easy exit 
Add an alternate way to wait for a signal. This just makes code look a
bit cleaner and less cluttered.
 2019-04-02 07:07:36 -04:00
Michael Schubert
e2fa7f59a1 docs: Fix link 2019-04-02 10:33:41 +02:00
James Shubin
d5b7dc0acc github: Add funding information 2019-03-24 15:37:54 -04:00
James Shubin
e4d874cc69 engine: resources: Remove named return params 
Named return params aren't a favourite feature of mine, and they're
rarely used in the resource writing. They keep popping up because I once
used them and we've been copying and pasting code ever since. Get rid of
them all to help prevent the unnecessary spread.
 2019-03-24 15:30:02 -04:00
James Shubin
80a0abeead docs: Add FAQ entry about root requirements 2019-03-24 15:16:14 -04:00
James Shubin
0df2d46ca7 lib: Add static hello message 2019-03-24 15:11:01 -04:00
James Shubin
07f542b4d7 legal: Happy 2019 everyone... 
Done with:

ack '2018+' -l | xargs sed -i -e 's/2018+/2019+/g'

Checked manually with:

git add -p

Hello to future James from 2020, and Happy Hacking!
 2019-03-24 15:08:50 -04:00
Mitch Fossen
7db3e8556a lang, funcs: Remove deprecated syscall import 2019-03-24 14:46:58 -04:00
Felix Frank
dc03e67b81 docs: Slightly clarify parameter defaults 2019-03-24 14:44:51 -04:00
Adam Sigal
e587324b81 faq: Amended faq mailing list information 
There was outdated information concerning the mailing list. I amended
this and added a link.
 2019-03-19 14:39:13 -04:00
James Shubin
65a66492f4 docs: Move faq entry to more appropriate resource docs 2019-03-17 17:54:48 -04:00
James Shubin
17602d7065 docs: Add two faq entries 2019-03-17 17:54:48 -04:00
James Shubin
ae56261961 engine: util, resources: virt: Clean up virt resource 
Do some cleanups which were long overdue.
 2019-03-15 15:24:40 -04:00
James Shubin
c4f57608d0 test: Port yaml test to mcl 2019-03-15 13:01:50 -04:00
James Shubin
753d1104ef util: Port all multierr code to new errwrap package 
This cleans things up and simplifies a lot of the code. Also it's easier
to just import one error package when needed.
 2019-03-12 16:51:37 -04:00
James Shubin
880652f5d4 util: Port all code to new errwrap package 
This should keep things more uniform.
 2019-03-12 16:49:01 -04:00
James Shubin
54c81d6bb2 engine, pgp: Fixup incorrect error usage 
Small fixups found by the next commit.
 2019-03-12 16:49:01 -04:00
James Shubin
2bf43eae24 test: Improve test depth 
Make sure we're catching everything, including new, deeper code.
 2019-03-12 16:49:01 -04:00
James Shubin
58961d23bb test: Improve test depth 
Make sure we're catching everything, including new, deeper code.
 2019-03-12 15:50:30 -04:00
James Shubin
6044ade373 util: Add errwrap package 
Simplify working with errors across our code base. Instead of constantly
importing the necessary error helpers, assemble them all into one
package and import and use that as needed.
 2019-03-12 15:45:39 -04:00
James Shubin
da1c96c6fd examples: lang: Refresh examples 
Removed two old examples which were no longer valid.
 2019-03-09 18:54:50 -05:00
James Shubin
5bbb474db6 engine: resources: Clean up KV resource 2019-03-09 18:48:26 -05:00
James Shubin
a0c909914d lang: funcs: Don't allow interpolation in printf format string 
We'd like to pre-compute the interpolation if we can, so that we can run
this code properly... For now, we can't so it's a compile time error...
Hopefully we can remove this restriction in the future. The problem is
the string must be a constant, or it would be possible to switch it from
"%d %s" to "%s %d %d" or anything that changes the type signature.
 2019-03-09 18:06:18 -05:00
James Shubin
170e56b34a lang: Improve test case with more specific errors 2019-03-09 17:37:58 -05:00
James Shubin
de43569fa2 engine, lang: Improve send/recv significantly 
Part of this was rotten, and not fully functional. This fixes the rot,
adds some tests, and improves the type checking that occurs when sending
and receiving values. In addition, a significant portion of this happens
at compile time.

There is still more work to be done here, but this should get us a good
chunk of the way for now.
 2019-03-09 17:37:58 -05:00
James Shubin
aa6b701b77 lang: Improve the test case infra so it can detect different errors 
Let the tests detect which specific error we want to fail on.
 2019-03-09 17:14:45 -05:00
James Shubin
d69eb27557 lang: Small fixes about send/recv 2019-03-09 16:07:22 -05:00
James Shubin
0ca57d6a09 examples: lang: Add a basic file example 2019-03-09 16:05:12 -05:00
James Shubin
4c104d55cb engine: util: Add a new utility function for send/recv 
This new utility function makes verifying send/recv struct comparisons
consistent. Unfortunately it doesn't yet support coercing from *string
to string or from string to *string.
 2019-03-09 16:03:30 -05:00
James Shubin
8a8215fabe engine: util: Improve StructTagToFieldName and add tests 
This improves this function to make it more generic.
 2019-03-09 16:02:33 -05:00
James Shubin
4badeafb98 engine: resources: Add missing struct tags 
These are needed for send/recv.
 2019-03-09 16:01:25 -05:00
James Shubin
7cb79bec49 engine: resources: Replace the cached values with a live calculation 
This replaces the static obj.path and obj.isDir with live variants. I
don't know why I ever cared about caching these before, and if we ever
care we can memoize properly in the future.

This caused a small bug to actually be masked in the gob code. It is now
fixed in the previous commit.
 2019-03-06 10:08:14 -05:00
James Shubin
8da0da02d9 engine: traits: Make encoded fields public 
The fields that we might want to store with encoding/gob or any other
encoding package, need to be public. We currently don't use any of these
at the moment, but we might in the future.
 2019-03-06 10:08:14 -05:00
James Shubin
efef260764 engine: resources: Improve file Cmp function 2019-03-06 07:09:39 -05:00
James Shubin
a56991d081 engine: resources: Remove possible panic from within file res 
Not sure how I let this in, but we should never do this. Hopefully the
Validate should catch this issue in advance, and if not, at least we'll
only error.
 2019-03-06 07:09:39 -05:00
James Shubin
f0196540ab resources: file: Make some small cleanups to file res 
This does some small cleaning for consistency, since I haven't reviewed
this code in a long while.
 2019-03-06 07:09:39 -05:00
James Shubin
426b15313e engine: resources: Fix missing file when specified without content 
If the file res was defined with state => "exists" but no content
specified, it was not created. This patch fixes this bug and adds a test
and an example.
 2019-03-06 07:09:39 -05:00
James Shubin
11fc55d679 lang: funcs: Add a new test for readfile and fix a small bug 
This adds a new test for readfile. Interestingly, it actually caught a
small bug, which was also fixed with this commit. I think the bug was
actually always masked, because it only occurred on shutdown, and in
this case we often don't care about how the stream exited, but it's a
good example of how a test case focused on just one small aspect can be
important.

As an aside, this test case also would have caught the bug fixed in
94c40909cc and by reverting that patch it
indeed fails.
 2019-03-06 07:09:39 -05:00
James Shubin
de1691665f lang: funcs: Add live function stream test infrastructure 
This adds the ability to test that functions return the expected
streams, and to model this behaviour over time. This is done via a
"timeline" which runs an ordered list of actions that can both push new
values into the function stream, and wait and expect particular outputs.
Hopefully this will make our function implementations more robust!
 2019-03-06 07:09:39 -05:00
James Shubin
b1f93b40ae lang: funcs: Add runner pure func execution 
This adds a function runner that runs pure functions. It will hopefully
be useful for speculative execution of functions for compile time
determination of types.
 2019-03-05 11:42:33 -05:00
James Shubin
5e58251026 test: Improve govet log newline check 
We don't match for log.Fatalf but we shouldn't really be using that.
 2019-03-05 11:42:33 -05:00
James Shubin
4f4091a9bd engine: resources: Improve test case readability 2019-03-04 10:26:51 -05:00
James Shubin
e9fb41fdc8 test: shell: Fix rare breakage in load test 
For some reason the load is occasionally zero. This broke the regexp.
Let's see if this ever happens with the other digits.
 2019-03-04 10:16:21 -05:00
James Shubin
6b803656b2 engine: resources: Improve exec resource 
The exec resource was an early addition to the project, and it was due
for some fixes and integration into our automated tests. This patch
fixes a number of issues, and makes it ready for more general use.
 2019-03-04 10:16:21 -05:00
James Shubin
829741e2ac lang: Print a clear message on module import containing unused stmt 
If you run an import, you only include everything that's part of a
scope. This includes, variables, classes, and functions. Anything else
should cause a compile error. This cleans up the error by adding a
String() method to each Stmt in our AST.
 2019-02-28 09:35:13 -05:00
James Shubin
94c40909cc lang: funcs: Avoid erroneous empty message in readfile 
Readfile had a bug where it sent an empty string on startup. This has
ben fixed, and it now waits until the file contents are ready before
sending a string.
 2019-02-28 08:56:10 -05:00
James Shubin
95dab16e6e lang: funcs: Allow the len function to determine str length 2019-02-28 08:54:11 -05:00
James Shubin
c049413b47 examples: lang: Add is_debian and is_redhat family example 
This is just the beginning.
 2019-02-28 08:53:25 -05:00
Nicolas Charles
2d45f95501 engine: resources: print: Add RefreshOnly option 
Add option RefreshOnly (default to false) on print ressource, to print only when
notified by other resource. When a print is RefreshOnly, it can't be grouped anymore.
 2019-02-27 15:31:45 +01:00
Nicolas Charles
3cfc76b635 lang: funcs: Added a function to detect Debian and RedHat like systems 2019-02-26 18:13:34 +01:00
James Shubin
d88874845c test: shell: Improve load test 
This might have failed once in travis because of a short timeout.
Hopefully if this happens again, we'll now know why.
 2019-02-24 14:10:01 -05:00
James Shubin
5e38c1c8fe examples: Remove old hcl examples 
The hcl frontend was removed a while back. Might as well remove these
examples too.
 2019-02-24 14:10:01 -05:00
James Shubin
ae7ebeedd1 engine: resources: Add CheckApply event detection to resource tests 
This adds the ability to wait with a timeout for CheckApply happenings
in a resource. This helps avoid unnecessary long sleeping and timing
guesses. This also adds a cleanup function to run at the end.
 2019-02-24 14:10:01 -05:00
James Shubin
652b657809 resources: exec: Avoid possible deadlock race 
Some of the early code I wrote probably wouldn't pass my own reviews
today. Here's one example of a rare deadlock that could sometimes occur
when a Process/CheckApply caused a shutdown, but the bufio tried to send
on a channel that nobody was going to read any more. Now we properly
unblock that send with a context.
 2019-02-24 12:28:59 -05:00
James Shubin
62a6e0da1d misc: Add two test helpers 
Hopefully these make testing and debugging easier!
 2019-02-24 12:28:59 -05:00
James Shubin
0d0d48d9f6 test: Shell tests should use unified timeout command 2019-02-24 12:28:59 -05:00
James Shubin
ab5957f1e9 make: Clean up the Makefiles so the output is more elegant 
This avoids printing erroneous messages when nothing is actually
happening.
 2019-02-24 12:28:59 -05:00
James Shubin
463ba23003 util: Improve the sync primitives. 2019-02-24 12:28:59 -05:00
James Shubin
ccad6e7e1a test: Enable and fix up some more tests 
An unstable engine probably masked some of these issues.
 2019-02-24 12:28:59 -05:00
James Shubin
aa165b5e17 engine: Add the retry loop around Process 
This adds back the retry loop around Process. This is done as a
separate commit so you can more easily see the logic of the retry magic
This commit is similar but different to the earlier commit adding retry
around Watch.
 2019-02-24 12:28:59 -05:00
James Shubin
f06e87377c engine: Add limit delay before Process can run 
This adds back the limit delay around Process.
 2019-02-24 12:28:59 -05:00
James Shubin
4c3bf9fc7a engine: Add the retry loop around Watch 
This adds back the retry loop around Watch. This is done as a separate
commit so you can more easily see the logic of the retry magic.
 2019-02-24 12:28:59 -05:00
James Shubin
253ed78cc6 engine: Rewrite the core algorithm 
The engine core had some unfortunate bugs that were the result of some
early design errors when I wasn't as familiar with channels. I've
finally rewritten most of the bad parts, and I think it's much more
logical and stable now.

This also simplifies the resource API, since more of the work is done
completely in the engine, and hidden from view.

Lastly, this adds a few new metaparameters and associated code.

There are still some open problems left to solve, but hopefully this
brings us one step closer.
 2019-02-24 12:28:59 -05:00
James Shubin
4860d833c7 converger: Rewrite the converger module 
I found a deadlock in the converger code, and I realized the code was
sufficiently bad that it needed a good clean up.
 2019-02-24 12:28:59 -05:00
James Shubin
450d5c1a59 util: Add an easy ACK sync primitive 2019-02-24 12:28:59 -05:00
Toshaan Bharvani
88fcda2c99 lang: funcs: Added an uptime function 
Signed-off-by: Toshaan Bharvani <toshaan@vantosh.com>
 2019-02-24 12:20:58 -05:00
James Shubin
00db953c9f lang: funcs: funcgen: Clean up some small details 
Some small changes were needed, here they are. Unfortunately this only
supports the `string` type at the moment.
 2019-02-21 13:06:29 -05:00
Julien Pivotto
a0df4829a8 lang: Add more string functions, autogenerated 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2019-02-21 17:50:06 +01:00
James Shubin
b0e1f12c22 test: Add expanders when running in travis 
Hopefully this makes things more readable.
 2019-02-20 09:35:31 -05:00
James Shubin
ee56155ec4 test: Split travis tests into three blocks 
Our tests were taking near 50 minutes which kills them. This also makes
it easier to spot small issues faster.
 2019-02-20 09:35:02 -05:00
Jeff Waugh
16d7c6a933 build: Fix macOS build 
Add pkg-config to fix builds with augeas and libvirt on macOS.
 2019-02-14 23:06:18 +11:00
Johan Bloemberg
f7a06c1da9 etcd: Connection options (socket file, ipv6) 
- Allow unix domain socket to be used as client url
- Using ::1 as clienturl should not create default local ipv4 listener
- Add shell tests
 2019-02-13 18:55:20 +01:00
James Shubin
4c8086977a engine: resources: file: Update the format string 
The %s in the format string is not technically correct here.
 2019-02-08 12:38:10 -05:00
James Shubin
b1f088e5fa engine: resources: Add a test running for testing individual resources 
This adds a simulated engine that can run and test single resources. It
can't test all aspects and features that the engine supports, but is
probably pretty decent for testing the actual CheckApply and Watch
semantics. Be warned that it actually applies changes on your machine,
so please don't write tests that make undesirable changes.
 2019-02-08 12:36:37 -05:00
James Shubin
1247c789aa lang: Remove unnecessary log package 2019-02-08 10:23:44 -05:00
Johan Bloemberg
749038c76d misc: Make build on macOS work 2019-02-08 00:14:17 +01:00
Johan Bloemberg
0a052494c4 misc: Add goimports dep 2019-02-08 00:14:17 +01:00
James Shubin
90fa83a5cf lang: funcs: core: Move world API functions 
Some of the core functions interact with the remote "world" API. Move
them all into the same package.
 2019-02-07 12:32:32 -05:00
James Shubin
4eaff892c1 lang: funcs: core: Rename core module files 
More cleanup...
 2019-02-07 12:19:59 -05:00
James Shubin
f368f75209 lang: funcs: core: Drop unnecessary core prefix from imports 
This unbreaks the mcl bindata code. Of course we could change the parser
to allow this prefix, but this is cleaner. The packages still have a
core prefix, which it seems we could also remove, but this isn't
particularly important for anything.
 2019-02-07 09:33:20 -05:00
Lander Van den Bulcke
04048b13ed lang: funcs: Add strings.split function 
Signed-off-by: Lander Van den Bulcke <landervdb@inuits.eu>
 2019-02-07 10:55:39 +01:00
Lander Van den Bulcke
5acc33c751 lang: funcs: Add tests for sqrt function 
Signed-off-by: Lander Van den Bulcke <landervdb@inuits.eu>
 2019-02-06 17:11:42 +01:00
James Shubin
b449be89a7 examples: Add uncommited nspawn example 2019-02-06 08:57:11 -05:00
Lander Van den Bulcke
dac019290d lang: funcs: Add sqrt function 
Signed-off-by: Lander Van den Bulcke <landervdb@inuits.eu>
 2019-02-06 14:32:13 +01:00
Julien Pivotto
bdc424e39d lang: Add to_lower and to_upper functions 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2019-02-06 14:24:15 +01:00
Lander Van den Bulcke
10193a2796 make: Use gem --no-document instead of deprecated flags 
Signed-off-by: Lander Van den Bulcke <landervdb@inuits.eu>
 2019-02-06 12:02:10 +01:00
Julien Pivotto
2c9a12e941 docker: Update FROM to go:1.11 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2019-02-06 10:48:24 +01:00
Felix Frank
8ba6c40f0c langpuppet: Fix Cli method invocations for wrapped GAPIs 
Since the langpuppet GAPI creates fresh new CliContext objects,
it has to make sure to provide the original parent context, because
the child GAPIs expect to be able to access its data.
 2019-02-05 16:34:55 +01:00
James Shubin
bbfeb49cdf examples: Add more examples and clean up some 2019-02-04 05:03:37 -05:00
James Shubin
f61e1cb36d examples: Add missing mcl files 
I forgot to add these, sorry.
 2019-02-03 09:58:04 -05:00
James Shubin
4a3e2c3611 engine: nspawn: Add an nspawn example with an improved exec 
This adds the cwd fields to exec, better error messages to svc (which is
nested in nspawn) and a fancier nspawn example!
 2019-02-01 09:44:55 -05:00
James Shubin
81faec508c integration: Avoid duplicate events from recwatch 2019-02-01 07:58:38 -05:00
James Shubin
9966ca2e85 examples: Improve dynamic cpus virt example 2019-02-01 07:58:38 -05:00
James Shubin
35c26f9ee5 engine: resources: virt: Clean up virt resource for lang 2019-02-01 07:58:38 -05:00
James Shubin
b5e29771ab lang: funcs: Add a trim space function to the new strings module 2019-02-01 07:00:05 -05:00
James Shubin
f5f09d3640 lang: funcs: Add str2int example function 
We might want to move this into a real module eventually.
 2019-02-01 06:59:07 -05:00
James Shubin
5a531b7948 lang: funcs: Add a new readfile function 
This adds a new function that reads files from the local host.
 2019-02-01 05:20:22 -05:00
James Shubin
f716a3a73b lang: funcs: Rename template functions to remove periods 
Due to a limitation in the template library, we need to rename some
functions. It's probably worth looking into modifying this library or
finding an alternate version.
 2019-02-01 03:58:02 -05:00
James Shubin
ce8c8c8eea engine: resources: Fix a small typo in error message 2019-02-01 03:49:08 -05:00
James Shubin
fc48fda7e5 engine: resources: Fix a possible panic on closed channel 
I don't know how often it happens, but we should catch it.
 2019-02-01 03:48:24 -05:00
James Shubin
78936c5ce8 examples: lang: Update examples to fix imports and port from yaml 
Some small fixes that are useful for demos!
 2019-02-01 03:47:18 -05:00
Kevin Kuehler
5d0efce278 engine: lang: util: Kill race in socketset 
After some investigation, it appears that SocketSet.Shutdown() and
SocketSet.Close() are not synchronous operations. The sendto system call
called in SocketSet.Shutdown() is not a blocking send. That means there
is a race in which SocketSet.Shutdown() sends a message to a file
descriptor to unblock select, while SocketSet.Close() will close the
file descriptor that the message is being sent to. If SocketSet.Close()
wins the race, select is listening on a dead file descriptor and will
hang indefinitely.

This is fixed in the current master by putting SocketSet.Close() inside
of the goroutine in which data from the socket is being received. It
relies on SocketSet.Shutdown() being called to terminate the goroutine.
While this works most of the time, there is a race here. All the
goroutines can also be terminated by a closeChan. If the goroutine
receives an event (thus unblocking select) and then closeChan is
triggered, both SocketSet.Shutdown() and SocketSet.Close() race, leading
to undefined behavior.

This patch ensures the ordering of the two function calls by pulling
them both out of the goroutine and separating them with a WaitGroup.

Co-authored-by: James Shubin <james@shubin.ca>
 2019-01-22 20:59:17 -08:00
Kevin Kuehler
0c17a0b4f2 util: Add TestShutdown to socketset 
Test to ensure that SocketSet is nonblocking and will close when
SocketSet.Shutdown() is called. Create a SocketSet that will never
receive any data and leave it running in a goroutine with a WaitGroup
for a second. If Shutdown is working correctly, the goroutine will be
terminated after the timer expires.
 2019-01-22 20:59:17 -08:00
Kevin Kuehler
3f396a7c52 lang: funcs: Add cpucount fact 
Adds a CPU count fact, that can be used to determine how many CPUs are
presently on the machine and ready for use (online). We get this by
reading from a netlink socket to the kernel, and the kernel sends us
uevents when CPUs are added, removed, and brought online or offline.
Whenever one of these events are received, we look in sysfs to update
the fact's Stream with the number of online CPUs.
 2019-01-22 20:59:16 -08:00
Kevin Kuehler
8697f8f91f util: Libify socketset 
Add the ReceiveBytes, ReceiveNetlinkMessage, and ReceiveUEvent methods.
This is because not everything passed through a netlink socket cannot
reliably be parsed using the ParseNetLinkMessage function.

With the ReceiveUEvent method, we add support for "uevent" kernel
events, which updates us about the state of devices currently on the
system. To make using this method easier, we add a UEvent struct, that
has the action (what event), Devpath (where the device lives in /proc or
/sysfs), and Subsystem (what subsystem this event belows to).
 2019-01-22 20:59:16 -08:00
Kevin Kuehler
06c67685f1 util: Move socketset from net resource to util 
Prepare the socketset api to be used outside of the scope the net
resource.
 2019-01-22 20:59:11 -08:00
James Shubin
dc2e7de9e5 engine: resources: pkg: Clarify that correct state is newest 
I accidentally typed "latest" which got me confused why everything was
broken. Surprised it didn't error earlier anyways.
 2019-01-21 04:28:34 -05:00
James Shubin
db1dbe7a27 lang: Edges should allow lists of strings 
This continues the earlier patch that allowed resource names to be lists
of strings so that edges can now allow the same. This also includes a
new fancy test!
 2019-01-20 17:27:40 -05:00
James Shubin
d6bbb94be5 lang: test: Add a new giant test infra for matching static output 
This greatly expands our test infra to allow us to drop in mcl tests and
look at their resource graph output. The only downside is that this only
runs the function engine once, so if the function graph would be
constantly changing over time, then this is not a good fit here.
 2019-01-20 17:27:40 -05:00
James Shubin
e3b4c0aee3 test: Fix a small copy pasta typo 2019-01-20 17:27:40 -05:00
James Shubin
a1fbe152bb lang: unification: Fix up small typos in example code 2019-01-20 04:22:05 -05:00
James Shubin
9d28ff9b23 lang: unification: Catch unification error on typed var expr 
This was similar to the typed if expr error.
 2019-01-20 04:19:39 -05:00
James Shubin
43f0ddd25d lang: unification: Catch unification error on typed if expr 
I found a case where we had two missing unification rules. Now fixed in
the previous commits, and including this test to show I'm responsible.
I've added the same test in two locations for redundancy and as an
example.
 2019-01-20 04:19:39 -05:00
James Shubin
7a28b00d75 lang: If expression was missing two invariants 
I forgot to ensure that the type of the final expression matched the
type of each of the branches. It's rare, but possible for this to occur.
Luckily, this never would have caused a panic, because the func engine
would have caught the issue anyways, but it's still better we catch it
here first!
 2019-01-20 04:02:54 -05:00
James Shubin
32e29862f2 lang: Check that set type matches actual expression 
I forgot to include these two invariants which are occasionally
necessary, although in most cases they're necessary to prevent incorrect
code from getting past unification. In any case, they would have been
caught by the engine.
 2019-01-20 04:02:54 -05:00
James Shubin
6c5c38f5a7 lang: unification: Allow err string comparisons in tests 
Let's improve our test infra to make it more capable. It's important to
catch we failed for the _right_ reason so as to not mask the wrong
errors.
 2019-01-20 03:39:02 -05:00
James Shubin
2da7854b24 lang: unification: Add logging to make capturing errors easier 
This makes building new tests easier.
 2019-01-20 03:39:02 -05:00
James Shubin
6d0c5ab2d5 lang: unification: Add missing return to exit early 
This exits the test early, since we don't need to continue.
 2019-01-20 03:39:02 -05:00
701 changed files with 28503 additions and 7551 deletions
Expand all files Collapse all files

2
.github/FUNDING.yml vendored Normal file
View File

@@ -0,0 +1,2 @@
# You can add one username per supported platform and one custom link
patreon: purpleidea

20
.travis.yml
View File

@@ -1,10 +1,6 @@
language: go language: go
os: os:
- linux - linux
go:
- 1.10.x
- 1.11.x
- tip
go_import_path: github.com/purpleidea/mgmt go_import_path: github.com/purpleidea/mgmt
sudo: true sudo: true
dist: xenial dist: xenial
@@ -25,17 +21,27 @@ before_install:
- git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*" - git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
- git fetch --unshallow - git fetch --unshallow
install: 'make deps' install: 'make deps'
script: 'make test'
matrix: matrix:
fast_finish: false fast_finish: false
allow_failures: allow_failures:
- go: 1.11.x - go: 1.12.x
- go: tip - go: tip
- os: osx - os: osx
# include only one build for osx for a quicker build as the nr. of these runners are sparse # include only one build for osx for a quicker build as the nr. of these runners are sparse
include: include:
- name: "basic tests"
go: 1.11.x
env: TEST_BLOCK=basic
- name: "shell tests"
go: 1.11.x
env: TEST_BLOCK=shell
- name: "race tests"
go: 1.11.x
env: TEST_BLOCK=race
- go: 1.12.x
- go: tip
- os: osx - os: osx
go: 1.10.x script: 'TEST_BLOCK="$TEST_BLOCK" make test'
# the "secure" channel value is the result of running: ./misc/travis-encrypt.sh # the "secure" channel value is the result of running: ./misc/travis-encrypt.sh
# with a value of: irc.freenode.net#mgmtconfig to eliminate noise from forks... # with a value of: irc.freenode.net#mgmtconfig to eliminate noise from forks...

2
COPYRIGHT
View File

@@ -1,5 +1,5 @@
Mgmt Mgmt
Copyright (C) 2013-2018+ James Shubin and the project contributors Copyright (C) 2013-2019+ James Shubin and the project contributors
Written by James Shubin <james@shubin.ca> and the project contributors Written by James Shubin <james@shubin.ca> and the project contributors
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify

161
Makefile
View File

@@ -1,5 +1,5 @@
# Mgmt # Mgmt
# Copyright (C) 2013-2018+ James Shubin and the project contributors # Copyright (C) 2013-2019+ James Shubin and the project contributors
# Written by James Shubin <james@shubin.ca> and the project contributors # Written by James Shubin <james@shubin.ca> and the project contributors
# #
# This program is free software: you can redistribute it and/or modify # This program is free software: you can redistribute it and/or modify
@@ -16,11 +16,16 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
SHELL = /usr/bin/env bash SHELL = /usr/bin/env bash
.PHONY: all art cleanart version program lang path deps run race bindata generate build build-debug crossbuild clean test gofmt yamlfmt format docs rpmbuild mkdirs rpm srpm spec tar upload upload-sources upload-srpms upload-rpms copr tag release .PHONY: all art cleanart version program lang path deps run race bindata generate build build-debug crossbuild clean test gofmt yamlfmt format docs
.PHONY: rpmbuild mkdirs rpm srpm spec tar upload upload-sources upload-srpms upload-rpms upload-releases copr tag
.PHONY: mkosi mkosi_fedora-30 mkosi_fedora-29 mkosi_debian-10 mkosi_ubuntu-bionic mkosi_archlinux
.PHONY: release releases_path release_fedora-30 release_fedora-29 release_debian-10 release_ubuntu-bionic release_archlinux
.PHONY: funcgen
.SILENT: clean bindata .SILENT: clean bindata
# a large amount of output from this `find`, can cause `make` to be much slower! # a large amount of output from this `find`, can cause `make` to be much slower!
GO_FILES := $(shell find * -name '*.go' -not -path 'old/*' -not -path 'tmp/*') GO_FILES := $(shell find * -name '*.go' -not -path 'old/*' -not -path 'tmp/*')
MCL_FILES := $(shell find lang/funcs/ -name '*.mcl' -not -path 'old/*' -not -path 'tmp/*')
SVERSION := $(or $(SVERSION),$(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --dirty --always)) SVERSION := $(or $(SVERSION),$(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --dirty --always))
VERSION := $(or $(VERSION),$(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --abbrev=0)) VERSION := $(or $(VERSION),$(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --abbrev=0))
@@ -48,9 +53,23 @@ GOOSARCHES ?= linux/amd64 linux/ppc64 linux/ppc64le linux/arm64 darwin/amd64
GOHOSTOS = $(shell go env GOHOSTOS) GOHOSTOS = $(shell go env GOHOSTOS)
GOHOSTARCH = $(shell go env GOHOSTARCH) GOHOSTARCH = $(shell go env GOHOSTARCH)
RPM_PKG = releases/$(VERSION)/rpm/mgmt-$(VERSION)-1.x86_64.rpm TOKEN_FEDORA-30 = fedora-30
DEB_PKG = releases/$(VERSION)/deb/mgmt_$(VERSION)_amd64.deb TOKEN_FEDORA-29 = fedora-29
PACMAN_PKG = releases/$(VERSION)/pacman/mgmt-$(VERSION)-1-x86_64.pkg.tar.xz TOKEN_DEBIAN-10 = debian-10
TOKEN_UBUNTU-BIONIC = ubuntu-bionic
TOKEN_ARCHLINUX = archlinux
FILE_FEDORA-30 = mgmt-$(TOKEN_FEDORA-30)-$(VERSION)-1.x86_64.rpm
FILE_FEDORA-29 = mgmt-$(TOKEN_FEDORA-29)-$(VERSION)-1.x86_64.rpm
FILE_DEBIAN-10 = mgmt_$(TOKEN_DEBIAN-10)_$(VERSION)_amd64.deb
FILE_UBUNTU-BIONIC = mgmt_$(TOKEN_UBUNTU-BIONIC)_$(VERSION)_amd64.deb
FILE_ARCHLINUX = mgmt-$(TOKEN_ARCHLINUX)-$(VERSION)-1-x86_64.pkg.tar.xz
PKG_FEDORA-30 = releases/$(VERSION)/$(TOKEN_FEDORA-30)/$(FILE_FEDORA-30)
PKG_FEDORA-29 = releases/$(VERSION)/$(TOKEN_FEDORA-29)/$(FILE_FEDORA-29)
PKG_DEBIAN-10 = releases/$(VERSION)/$(TOKEN_DEBIAN-10)/$(FILE_DEBIAN-10)
PKG_UBUNTU-BIONIC = releases/$(VERSION)/$(TOKEN_UBUNTU-BIONIC)/$(FILE_UBUNTU-BIONIC)
PKG_ARCHLINUX = releases/$(VERSION)/$(TOKEN_ARCHLINUX)/$(FILE_ARCHLINUX)
SHA256SUMS = releases/$(VERSION)/SHA256SUMS SHA256SUMS = releases/$(VERSION)/SHA256SUMS
SHA256SUMS_ASC = $(SHA256SUMS).asc SHA256SUMS_ASC = $(SHA256SUMS).asc
@@ -117,7 +136,6 @@ race:
# generate go files from non-go source # generate go files from non-go source
bindata: ## generate go files from non-go sources bindata: ## generate go files from non-go sources
@echo "Generating: bindata..."
$(MAKE) --quiet -C bindata $(MAKE) --quiet -C bindata
$(MAKE) --quiet -C lang/funcs $(MAKE) --quiet -C lang/funcs
@@ -126,14 +144,13 @@ generate:
lang: ## generates the lexer/parser for the language frontend lang: ## generates the lexer/parser for the language frontend
@# recursively run make in child dir named lang @# recursively run make in child dir named lang
@echo "Generating: lang..." @$(MAKE) --quiet -C lang
$(MAKE) --quiet -C lang
# build a `mgmt` binary for current host os/arch # build a `mgmt` binary for current host os/arch
$(PROGRAM): build/mgmt-${GOHOSTOS}-${GOHOSTARCH} ## build an mgmt binary for current host os/arch $(PROGRAM): build/mgmt-${GOHOSTOS}-${GOHOSTARCH} ## build an mgmt binary for current host os/arch
cp -a $< $@ cp -a $< $@
$(PROGRAM).static: $(GO_FILES) $(PROGRAM).static: $(GO_FILES) $(MCL_FILES)
@echo "Building: $(PROGRAM).static, version: $(SVERSION)..." @echo "Building: $(PROGRAM).static, version: $(SVERSION)..."
go generate go generate
go build -a -installsuffix cgo -tags netgo -ldflags '-extldflags "-static" -X main.program=$(PROGRAM) -X main.version=$(SVERSION) -s -w' -o $(PROGRAM).static $(BUILD_FLAGS); go build -a -installsuffix cgo -tags netgo -ldflags '-extldflags "-static" -X main.program=$(PROGRAM) -X main.version=$(SVERSION) -s -w' -o $(PROGRAM).static $(BUILD_FLAGS);
@@ -148,10 +165,10 @@ build-debug: $(PROGRAM)
# extract os and arch from target pattern # extract os and arch from target pattern
GOOS=$(firstword $(subst -, ,$*)) GOOS=$(firstword $(subst -, ,$*))
GOARCH=$(lastword $(subst -, ,$*)) GOARCH=$(lastword $(subst -, ,$*))
build/mgmt-%: $(GO_FILES) | bindata lang build/mgmt-%: $(GO_FILES) $(MCL_FILES) | bindata lang funcgen
@echo "Building: $(PROGRAM), os/arch: $*, version: $(SVERSION)..." @echo "Building: $(PROGRAM), os/arch: $*, version: $(SVERSION)..."
@# reassigning GOOS and GOARCH to make build command copy/pastable @# reassigning GOOS and GOARCH to make build command copy/pastable
@# go 1.10 requires specifying the package for ldflags @# go 1.10+ requires specifying the package for ldflags
@if go version | grep -qE 'go1.9'; then \ @if go version | grep -qE 'go1.9'; then \
time env GOOS=${GOOS} GOARCH=${GOARCH} go build -i -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION) ${LDFLAGS}" -o $@ $(BUILD_FLAGS); \ time env GOOS=${GOOS} GOARCH=${GOARCH} go build -i -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION) ${LDFLAGS}" -o $@ $(BUILD_FLAGS); \
else \ else \
@@ -166,6 +183,9 @@ clean: ## clean things up
$(MAKE) --quiet -C bindata clean $(MAKE) --quiet -C bindata clean
$(MAKE) --quiet -C lang/funcs clean $(MAKE) --quiet -C lang/funcs clean
$(MAKE) --quiet -C lang clean $(MAKE) --quiet -C lang clean
$(MAKE) --quiet -C misc/mkosi clean
rm -f lang/funcs/core/generated_funcs.go || true
rm -f lang/funcs/core/generated_funcs_test.go || true
[ ! -e $(PROGRAM) ] || rm $(PROGRAM) [ ! -e $(PROGRAM) ] || rm $(PROGRAM)
rm -f *_stringer.go # generated by `go generate` rm -f *_stringer.go # generated by `go generate`
rm -f *_mock.go # generated by `go generate` rm -f *_mock.go # generated by `go generate`
@@ -188,8 +208,8 @@ $(addprefix test-shell-,${test_shell}): test-shell-%: build
gofmt: gofmt:
# TODO: remove gofmt once goimports has a -s option # TODO: remove gofmt once goimports has a -s option
find . -maxdepth 6 -type f -name '*.go' -not -path './old/*' -not -path './tmp/*' -not -path './vendor/*' -exec gofmt -s -w {} \; find . -maxdepth 9 -type f -name '*.go' -not -path './old/*' -not -path './tmp/*' -not -path './vendor/*' -exec gofmt -s -w {} \;
find . -maxdepth 6 -type f -name '*.go' -not -path './old/*' -not -path './tmp/*' -not -path './vendor/*' -exec goimports -w {} \; find . -maxdepth 9 -type f -name '*.go' -not -path './old/*' -not -path './tmp/*' -not -path './vendor/*' -exec goimports -w {} \;
yamlfmt: yamlfmt:
find . -maxdepth 3 -type f -name '*.yaml' -not -path './old/*' -not -path './tmp/*' -not -path './omv.yaml' -exec ruby -e "require 'yaml'; x=YAML.load_file('{}').to_yaml.each_line.map(&:rstrip).join(10.chr)+10.chr; File.open('{}', 'w').write x" \; find . -maxdepth 3 -type f -name '*.yaml' -not -path './old/*' -not -path './tmp/*' -not -path './omv.yaml' -exec ruby -e "require 'yaml'; x=YAML.load_file('{}').to_yaml.each_line.map(&:rstrip).join(10.chr)+10.chr; File.open('{}', 'w').write x" \;
@@ -326,6 +346,10 @@ upload-rpms: rpmbuild/RPMS/ rpmbuild/RPMS/SHA256SUMS rpmbuild/RPMS/SHA256SUMS.as
rsync -avz --prune-empty-dirs rpmbuild/RPMS/ $(SERVER):$(REMOTE_PATH)/RPMS/; \ rsync -avz --prune-empty-dirs rpmbuild/RPMS/ $(SERVER):$(REMOTE_PATH)/RPMS/; \
fi fi
upload-releases:
echo Running releases/ upload...
rsync -avz --exclude '.mkdir' --exclude 'mgmt-release.url' releases/ $(SERVER):$(REMOTE_PATH)/releases/
# #
# copr build # copr build
# #
@@ -338,18 +362,57 @@ copr: upload-srpms ## build in copr
tag: ## tags a new release tag: ## tags a new release
./misc/tag.sh ./misc/tag.sh
#
# mkosi
#
mkosi: mkosi_fedora-30 mkosi_fedora-29 mkosi_debian-10 mkosi_ubuntu-bionic mkosi_archlinux ## builds distro packages via mkosi
mkosi_fedora-30: releases/$(VERSION)/.mkdir
@title='$@' ; echo "Generating: $${title#'mkosi_'} via mkosi..."
@title='$@' ; distro=$${title#'mkosi_'} ; ./misc/mkosi/make.sh $${distro} `realpath "releases/$(VERSION)/"`
mkosi_fedora-29: releases/$(VERSION)/.mkdir
@title='$@' ; echo "Generating: $${title#'mkosi_'} via mkosi..."
@title='$@' ; distro=$${title#'mkosi_'} ; ./misc/mkosi/make.sh $${distro} `realpath "releases/$(VERSION)/"`
mkosi_debian-10: releases/$(VERSION)/.mkdir
@title='$@' ; echo "Generating: $${title#'mkosi_'} via mkosi..."
@title='$@' ; distro=$${title#'mkosi_'} ; ./misc/mkosi/make.sh $${distro} `realpath "releases/$(VERSION)/"`
mkosi_ubuntu-bionic: releases/$(VERSION)/.mkdir
@title='$@' ; echo "Generating: $${title#'mkosi_'} via mkosi..."
@title='$@' ; distro=$${title#'mkosi_'} ; ./misc/mkosi/make.sh $${distro} `realpath "releases/$(VERSION)/"`
mkosi_archlinux: releases/$(VERSION)/.mkdir
@title='$@' ; echo "Generating: $${title#'mkosi_'} via mkosi..."
@title='$@' ; distro=$${title#'mkosi_'} ; ./misc/mkosi/make.sh $${distro} `realpath "releases/$(VERSION)/"`
# #
# release # release
# #
release: releases/$(VERSION)/mgmt-release.url ## generates and uploads a release release: releases/$(VERSION)/mgmt-release.url ## generates and uploads a release
releases/$(VERSION)/mgmt-release.url: $(RPM_PKG) $(DEB_PKG) $(PACMAN_PKG) $(SHA256SUMS_ASC) releases_path:
@#Don't put any other output or dependencies in here or they'll show!
@echo "releases/$(VERSION)/"
release_fedora-30: $(PKG_FEDORA-30)
release_fedora-29: $(PKG_FEDORA-29)
release_debian-10: $(PKG_DEBIAN-10)
release_ubuntu-bionic: $(PKG_UBUNTU-BIONIC)
release_archlinux: $(PKG_ARCHLINUX)
releases/$(VERSION)/mgmt-release.url: $(PKG_FEDORA-30) $(PKG_FEDORA-29) $(PKG_DEBIAN-10) $(PKG_UBUNTU-BIONIC) $(PKG_ARCHLINUX) $(SHA256SUMS_ASC)
@echo "Pushing git tag $(VERSION) to origin..."
git push origin $(VERSION)
@echo "Creating github release..." @echo "Creating github release..."
hub release create \ hub release create \
-F <( echo -e "$(VERSION)\n";echo "Verify the signatures of all packages before you use them. The signing key can be downloaded from https://purpleidea.com/contact/#pgp-key to verify the release." ) \ -F <( echo -e "$(VERSION)\n";echo "Verify the signatures of all packages before you use them. The signing key can be downloaded from https://purpleidea.com/contact/#pgp-key to verify the release." ) \
-a $(RPM_PKG) \ -a $(PKG_FEDORA-30) \
-a $(DEB_PKG) \ -a $(PKG_FEDORA-29) \
-a $(PACMAN_PKG) \ -a $(PKG_DEBIAN-10) \
-a $(PKG_UBUNTU-BIONIC) \
-a $(PKG_ARCHLINUX) \
-a $(SHA256SUMS_ASC) \ -a $(SHA256SUMS_ASC) \
$(VERSION) \ $(VERSION) \
> releases/$(VERSION)/mgmt-release.url \ > releases/$(VERSION)/mgmt-release.url \
@@ -357,32 +420,48 @@ releases/$(VERSION)/mgmt-release.url: $(RPM_PKG) $(DEB_PKG) $(PACMAN_PKG) $(SHA2
|| rm -f releases/$(VERSION)/mgmt-release.url || rm -f releases/$(VERSION)/mgmt-release.url
releases/$(VERSION)/.mkdir: releases/$(VERSION)/.mkdir:
mkdir -p releases/$(VERSION)/{deb,rpm,pacman}/ && touch releases/$(VERSION)/.mkdir mkdir -p releases/$(VERSION)/{$(TOKEN_FEDORA-30),$(TOKEN_FEDORA-29),$(TOKEN_DEBIAN-10),$(TOKEN_UBUNTU-BIONIC),$(TOKEN_ARCHLINUX)}/ && touch releases/$(VERSION)/.mkdir
releases/$(VERSION)/rpm/changelog: $(PROGRAM) releases/$(VERSION)/.mkdir releases/$(VERSION)/$(TOKEN_FEDORA-30)/changelog: $(PROGRAM) releases/$(VERSION)/.mkdir
@echo "Generating rpm changelog..." @title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; echo "Generating: $${distro} changelog..."
./misc/make-rpm-changelog.sh $(VERSION) @title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; ./misc/make-rpm-changelog.sh "$${distro}" $(VERSION)
$(RPM_PKG): releases/$(VERSION)/rpm/changelog $(PKG_FEDORA-30): releases/$(VERSION)/$(TOKEN_FEDORA-30)/changelog
@echo "Building rpm package..." @title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; echo "Building: $${distro} package..."
./misc/fpm-pack.sh rpm $(VERSION) libvirt-devel augeas-devel @title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; ./misc/fpm-pack.sh $${distro} $(VERSION) "$(FILE_FEDORA-30)" libvirt-devel augeas-devel
releases/$(VERSION)/deb/changelog: $(PROGRAM) releases/$(VERSION)/.mkdir releases/$(VERSION)/$(TOKEN_FEDORA-29)/changelog: $(PROGRAM) releases/$(VERSION)/.mkdir
@echo "Generating deb changelog..." @title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; echo "Generating: $${distro} changelog..."
./misc/make-deb-changelog.sh $(VERSION) @title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; ./misc/make-rpm-changelog.sh "$${distro}" $(VERSION)
$(DEB_PKG): releases/$(VERSION)/deb/changelog $(PKG_FEDORA-29): releases/$(VERSION)/$(TOKEN_FEDORA-29)/changelog
@echo "Building deb package..." @title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; echo "Building: $${distro} package..."
./misc/fpm-pack.sh deb $(VERSION) libvirt-dev libaugeas-dev @title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; ./misc/fpm-pack.sh $${distro} $(VERSION) "$(FILE_FEDORA-29)" libvirt-devel augeas-devel
$(PACMAN_PKG): $(PROGRAM) releases/$(VERSION)/.mkdir releases/$(VERSION)/$(TOKEN_DEBIAN-10)/changelog: $(PROGRAM) releases/$(VERSION)/.mkdir
@echo "Building pacman package..." @title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; echo "Generating: $${distro} changelog..."
./misc/fpm-pack.sh pacman $(VERSION) libvirt augeas @title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; ./misc/make-deb-changelog.sh "$${distro}" $(VERSION)
$(SHA256SUMS): $(RPM_PKG) $(DEB_PKG) $(PACMAN_PKG) $(PKG_DEBIAN-10): releases/$(VERSION)/$(TOKEN_DEBIAN-10)/changelog
@title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; echo "Building: $${distro} package..."
@title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; ./misc/fpm-pack.sh $${distro} $(VERSION) "$(FILE_DEBIAN-10)" libvirt-dev libaugeas-dev
releases/$(VERSION)/$(TOKEN_UBUNTU-BIONIC)/changelog: $(PROGRAM) releases/$(VERSION)/.mkdir
@title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; echo "Generating: $${distro} changelog..."
@title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; ./misc/make-deb-changelog.sh "$${distro}" $(VERSION)
$(PKG_UBUNTU-BIONIC): releases/$(VERSION)/$(TOKEN_UBUNTU-BIONIC)/changelog
@title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; echo "Building: $${distro} package..."
@title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; ./misc/fpm-pack.sh $${distro} $(VERSION) "$(FILE_UBUNTU-BIONIC)" libvirt-dev libaugeas-dev
$(PKG_ARCHLINUX): $(PROGRAM) releases/$(VERSION)/.mkdir
@title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; echo "Building: $${distro} package..."
@title='$(@D)' ; distro=$${title#'releases/$(VERSION)/'} ; ./misc/fpm-pack.sh $${distro} $(VERSION) "$(FILE_ARCHLINUX)" libvirt augeas
$(SHA256SUMS): $(PKG_FEDORA-30) $(PKG_FEDORA-29) $(PKG_DEBIAN-10) $(PKG_UBUNTU-BIONIC) $(PKG_ARCHLINUX)
@# remove the directory separator in the SHA256SUMS file @# remove the directory separator in the SHA256SUMS file
@echo "Generating sha256 sum..." @echo "Generating: sha256 sum..."
sha256sum $(RPM_PKG) $(DEB_PKG) $(PACMAN_PKG) | awk -F '/| ' '{print $$1" "$$6}' > $(SHA256SUMS) sha256sum $(PKG_FEDORA-30) $(PKG_FEDORA-29) $(PKG_DEBIAN-10) $(PKG_UBUNTU-BIONIC) $(PKG_ARCHLINUX) | awk -F '/| ' '{print $$1" "$$6}' > $(SHA256SUMS)
$(SHA256SUMS_ASC): $(SHA256SUMS) $(SHA256SUMS_ASC): $(SHA256SUMS)
@echo "Signing sha256 sum..." @echo "Signing sha256 sum..."
@@ -408,4 +487,14 @@ help: ## show this help screen
awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}' awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'
@echo '' @echo ''
funcgen: lang/funcs/core/generated_funcs_test.go lang/funcs/core/generated_funcs.go
lang/funcs/core/generated_funcs_test.go: lang/funcs/funcgen/*.go lang/funcs/core/funcgen.yaml lang/funcs/funcgen/templates/generated_funcs_test.go.tpl
@echo "Generating: funcs test..."
@go run lang/funcs/funcgen/*.go -templates lang/funcs/funcgen/templates/generated_funcs_test.go.tpl 2>/dev/null
lang/funcs/core/generated_funcs.go: lang/funcs/funcgen/*.go lang/funcs/core/funcgen.yaml lang/funcs/funcgen/templates/generated_funcs.go.tpl
@echo "Generating: funcs..."
@go run lang/funcs/funcgen/*.go -templates lang/funcs/funcgen/templates/generated_funcs.go.tpl 2>/dev/null
# vim: ts=8 # vim: ts=8

66
README.md
View File

@@ -9,6 +9,56 @@
[![Patreon](https://img.shields.io/badge/patreon-donate-yellow.svg?style=flat-square)](https://www.patreon.com/purpleidea) [![Patreon](https://img.shields.io/badge/patreon-donate-yellow.svg?style=flat-square)](https://www.patreon.com/purpleidea)
[![Liberapay](https://img.shields.io/badge/liberapay-donate-yellow.svg?style=flat-square)](https://liberapay.com/purpleidea/donate) [![Liberapay](https://img.shields.io/badge/liberapay-donate-yellow.svg?style=flat-square)](https://liberapay.com/purpleidea/donate)
## About:
`Mgmt` is a real-time automation tool. It is familiar to existing configuration
management software, but is drastically more powerful as it can allow you to
build real-time, closed-loop feedback systems, in a very safe way, and with a
surprisingly small amout of our `mcl` code. For example, the following code will
ensure that your file server is set to read-only when it's friday.
```mcl
import "datetime"
$is_friday = datetime.weekday(datetime.now()) == "friday"
file "/srv/files/" {
state => "exists",
mode => if $is_friday { # this updates the mode, the instant it changes!
"0550"
} else {
"0770"
},
}
```
It can run continuously, intermittently, or on-demand, and in the first case, it
will guarantee that your system is always in the desired state for that instant!
In this mode it can run as a decentralized cluster of agents across your
network, each exchanging information with the others in real-time, to respond to
your changing needs. For example, if you want to ensure that some resource runs
on a maximum of two hosts in your cluster, you can specify that as well:
```mcl
import "sys"
import "world"
# we'll set a few scheduling options:
$opts = struct{strategy => "rr", max => 2, ttl => 10,}
# schedule in a particular namespace with options:
$set = world.schedule("xsched", $opts)
if sys.hostname() in $set {
# use your imagination to put something more complex right here...
print "i got scheduled" {} # this will run on the chosen machines
}
```
As you add and remove hosts from the cluster, the real-time `schedule` function
will dynamically pick up to two hosts from the available pool. These specific
functions aren't intrinsic to the core design, and new ones can be easily added.
Please read on if you'd like to learn more...
## Community: ## Community:
Come join us in the `mgmt` community! Come join us in the `mgmt` community!
@@ -30,7 +80,7 @@ approach. The project contains an engine and a language.
Mgmt is a fairly new project. It is usable today, but not yet feature complete. Mgmt is a fairly new project. It is usable today, but not yet feature complete.
With your help you'll be able to influence our design and get us to 1.0 sooner! With your help you'll be able to influence our design and get us to 1.0 sooner!
Interested developers should read the [quick start guide](docs/quick-start-guide.md). Interested users should read the [quick start guide](docs/quick-start-guide.md).
## Documentation: ## Documentation:
@@ -38,7 +88,7 @@ Please read, enjoy and help improve our documentation!
| Documentation | Additional Notes | | Documentation | Additional Notes |
|---|---| |---|---|
| [quick start guide](docs/quick-start-guide.md) | for mgmt developers | | [quick start guide](docs/quick-start-guide.md) | for everyone |
| [frequently asked questions](docs/faq.md) | for everyone | | [frequently asked questions](docs/faq.md) | for everyone |
| [general documentation](docs/documentation.md) | for everyone | | [general documentation](docs/documentation.md) | for everyone |
| [language guide](docs/language-guide.md) | for everyone | | [language guide](docs/language-guide.md) | for everyone |
@@ -57,22 +107,18 @@ If you have a well phrased question that might benefit others, consider asking
it by sending a patch to the [FAQ](docs/faq.md) section. I'll merge your it by sending a patch to the [FAQ](docs/faq.md) section. I'll merge your
question, and a patch with the answer! question, and a patch with the answer!
## Roadmap: ## Get involved:
Feel free to grab one of the straightforward [#mgmtlove](https://github.com/purpleidea/mgmt/labels/mgmtlove) Feel free to grab one of the straightforward [#mgmtlove](https://github.com/purpleidea/mgmt/labels/mgmtlove)
issues if you're a first time contributor to the project or if you're unsure issues if you're a first time contributor to the project or if you're unsure
about what to hack on! about what to hack on! Please get involved by working on one of these items or
Please see: [TODO.md](TODO.md) for a list of upcoming work and TODO items. by suggesting something else! There are some lower priority issues and harder
Please get involved by working on one of these items or by suggesting something issues available in our [TODO](TODO.md) file. Please have a look.
else!
## Bugs: ## Bugs:
Please set the `DEBUG` constant in [main.go](https://github.com/purpleidea/mgmt/blob/master/main.go) Please set the `DEBUG` constant in [main.go](https://github.com/purpleidea/mgmt/blob/master/main.go)
to `true`, and post the logs when you report the [issue](https://github.com/purpleidea/mgmt/issues). to `true`, and post the logs when you report the [issue](https://github.com/purpleidea/mgmt/issues).
Bonus points if you provide a [shell](https://github.com/purpleidea/mgmt/tree/master/test/shell)
or [OMV](https://github.com/purpleidea/mgmt/tree/master/test/omv) reproducible
test case.
Feel free to read my article on [debugging golang programs](https://purpleidea.com/blog/2016/02/15/debugging-golang-programs/). Feel free to read my article on [debugging golang programs](https://purpleidea.com/blog/2016/02/15/debugging-golang-programs/).
## Patches: ## Patches:

65
TODO.md
View File

@@ -1,10 +1,18 @@
# TODO # TODO
If you're looking for something to do, look here! Here is a TODO list of longstanding items that are either lower-priority, or
Let us know if you're working on one of the items. more involved in terms of time, skill-level, and/or motivation.
If you'd like something to work on, ping @purpleidea and I'll create an issue
tailored especially for you! Just let me know your approximate golang skill Please have a look, and let us know if you're working on one of the items. It's
level and how many hours you'd like to spend on the patch. best to open an issue to track your progress and to discuss any implementation
questions you might have.
Lastly, if you'd like something different to work on, please ping @purpleidea
and I'll create an issue tailored especially for your approximate golang skill
level and available time commitment in terms of hours you'd need to spend on the
patch.
Happy Hacking!
## Package resource ## Package resource
@@ -19,7 +27,7 @@ level and how many hours you'd like to spend on the patch.
## Svc resource ## Svc resource
- [ ] base resource improvements - [ ] refreshonly support [:heart:](https://github.com/purpleidea/mgmt/issues/464)
## Exec resource ## Exec resource
@@ -33,33 +41,14 @@ level and how many hours you'd like to spend on the patch.
- [ ] automatic edges to file resource [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove) - [ ] automatic edges to file resource [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove)
## Virt (libvirt) resource
- [ ] base resource improvements [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove)
## Net (systemd-networkd) resource
- [ ] base resource
## Nspawn (systemd-nspawn) resource
- [ ] base resource [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove)
## Mount (systemd-mount) resource
- [ ] base resource [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove)
## Cron (systemd-timer) resource
- [ ] base resource [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove)
## Http resource ## Http resource
- [ ] base resource [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove) - [ ] base resource [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove)
## Etcd improvements ## Etcd improvements
- [ ] fix embedded etcd master race - [ ] fix etcd race bug that only happens during CI testing (intermittently
failing test case issue)
## Torrent/dht file transfer ## Torrent/dht file transfer
@@ -69,17 +58,33 @@ level and how many hours you'd like to spend on the patch.
- [ ] base plumbing - [ ] base plumbing
## Resource improvements
- [ ] more reversible resources implemented
- [ ] more "cloud" resources
## Language improvements ## Language improvements
- [ ] more core functions - [ ] more core functions
- [ ] automatic language formatter, ala `gofmt` - [ ] automatic language formatter, ala `gofmt`
- [ ] gedit/gnome-builder/gtksourceview syntax highlighting - [ ] gedit/gnome-builder/gtksourceview syntax highlighting
- [ ] vim syntax highlighting - [ ] vim syntax highlighting
- [x] emacs syntax highlighting: see `misc/emacs/` - [ ] emacs syntax highlighting: see `misc/emacs/` (needs updating)
- [ ] exposed $error variable for feedback in the language
- [ ] improve the printf function to add %[]s, %[]f ([]str, []float) and map,
struct, nested etc... %v would be nice too!
- [ ] add line/col/file annotations to AST so we can get locations of errors
that the parser finds
- [ ] add more error messages with the `%error` pattern in parser.y
- [ ] we should have helper functions or language sugar to pull a field out of a
struct, or a value out of a map, or an index out of a list, etc...
## Engine improvements
- [ ] add a "waiting for func" message in the func engine to notify the user
about slow functions...
## Other ## Other
- [ ] better error/retry handling
- [ ] deb package target in Makefile
- [ ] reproducible builds - [ ] reproducible builds
- [ ] add your suggestions! - [ ] add your suggestions!

BIN
art/mgmt_poobear_meme.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 102 KiB

3
bindata/Makefile
View File

@@ -1,5 +1,5 @@
# Mgmt # Mgmt
# Copyright (C) 2013-2018+ James Shubin and the project contributors # Copyright (C) 2013-2019+ James Shubin and the project contributors
# Written by James Shubin <james@shubin.ca> and the project contributors # Written by James Shubin <james@shubin.ca> and the project contributors
# #
# This program is free software: you can redistribute it and/or modify # This program is free software: you can redistribute it and/or modify
@@ -30,6 +30,7 @@ build: bindata.go
# add more input files as dependencies at the end here... # add more input files as dependencies at the end here...
bindata.go: ../COPYING bindata.go: ../COPYING
@echo "Generating: bindata..."
# go-bindata --pkg bindata -o <OUTPUT> <INPUT> # go-bindata --pkg bindata -o <OUTPUT> <INPUT>
go-bindata --pkg bindata -o ./$@ $^ go-bindata --pkg bindata -o ./$@ $^
# gofmt the output file # gofmt the output file

627
converger/converger.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -25,139 +25,251 @@ import (
"time" "time"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
multierr "github.com/hashicorp/go-multierror"
) )
// TODO: we could make a new function that masks out the state of certain // New builds a new converger coordinator.
// UID's, but at the moment the new Timer code has obsoleted the need... func New(timeout int64) *Coordinator {
return &Coordinator{
// Converger is the general interface for implementing a convergence watcher.
type Converger interface { // TODO: need a better name
Register() UID
IsConverged(UID) bool // is the UID converged ?
SetConverged(UID, bool) error // set the converged state of the UID
Unregister(UID)
Start()
Pause()
Loop(bool)
ConvergedTimer(UID) <-chan time.Time
Status() map[uint64]bool
Timeout() int // returns the timeout that this was created with
AddStateFn(string, func(bool) error) error // adds a stateFn with a name
RemoveStateFn(string) error // remove a stateFn with a given name
}
// UID is the interface resources can use to notify with if converged. You'll
// need to use part of the Converger interface to Register initially too.
type UID interface {
ID() uint64 // get Id
Name() string // get a friendly name
SetName(string)
IsValid() bool // has Id been initialized ?
InvalidateID() // set Id to nil
IsConverged() bool
SetConverged(bool) error
Unregister()
ConvergedTimer() <-chan time.Time
StartTimer() (func() error, error) // cancellable is the same as StopTimer()
ResetTimer() error // resets counter to zero
StopTimer() error
}
// converger is an implementation of the Converger interface.
type converger struct {
timeout int // must be zero (instant) or greater seconds to run
converged bool // did we converge (state changes of this run Fn)
channel chan struct{} // signal here to run an isConverged check
control chan bool // control channel for start/pause
mutex *sync.RWMutex // used for controlling access to status and lastid
lastid uint64
status map[uint64]bool
stateFns map[string]func(bool) error // run on converged state changes with state bool
smutex *sync.RWMutex // used for controlling access to stateFns
}
// cuid is an implementation of the UID interface.
type cuid struct {
converger Converger
id uint64
name string // user defined, friendly name
mutex *sync.Mutex
timer chan struct{}
running bool // is the above timer running?
wg *sync.WaitGroup
}
// NewConverger builds a new converger struct.
func NewConverger(timeout int) Converger {
return &converger{
timeout: timeout, timeout: timeout,
channel: make(chan struct{}),
control: make(chan bool),
mutex: &sync.RWMutex{}, mutex: &sync.RWMutex{},
lastid: 0,
status: make(map[uint64]bool), //lastid: 0,
status: make(map[*UID]struct{}),
//converged: false, // initial state
pokeChan: make(chan struct{}, 1), // must be buffered
readyChan: make(chan struct{}), // ready signal
//paused: false, // starts off as started
pauseSignal: make(chan struct{}),
//resumeSignal: make(chan struct{}), // happens on pause
//pausedAck: util.NewEasyAck(), // happens on pause
stateFns: make(map[string]func(bool) error), stateFns: make(map[string]func(bool) error),
smutex: &sync.RWMutex{}, smutex: &sync.RWMutex{},
closeChan: make(chan struct{}),
wg: &sync.WaitGroup{},
} }
} }
// Register assigns a UID to the caller. // Coordinator is the central converger engine.
func (obj *converger) Register() UID { type Coordinator struct {
// timeout must be zero (instant) or greater seconds to run. If it's -1
// then this is disabled, and we never run stateFns.
timeout int64
// mutex is used for controlling access to status and lastid.
mutex *sync.RWMutex
// lastid contains the last uid we used for registration.
//lastid uint64
// status contains a reference to each active UID.
status map[*UID]struct{}
// converged stores the last convergence state. When this changes, we
// run the stateFns.
converged bool
// pokeChan receives a message every time we might need to re-calculate.
pokeChan chan struct{}
// readyChan closes to notify any interested parties that the main loop
// is running.
readyChan chan struct{}
// paused represents if this coordinator is paused or not.
paused bool
// pauseSignal closes to request a pause of this coordinator.
pauseSignal chan struct{}
// resumeSignal closes to request a resume of this coordinator.
resumeSignal chan struct{}
// pausedAck is used to send an ack message saying that we've paused.
pausedAck *util.EasyAck
// stateFns run on converged state changes.
stateFns map[string]func(bool) error
// smutex is used for controlling access to the stateFns map.
smutex *sync.RWMutex
// closeChan closes when we've been requested to shutdown.
closeChan chan struct{}
// wg waits for everything to finish.
wg *sync.WaitGroup
}
// Register creates a new UID which can be used to report converged state. You
// must Unregister each UID before Shutdown will be able to finish running.
func (obj *Coordinator) Register() *UID {
obj.wg.Add(1) // additional tracking for each UID
obj.mutex.Lock() obj.mutex.Lock()
defer obj.mutex.Unlock() defer obj.mutex.Unlock()
obj.lastid++ //obj.lastid++
obj.status[obj.lastid] = false // initialize as not converged uid := &UID{
return &cuid{ timeout: obj.timeout, // copy the timeout here
converger: obj, //id: obj.lastid,
id: obj.lastid, //name: fmt.Sprintf("%d", obj.lastid), // some default
name: fmt.Sprintf("%d", obj.lastid), // some default
poke: obj.poke,
// timer
mutex: &sync.Mutex{}, mutex: &sync.Mutex{},
timer: nil, timer: nil,
running: false, running: false,
wg: &sync.WaitGroup{}, wg: &sync.WaitGroup{},
} }
uid.unregister = func() { obj.Unregister(uid) } // add unregister func
obj.status[uid] = struct{}{} // TODO: add converged state here?
return uid
} }
// IsConverged gets the converged status of a uid. // Unregister removes the UID from the converger coordinator. If you supply an
func (obj *converger) IsConverged(uid UID) bool { // invalid or unregistered uid to this function, it will panic. An unregistered
if !uid.IsValid() { // UID is no longer part of the convergence checking.
panic(fmt.Sprintf("the ID of UID(%s) is nil", uid.Name())) func (obj *Coordinator) Unregister(uid *UID) {
} defer obj.wg.Done() // additional tracking for each UID
obj.mutex.RLock()
isConverged, found := obj.status[uid.ID()] // lookup
obj.mutex.RUnlock()
if !found {
panic("the ID of UID is unregistered")
}
return isConverged
}
// SetConverged updates the converger with the converged state of the UID.
func (obj *converger) SetConverged(uid UID, isConverged bool) error {
if !uid.IsValid() {
return fmt.Errorf("the ID of UID(%s) is nil", uid.Name())
}
obj.mutex.Lock() obj.mutex.Lock()
if _, found := obj.status[uid.ID()]; !found { defer obj.mutex.Unlock()
panic("the ID of UID is unregistered")
if _, exists := obj.status[uid]; !exists {
panic("uid is not registered")
} }
obj.status[uid.ID()] = isConverged // set uid.StopTimer() // ignore any errors
obj.mutex.Unlock() // unlock *before* poke or deadlock! delete(obj.status, uid)
if isConverged != obj.converged { // only poke if it would be helpful }
// run in a go routine so that we never block... just queue up!
// this allows us to send events, even if we haven't started... // Run starts the main loop for the converger coordinator. It is commonly run
go func() { obj.channel <- struct{}{} }() // from a go routine. It blocks until the Shutdown method is run to close it.
// NOTE: when we have very short timeouts, if we start before all the resources
// have joined the map, then it might appear as if we converged before we did!
func (obj *Coordinator) Run(startPaused bool) {
obj.wg.Add(1)
wg := &sync.WaitGroup{} // needed for the startPaused
defer wg.Wait() // don't leave any leftover go routines running
if startPaused {
wg.Add(1)
go func() {
defer wg.Done()
obj.Pause() // ignore any errors
close(obj.readyChan)
}()
} else {
close(obj.readyChan) // we must wait till the wg.Add(1) has happened...
} }
defer obj.wg.Done()
for {
// pause if one was requested...
select {
case <-obj.pauseSignal: // channel closes
obj.pausedAck.Ack() // send ack
// we are paused now, and waiting for resume or exit...
select {
case <-obj.resumeSignal: // channel closes
// resumed!
case <-obj.closeChan: // we can always escape
return
}
case _, ok := <-obj.pokeChan: // we got an event (re-calculate)
if !ok {
return
}
if err := obj.test(); err != nil {
// FIXME: what to do on error ?
}
case <-obj.closeChan: // we can always escape
return
}
}
}
// Ready blocks until the Run loop has started up. This is useful so that we
// don't run Shutdown before we've even started up properly.
func (obj *Coordinator) Ready() {
select {
case <-obj.readyChan:
}
}
// Shutdown sends a signal to the Run loop that it should exit. This blocks
// until it does.
func (obj *Coordinator) Shutdown() {
close(obj.closeChan)
obj.wg.Wait()
close(obj.pokeChan) // free memory?
}
// Pause pauses the coordinator. It should not be called on an already paused
// coordinator. It will block until the coordinator pauses with an
// acknowledgment, or until an exit is requested. If the latter happens it will
// error. It is NOT thread-safe with the Resume() method so only call either one
// at a time.
func (obj *Coordinator) Pause() error {
if obj.paused {
return fmt.Errorf("already paused")
}
obj.pausedAck = util.NewEasyAck()
obj.resumeSignal = make(chan struct{}) // build the resume signal
close(obj.pauseSignal)
// wait for ack (or exit signal)
select {
case <-obj.pausedAck.Wait(): // we got it!
// we're paused
case <-obj.closeChan:
return fmt.Errorf("closing")
}
obj.paused = true
return nil return nil
} }
// isConverged returns true if *every* registered uid has converged. // Resume unpauses the coordinator. It can be safely called on a brand-new
func (obj *converger) isConverged() bool { // coordinator that has just started running without incident. It is NOT
obj.mutex.RLock() // take a read lock // thread-safe with the Pause() method, so only call either one at a time.
defer obj.mutex.RUnlock() func (obj *Coordinator) Resume() {
for _, v := range obj.status { // TODO: do we need a mutex around Resume?
if !obj.paused { // no need to unpause brand-new resources
return
}
obj.pauseSignal = make(chan struct{}) // rebuild for next pause
close(obj.resumeSignal)
obj.poke() // unblock and notice the resume if necessary
obj.paused = false
// no need to wait for it to resume
//return // implied
}
// poke sends a message to the coordinator telling it that it should re-evaluate
// whether we're converged or not. This does not block. Do not run this in a
// goroutine. It must not be called after Shutdown has been called.
func (obj *Coordinator) poke() {
// redundant
//if len(obj.pokeChan) > 0 {
// return
//}
select {
case obj.pokeChan <- struct{}{}:
default: // if chan is now full because more than one poke happened...
}
}
// IsConverged returns true if *every* registered uid has converged. If there
// are no registered UID's, then this will return true.
func (obj *Coordinator) IsConverged() bool {
for _, v := range obj.Status() {
if !v { // everyone must be converged for this to be true if !v { // everyone must be converged for this to be true
return false return false
} }
@@ -165,145 +277,40 @@ func (obj *converger) isConverged() bool {
return true return true
} }
// Unregister dissociates the ConvergedUID from the converged checking. // test evaluates whether we're converged or not and runs the state change. It
func (obj *converger) Unregister(uid UID) { // is NOT thread-safe.
if !uid.IsValid() { func (obj *Coordinator) test() error {
panic(fmt.Sprintf("the ID of UID(%s) is nil", uid.Name())) // TODO: add these checks elsewhere to prevent anything from running?
} if obj.timeout < 0 {
obj.mutex.Lock() return nil // nothing to do (only run if timeout is valid)
uid.StopTimer() // ignore any errors
delete(obj.status, uid.ID())
obj.mutex.Unlock()
uid.InvalidateID()
}
// Start causes a Converger object to start or resume running.
func (obj *converger) Start() {
obj.control <- true
}
// Pause causes a Converger object to stop running temporarily.
func (obj *converger) Pause() { // FIXME: add a sync ACK on pause before return
obj.control <- false
}
// Loop is the main loop for a Converger object. It usually runs in a goroutine.
// TODO: we could eventually have each resource tell us as soon as it converges,
// and then keep track of the time delays here, to avoid callers needing select.
// NOTE: when we have very short timeouts, if we start before all the resources
// have joined the map, then it might appear as if we converged before we did!
func (obj *converger) Loop(startPaused bool) {
if obj.control == nil {
panic("converger not initialized correctly")
}
if startPaused { // start paused without racing
select {
case e := <-obj.control:
if !e {
panic("converger expected true")
}
}
}
for {
select {
case e := <-obj.control: // expecting "false" which means pause!
if e {
panic("converger expected false")
}
// now i'm paused...
select {
case e := <-obj.control:
if !e {
panic("converger expected true")
}
// restart
// kick once to refresh the check...
go func() { obj.channel <- struct{}{} }()
continue
} }
case <-obj.channel: converged := obj.IsConverged()
if !obj.isConverged() { defer func() {
if obj.converged { // we're doing a state change obj.converged = converged // set this only at the end...
}()
if !converged {
if !obj.converged { // were we previously also not converged?
return nil // nothing to do
}
// we're doing a state change
// call the arbitrary functions (takes a read lock!) // call the arbitrary functions (takes a read lock!)
if err := obj.runStateFns(false); err != nil { return obj.runStateFns(false)
// FIXME: what to do on error ?
}
}
obj.converged = false
continue
} }
// we have converged! // we have converged!
if obj.timeout >= 0 { // only run if timeout is valid if obj.converged { // were we previously also converged?
if !obj.converged { // we're doing a state change return nil // nothing to do
}
// call the arbitrary functions (takes a read lock!) // call the arbitrary functions (takes a read lock!)
if err := obj.runStateFns(true); err != nil { return obj.runStateFns(true)
// FIXME: what to do on error ?
}
}
}
obj.converged = true
// loop and wait again...
}
}
} }
// ConvergedTimer adds a timeout to a select call and blocks until then. // runStateFns runs the list of stored state functions.
// TODO: this means we could eventually have per resource converged timeouts func (obj *Coordinator) runStateFns(converged bool) error {
func (obj *converger) ConvergedTimer(uid UID) <-chan time.Time {
// be clever: if i'm already converged, this timeout should block which
// avoids unnecessary new signals being sent! this avoids fast loops if
// we have a low timeout, or in particular a timeout == 0
if uid.IsConverged() {
// blocks the case statement in select forever!
return util.TimeAfterOrBlock(-1)
}
return util.TimeAfterOrBlock(obj.timeout)
}
// Status returns a map of the converged status of each UID.
func (obj *converger) Status() map[uint64]bool {
status := make(map[uint64]bool)
obj.mutex.RLock() // take a read lock
defer obj.mutex.RUnlock()
for k, v := range obj.status { // make a copy to avoid the mutex
status[k] = v
}
return status
}
// Timeout returns the timeout in seconds that converger was created with. This
// is useful to avoid passing in the timeout value separately when you're
// already passing in the Converger struct.
func (obj *converger) Timeout() int {
return obj.timeout
}
// AddStateFn adds a state function to be run on change of converged state.
func (obj *converger) AddStateFn(name string, stateFn func(bool) error) error {
obj.smutex.Lock()
defer obj.smutex.Unlock()
if _, exists := obj.stateFns[name]; exists {
return fmt.Errorf("a stateFn with that name already exists")
}
obj.stateFns[name] = stateFn
return nil
}
// RemoveStateFn adds a state function to be run on change of converged state.
func (obj *converger) RemoveStateFn(name string) error {
obj.smutex.Lock()
defer obj.smutex.Unlock()
if _, exists := obj.stateFns[name]; !exists {
return fmt.Errorf("a stateFn with that name doesn't exist")
}
delete(obj.stateFns, name)
return nil
}
// runStateFns runs the listed of stored state functions.
func (obj *converger) runStateFns(converged bool) error {
obj.smutex.RLock() obj.smutex.RLock()
defer obj.smutex.RUnlock() defer obj.smutex.RUnlock()
var keys []string var keys []string
@@ -315,77 +322,125 @@ func (obj *converger) runStateFns(converged bool) error {
for _, name := range keys { // run in deterministic order for _, name := range keys { // run in deterministic order
fn := obj.stateFns[name] fn := obj.stateFns[name]
// call an arbitrary function // call an arbitrary function
if e := fn(converged); e != nil { e := fn(converged)
err = multierr.Append(err, e) // list of errors err = errwrap.Append(err, e) // list of errors
}
} }
return err return err
} }
// ID returns the unique id of this UID object. // AddStateFn adds a state function to be run on change of converged state.
func (obj *cuid) ID() uint64 { func (obj *Coordinator) AddStateFn(name string, stateFn func(bool) error) error {
return obj.id obj.smutex.Lock()
defer obj.smutex.Unlock()
if _, exists := obj.stateFns[name]; exists {
return fmt.Errorf("a stateFn with that name already exists")
}
obj.stateFns[name] = stateFn
return nil
} }
// Name returns a user defined name for the specific cuid. // RemoveStateFn removes a state function from running on change of converged
func (obj *cuid) Name() string { // state.
return obj.name func (obj *Coordinator) RemoveStateFn(name string) error {
obj.smutex.Lock()
defer obj.smutex.Unlock()
if _, exists := obj.stateFns[name]; !exists {
return fmt.Errorf("a stateFn with that name doesn't exist")
}
delete(obj.stateFns, name)
return nil
} }
// SetName sets a user defined name for the specific cuid. // Status returns a map of the converged status of each UID.
func (obj *cuid) SetName(name string) { func (obj *Coordinator) Status() map[*UID]bool {
obj.name = name status := make(map[*UID]bool)
obj.mutex.RLock() // take a read lock
defer obj.mutex.RUnlock()
for k := range obj.status {
status[k] = k.IsConverged()
}
return status
} }
// IsValid tells us if the id is valid or has already been destroyed. // Timeout returns the timeout in seconds that converger was created with. This
func (obj *cuid) IsValid() bool { // is useful to avoid passing in the timeout value separately when you're
return obj.id != 0 // an id of 0 is invalid // already passing in the Coordinator struct.
func (obj *Coordinator) Timeout() int64 {
return obj.timeout
} }
// InvalidateID marks the id as no longer valid. // UID represents one of the probes for the converger coordinator. It is created
func (obj *cuid) InvalidateID() { // by calling the Register method of the Coordinator struct. It should be freed
obj.id = 0 // an id of 0 is invalid // after use with Unregister.
type UID struct {
// timeout is a copy of the main timeout. It could eventually be used
// for per-UID timeouts too.
timeout int64
// isConverged stores the convergence state of this particular UID.
isConverged bool
// poke stores a reference to the main poke function.
poke func()
// unregister stores a reference to the unregister function.
unregister func()
// timer
mutex *sync.Mutex
timer chan struct{}
running bool // is the timer running?
wg *sync.WaitGroup
} }
// IsConverged is a helper function to the regular IsConverged method. // Unregister removes this UID from the converger coordinator. An unregistered
func (obj *cuid) IsConverged() bool { // UID is no longer part of the convergence checking.
return obj.converger.IsConverged(obj) func (obj *UID) Unregister() {
obj.unregister()
} }
// SetConverged is a helper function to the regular SetConverged notification. // IsConverged reports whether this UID is converged or not.
func (obj *cuid) SetConverged(isConverged bool) error { func (obj *UID) IsConverged() bool {
return obj.converger.SetConverged(obj, isConverged) return obj.isConverged
} }
// Unregister is a helper function to unregister myself. // SetConverged sets the convergence state of this UID. This is used by the
func (obj *cuid) Unregister() { // running timer if one is started. The timer will overwrite any value set by
obj.converger.Unregister(obj) // this method.
func (obj *UID) SetConverged(isConverged bool) {
obj.isConverged = isConverged
obj.poke() // notify of change
} }
// ConvergedTimer is a helper around the regular ConvergedTimer method. // ConvergedTimer adds a timeout to a select call and blocks until then.
func (obj *cuid) ConvergedTimer() <-chan time.Time { // TODO: this means we could eventually have per resource converged timeouts
return obj.converger.ConvergedTimer(obj) func (obj *UID) ConvergedTimer() <-chan time.Time {
// be clever: if i'm already converged, this timeout should block which
// avoids unnecessary new signals being sent! this avoids fast loops if
// we have a low timeout, or in particular a timeout == 0
if obj.IsConverged() {
// blocks the case statement in select forever!
return util.TimeAfterOrBlock(-1)
}
return util.TimeAfterOrBlock(int(obj.timeout))
} }
// StartTimer runs an invisible timer that automatically converges on timeout. // StartTimer runs a timer that sets us as converged on timeout. It also returns
func (obj *cuid) StartTimer() (func() error, error) { // a handle to the StopTimer function which should be run before exit.
func (obj *UID) StartTimer() (func() error, error) {
obj.mutex.Lock() obj.mutex.Lock()
if !obj.running { defer obj.mutex.Unlock()
obj.timer = make(chan struct{}) if obj.running {
obj.running = true
} else {
obj.mutex.Unlock()
return obj.StopTimer, fmt.Errorf("timer already started") return obj.StopTimer, fmt.Errorf("timer already started")
} }
obj.mutex.Unlock() obj.timer = make(chan struct{})
obj.running = true
obj.wg.Add(1) obj.wg.Add(1)
go func() { go func() {
defer obj.wg.Done() defer obj.wg.Done()
for { for {
select { select {
case _, ok := <-obj.timer: // reset signal channel case _, ok := <-obj.timer: // reset signal channel
if !ok { // channel is closed if !ok {
return // false to exit return
} }
obj.SetConverged(false) obj.SetConverged(false)
@@ -393,8 +448,8 @@ func (obj *cuid) StartTimer() (func() error, error) {
obj.SetConverged(true) // converged! obj.SetConverged(true) // converged!
select { select {
case _, ok := <-obj.timer: // reset signal channel case _, ok := <-obj.timer: // reset signal channel
if !ok { // channel is closed if !ok {
return // false to exit return
} }
} }
} }
@@ -403,8 +458,8 @@ func (obj *cuid) StartTimer() (func() error, error) {
return obj.StopTimer, nil return obj.StopTimer, nil
} }
// ResetTimer resets the counter to zero if using a StartTimer internally. // ResetTimer resets the timer to zero.
func (obj *cuid) ResetTimer() error { func (obj *UID) ResetTimer() error {
obj.mutex.Lock() obj.mutex.Lock()
defer obj.mutex.Unlock() defer obj.mutex.Unlock()
if obj.running { if obj.running {
@@ -414,8 +469,8 @@ func (obj *cuid) ResetTimer() error {
return fmt.Errorf("timer hasn't been started") return fmt.Errorf("timer hasn't been started")
} }
// StopTimer stops the running timer permanently until a StartTimer is run. // StopTimer stops the running timer.
func (obj *cuid) StopTimer() error { func (obj *UID) StopTimer() error {
obj.mutex.Lock() obj.mutex.Lock()
defer obj.mutex.Unlock() defer obj.mutex.Unlock()
if !obj.running { if !obj.running {

31
converger/converger_test.go Normal file
View File

@@ -0,0 +1,31 @@
// Mgmt
// Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
// +build !root
package converger
import (
"testing"
)
func TestBufferedChan1(t *testing.T) {
ch := make(chan bool, 1)
ch <- true
close(ch) // closing a channel that's not empty should not block
// must be able to exit without blocking anywhere
}

2
debian/copyright vendored
View File

@@ -3,7 +3,7 @@ Upstream-Name: mgmt
Source: <https://github.com/purpleidea/mgmt> Source: <https://github.com/purpleidea/mgmt>
Files: * Files: *
Copyright: Copyright (C) 2013-2018+ James Shubin and the project contributors Copyright: Copyright (C) 2013-2019+ James Shubin and the project contributors
License: GPL-3.0 License: GPL-3.0
License: GPL-3.0 License: GPL-3.0

2
doc.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

4
docker/Dockerfile.development
View File

@@ -1,10 +1,10 @@
FROM golang:1.9 FROM golang:1.11
MAINTAINER Michał Czeraszkiewicz <contact@czerasz.com> MAINTAINER Michał Czeraszkiewicz <contact@czerasz.com>
# Set the reset cache variable # Set the reset cache variable
# Read more here: http://czerasz.com/2014/11/13/docker-tip-and-tricks/#use-refreshedat-variable-for-better-cache-control # Read more here: http://czerasz.com/2014/11/13/docker-tip-and-tricks/#use-refreshedat-variable-for-better-cache-control
ENV REFRESHED_AT 2017-11-16 ENV REFRESHED_AT 2019-02-06
RUN apt-get update RUN apt-get update

2
docs/conf.py
View File

@@ -51,7 +51,7 @@ master_doc = 'index'
# General information about the project. # General information about the project.
project = u'mgmt' project = u'mgmt'
copyright = u'2013-2018+ James Shubin and the project contributors' copyright = u'2013-2019+ James Shubin and the project contributors'
author = u'James Shubin' author = u'James Shubin'
# The version info for the project you're documenting, acts as replacement for # The version info for the project you're documenting, acts as replacement for

119
docs/development.md
View File

@@ -3,7 +3,119 @@
This document contains some additional information and help regarding This document contains some additional information and help regarding
developing `mgmt`. Useful tools, conventions, etc. developing `mgmt`. Useful tools, conventions, etc.
Be sure to read [quick start guide](docs/quick-start-guide.md) first. Be sure to read [quick start guide](quick-start-guide.md) first.
## Vagrant
If you would like to avoid doing the above steps manually, we have prepared a
[Vagrant](https://www.vagrantup.com/) environment for your convenience. From the
project directory, run a `vagrant up`, and then a `vagrant status`. From there,
you can `vagrant ssh` into the `mgmt` machine. The `MOTD` will explain the rest.
This environment isn't commonly used by the `mgmt` developers, so it might not
be working properly.
## Using Docker
Alternatively, you can check out the [docker-guide](docker-guide.md) in order to
develop or deploy using docker. This method is not endorsed or supported, so use
at your own risk, as it might not be working properly.
## Information about dependencies
Software projects have a few different kinds of dependencies. There are _build_
dependencies, _runtime_ dependencies, and additionally, a few extra dependencies
required for running the _test_ suite.
### Build
* `golang` 1.11 or higher (required, available in some distros and distributed
as a binary officially by [golang.org](https://golang.org/dl/))
### Runtime
A relatively modern GNU/Linux system should be able to run `mgmt` without any
problems. Since `mgmt` runs as a single statically compiled binary, all of the
library dependencies are included. It is expected, that certain advanced
resources require host specific facilities to work. These requirements are
listed below:
| Resource | Dependency | Version | Check version with |
|----------|-------------------|-----------------------------|-----------------------------------------------------------|
| augeas | augeas-devel | `augeas 1.6` or greater | `dnf info augeas-devel` or `apt-cache show libaugeas-dev` |
| file | inotify | `Linux 2.6.27` or greater | `uname -a` |
| hostname | systemd-hostnamed | `systemd 25` or greater | `systemctl --version` |
| nspawn | systemd-nspawn | `systemd ???` or greater | `systemctl --version` |
| pkg | packagekitd | `packagekit 1.x` or greater | `pkcon --version` |
| svc | systemd | `systemd ???` or greater | `systemctl --version` |
| virt | libvirt-devel | `libvirt 1.2.0` or greater | `dnf info libvirt-devel` or `apt-cache show libvirt-dev` |
| virt | libvirtd | `libvirt 1.2.0` or greater | `libvirtd --version` |
For building a visual representation of the graph, `graphviz` is required.
To build `mgmt` without augeas support please run:
`GOTAGS='noaugeas' make build`
To build `mgmt` without libvirt support please run:
`GOTAGS='novirt' make build`
To build `mgmt` without docker support please run:
`GOTAGS='nodocker' make build`
To build `mgmt` without augeas, libvirt or docker support please run:
`GOTAGS='noaugeas novirt nodocker' make build`
## OSX/macOS/Darwin development
Developing and running `mgmt` on macOS is currently not supported (but not
discouraged either). Meaning it might work but in the case it doesn't you would
have to provide your own patches to fix problems (the project maintainer and
community are glad to assist where needed).
There are currently some issues that make `mgmt` less suitable to run for
provisioning macOS. But as a client to provision remote servers it should run
fine.
Since the primary supported systems are Linux and these are the environments
tested, it is wise to run these suites during macOS development as well. To ease
this, Docker can be leveraged ([Docker for Mac](https://docs.docker.com/docker-for-mac/)).
Before running any of the commands below create the development Docker image:
```
docker/scripts/build-development
```
This image requires updating every time dependencies (`make-deps.sh`) changes.
Then to run the test suite:
```
docker run --rm -ti \
-v $PWD:/go/src/github.com/purpleidea/mgmt/ \
-w /go/src/github.com/purpleidea/mgmt/ \
purpleidea/mgmt:development \
make test
```
For convenience this command is wrapped in `docker/scripts/exec-development`.
Basically any command can be executed this way. Because the repository source is
mounted into the Docker container invocation will be quick and allow rapid
testing, for example:
```
docker/scripts/exec-development test/test-shell.sh load0.sh
```
Other examples:
```
docker/scripts/exec-development make build
docker/scripts/exec-development ./mgmt run --tmp-prefix lang examples/lang/load0.mcl
```
Be advised that this method is not supported and it might not be working
properly.
## Testing ## Testing
@@ -45,5 +157,6 @@ individual tests to run.
### IDE/Editor support ### IDE/Editor support
- Emacs: see `misc/emacs/` * Emacs: see `misc/emacs/`
- [Textmate](https://github.com/aequitas/mgmt.tmbundle) * [Textmate](https://github.com/aequitas/mgmt.tmbundle)
* [VSCode](https://github.com/aequitas/mgmt.vscode)

45
docs/documentation.md
View File

@@ -147,7 +147,7 @@ Invoke `mgmt` with the `--puppet` switch, which supports 3 variants:
`mgmt run puppet --puppet 'file { "/etc/ntp.conf": ensure => file }'` `mgmt run puppet --puppet 'file { "/etc/ntp.conf": ensure => file }'`
For more details and caveats see [Puppet.md](Puppet.md). For more details and caveats see [puppet-guide.md](puppet-guide.md).
#### Blog post #### Blog post
@@ -250,6 +250,43 @@ integer, then that value is the max size for that semaphore. Valid semaphore
id's include: `some_id`, `hello:42`, `not:smart:4` and `:13`. It is expected id's include: `some_id`, `hello:42`, `not:smart:4` and `:13`. It is expected
that the last bare example be only used by the engine to add a global semaphore. that the last bare example be only used by the engine to add a global semaphore.
#### Rewatch
Boolean. Rewatch specifies whether we re-run the Watch worker during a graph
swap if it has errored. When doing a graph compare to swap the graphs, if this
is true, and this particular worker has errored, then we'll remove it and add it
back as a new vertex, thus causing it to run again. This is different from the
`Retry` metaparam which applies during the normal execution. It is only when
this is exhausted that we're in permanent worker failure, and only then can we
rely on this metaparam.
#### Realize
Boolean. Realize ensures that the resource is guaranteed to converge at least
once before a potential graph swap removes or changes it. This guarantee is
useful for fast changing graphs, to ensure that the brief creation of a resource
is seen. This guarantee does not prevent against the engine quitting normally,
and it can't guarantee it if the resource is blocked because of a failed
pre-requisite resource.
*XXX: This is currently not implemented!*
#### Reverse
Boolean. Reverse is a property that some resources can implement that specifies
that some "reverse" operation should happen when that resource "disappears". A
disappearance happens when a resource is defined in one instance of the graph,
and is gone in the subsequent one. This disappearance can happen if it was
previously in an if statement that then becomes false.
This is helpful for building robust programs with the engine. The engine adds a
"reversed" resource to that subsequent graph to accomplish the desired "reverse"
mechanics. The specifics of what this entails is a property of the particular
resource that is being "reversed".
It might be wise to combine the use of this meta parameter with the use of the
`realize` meta parameter to ensure that your reversed resource actually runs at
least once, if there's a chance that it might be gone for a while.
### Lang metadata file ### Lang metadata file
Any module *must* have a metadata file in its root. It must be named Any module *must* have a metadata file in its root. It must be named
@@ -298,10 +335,6 @@ recommended that you use this, since it's preferable to write code in the
The main interface to the `mgmt` tool is the command line. For the most recent The main interface to the `mgmt` tool is the command line. For the most recent
documentation, please run `mgmt --help`. documentation, please run `mgmt --help`.
#### `--yaml <graph.yaml>`
Point to a graph file to run.
#### `--converged-timeout <seconds>` #### `--converged-timeout <seconds>`
Exit if the machine has converged for approximately this many seconds. Exit if the machine has converged for approximately this many seconds.
@@ -455,7 +488,7 @@ To report any bugs, please file a ticket at: [https://github.com/purpleidea/mgmt
## Authors ## Authors
Copyright (C) 2013-2018+ James Shubin and the project contributors Copyright (C) 2013-2019+ James Shubin and the project contributors
Please see the Please see the
[AUTHORS](https://github.com/purpleidea/mgmt/tree/master/AUTHORS) file [AUTHORS](https://github.com/purpleidea/mgmt/tree/master/AUTHORS) file

166
docs/faq.md
View File

@@ -9,6 +9,18 @@ I wanted a next generation config management solution that didn't have all of
the design flaws or limitations that the current generation of tools do, and no the design flaws or limitations that the current generation of tools do, and no
tool existed! tool existed!
### Why did you choose `golang` for the project?
When I started working on the project, I needed to choose a language that
already had an implementation of a distributed consensus algorithm available.
That meant [Paxos](https://en.wikipedia.org/wiki/Paxos_(computer_science)) or
[Raft](https://en.wikipedia.org/wiki/Raft_(computer_science)). Golang was one
language that actually had two different Raft implementations, `etcd`, and
`consul`. Other design requirements included something that was reasonably fast,
typed and memory-safe, and suited for systems engineering. After a reasonably
extensive search, I chose `golang`. I think it was the right decision. There are
a number of other features of the language which helped influence the decision.
### How do I contribute to the project if I don't know `golang`? ### How do I contribute to the project if I don't know `golang`?
There are many different ways you can contribute to the project. They can be There are many different ways you can contribute to the project. They can be
@@ -125,6 +137,58 @@ The downside to this approach is that you won't benefit from the automatic
elastic nature of the embedded etcd servers, and that you're responsible if you elastic nature of the embedded etcd servers, and that you're responsible if you
accidentally break your etcd cluster, or if you use an unsupported version. accidentally break your etcd cluster, or if you use an unsupported version.
### In `mgmt` you talk about events. What is this referring to?
Mgmt has two main concepts that involve "events":
1. Events in the [resource primitive](resource-guide.md).
2. Events in the [reactive language](language-guide.md).
Each resource primitive in mgmt can test (check) and set (apply) the desired
state that was requested of it. This is familiar to what is common with existing
tools such as `Puppet`, `Ansible`, `Chef`, `Terraform`, etc... In addition,
`mgmt` can also **watch** the state and detect changes. As a result, it never
has to waste time and cpu resources by polling to test and set state, leading to
a design which is algorithmically much faster than the existing generation of
tools.
To describe the set of resources to apply, mgmt describes this collection with a
language. In order to model the time component of infrastructure, we use a
special kind of language called an [FRP](https://en.wikipedia.org/wiki/Functional_reactive_programming).
This language has a built-in concept that we call "events", and which means that
we re-evaluate the relevant portions of the code whenever a value or function
has an event that tells us that it changed. The `R` in `FRP` stands for
reactive. This is similar to how a spreadsheet updates dependent cells when a
pre-requisite value is modified. [This article](https://en.wikipedia.org/wiki/Reactive_programming)
provides a bit more background.
Whenever any of the streams of values in the language change, the program is
partially re-evaluated. The output of any mgmt program is a [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph)
of resources, or more precisely, a stream of resource graphs. Since we have
events per-resource, we can efficiently switch from one desired-state resource
graph to the next without re-checking their individual states, since we've been
monitoring them all along.
One side-effect of all this, is that if a rogue systems administrator manually
changes the state of any managed resource, mgmt will detect this and attempt to
revert the change. This makes for excellent live demos, but is not the primary
design goal. It is a consequence of tracking state so that graph changes are
efficient. We implement the event detection via an intentional per-resource
[main loop](https://en.wikipedia.org/wiki/Event_loop) which can enable other
interesting functionality too!
Make sure to get rid of your rogue sysadmin! ;)
### Do I need to run `mgmt` as `root`?
No and yes. It depends. Nothing in mgmt explicitly requires root in the design,
however mgmt will require root only if the changes to your system that you want
it to make require root.
For example, if you use it to manage files that require root access to modify,
then you'll need root. If you only use it to manage files and resources
elsewhere, then it shouldn't need root. Many resources are perfectly usable
without root, and virtually all of my live demos are done without root.
### How can I run `mgmt` on-demand, or in `cron`, instead of continuously? ### How can I run `mgmt` on-demand, or in `cron`, instead of continuously?
By default, `mgmt` will run continuously in an attempt to keep your machine in a By default, `mgmt` will run continuously in an attempt to keep your machine in a
@@ -148,43 +212,80 @@ requires a number of seconds as an argument.
#### Example: #### Example:
``` ```
./mgmt run lang --lang examples/lang/hello0.mcl --converged-timeout=5 ./mgmt run lang examples/lang/hello0.mcl --converged-timeout=5
``` ```
### What does the error message about an inconsistent dataDir mean? ### When I try to build `mgmt` I see: `no Go files in $GOPATH/src/github.com/purpleidea/mgmt/bindata`.
Due to the arcane way that `golang` designed its `$GOPATH`, the main project
directory must be inside your `$GOPATH`, and at the appropriate FQDN. This is:
`$GOPATH/src/github.com/purpleidea/mgmt/`. If you have your project root outside
of that directory, then you may get this error when you try to build it. In this
case there is likely a `go get` version of the project at this location. Remove
it and replace it with your git cloned directory. In my case, I like to work on
things in `~/code/mgmt/`, so that path is a symlink that points to the long
project directory.
### Why does my file resource error with `no such file or directory`?
If you create a file resource and only specify the content like this:
```
file "/tmp/foo" {
content => "hello world\n",
}
```
Then this will attempt to set the contents of that file to the desired string,
but *only* if that file already exists. If you'd like to ensure that it also
gets created in case it is not present, then you must also specify the state:
```
file "/tmp/foo" {
state => "exists",
content => "hello world\n",
}
```
Similar logic applies for situations when you only specify the `mode` parameter.
This all turns out to be more safe and "correct", in that it would error and
prevent masking an error for a situation when you expected a file to already be
at that location. It also turns out to simplify the internals significantly, and
remove an ambiguous scenario with the reversable file resource.
### On startup `mgmt` hangs after: `etcd: server: starting...`.
If you get an error message similar to: If you get an error message similar to:
``` ```
Etcd: Connect: CtxError... etcd: server: starting...
Etcd: CtxError: Reason: CtxDelayErr(5s): No endpoints available yet! etcd: server: start timeout of 1m0s reached
Etcd: Connect: Endpoints: [] etcd: server: close timeout of 15s reached
Etcd: The dataDir (/var/lib/mgmt/etcd) might be inconsistent or corrupt.
``` ```
This happens when there are a series of fatal connect errors in a row. This can But nothing happens afterwards, this can be due to a corrupt etcd storage
happen when you start `mgmt` using a dataDir that doesn't correspond to the directory. Each etcd server embedded in mgmt must have a special directory where
current cluster view. As a result, the embedded etcd server never finishes it stores local state. It must not be shared by more than one individual member.
starting up, and as a result, a default endpoint never gets added. The solution This dir is typically `/var/lib/mgmt/etcd/member/`. If you accidentally use it
is to either reconcile the mistake, and if there is no important data saved, you (for example during testing) with a different cluster view, then you can corrupt
can remove the etcd dataDir. This is typically `/var/lib/mgmt/etcd/member/`. it. This can happen if you use it with more than one different hostname.
### Why do resources have both a `Cmp` method and an `IFF` (on the UID) method? The solution is to avoid making this mistake, and if there is no important data
saved, you can remove the etcd member dir and start over.
The `Cmp()` methods are for determining if two resources are effectively the ### On running `make` to build a new version, it errors with: `Text file busy`.
same, which is used to make graph change delta's efficient. This is when we want
to change from the current running graph to a new graph, but preserve the common
vertices. Since we want to make this process efficient, we only update the parts
that are different, and leave everything else alone. This `Cmp()` method can
tell us if two resources are the same. In case it is not obvious, `cmp` is an
abbrev. for compare.
The `IFF()` method is part of the whole UID system, which is for discerning if a If you get an error like:
resource meets the requirements another expects for an automatic edge. This is
because the automatic edge system assumes a unified UID pattern to test for ```
equality. In the future it might be helpful or sane to merge the two similar cp: cannot create regular file 'mgmt': Text file busy
comparison functions although for now they are separate because they are ```
actually answer different questions.
This can happen if you ran `make build` (or just `make`) when there was already
an instance of mgmt running, or if a related file locking issue occurred. To
solve this, shutdown and running mgmt process, run `rm mgmt` to remove the file,
and then get a new one by running `make` again.
### Does this support Windows? OSX? GNU Hurd? ### Does this support Windows? OSX? GNU Hurd?
@@ -193,7 +294,7 @@ serious automation workloads. Support for non-Linux operating systems isn't a
high priority of mine, but we're happy to accept patches for missing features high priority of mine, but we're happy to accept patches for missing features
or resources that you think would make sense on your favourite platform. or resources that you think would make sense on your favourite platform.
### Why aren't you using `glide` or `godep` for dependency management? ### Why aren't you using `glide`, `godep` or `go mod` for dependency management?
Vendoring dependencies means that as the git master branch of each dependency Vendoring dependencies means that as the git master branch of each dependency
marches on, you're left behind using an old version. As a result, bug fixes and marches on, you're left behind using an old version. As a result, bug fixes and
@@ -262,9 +363,8 @@ which definitely existed before the band did.
### You didn't answer my question, or I have a question! ### You didn't answer my question, or I have a question!
It's best to ask on [IRC](https://webchat.freenode.net/?channels=#mgmtconfig) It's best to ask on [IRC](https://webchat.freenode.net/?channels=#mgmtconfig)
to see if someone can help you. Once we get a big enough community going, we'll to see if someone can help you. If you don't get a response from IRC, you can
add a mailing list. If you don't get any response from the above, you can contact me through my [technical blog](https://purpleidea.com/contact/) and I'll
contact me through my [technical blog](https://purpleidea.com/contact/) do my best to help. If you have a good question, please add it as a patch to
and I'll do my best to help. If you have a good question, please add it as a this documentation. I'll merge your question, and add a patch with the answer!
patch to this documentation. I'll merge your question, and add a patch with the For news and updates, subscribe to the [mailing list](https://www.redhat.com/mailman/listinfo/mgmtconfig-list).
answer!

8
docs/function-guide.md
View File

@@ -219,7 +219,7 @@ Init(init *interfaces.Init) error
This is called to initialize the function. If something goes wrong, it should This is called to initialize the function. If something goes wrong, it should
return an error. It is passed a struct that contains all the important return an error. It is passed a struct that contains all the important
information and poiinters that it might need to work with throughout its information and pointers that it might need to work with throughout its
lifetime. As a result, it will need to save a copy to that pointer for future lifetime. As a result, it will need to save a copy to that pointer for future
use in the other methods. use in the other methods.
@@ -377,9 +377,9 @@ might be different ways you would want to call `printf`, such as:
`printf("the %s is %d", "answer", 42)` or `printf("3 * 2 = %d", 3 * 2)`. Since `printf("the %s is %d", "answer", 42)` or `printf("3 * 2 = %d", 3 * 2)`. Since
you couldn't implement the infinite number of possible signatures, this API lets you couldn't implement the infinite number of possible signatures, this API lets
you write code which can be coerced into different forms. This makes you write code which can be coerced into different forms. This makes
implementing what would appear to be generic or polymorphic, instead something implementing what would appear to be generic or polymorphic, instead of
that is actually static and that still has the static type safety properties something that is actually static and that still has the static type safety
that were guaranteed by the mgmt language. properties that were guaranteed by the mgmt language.
Since this is an advanced topic, it is not described in full at this time. For Since this is an advanced topic, it is not described in full at this time. For
more information please have a look at the source code comments, some of the more information please have a look at the source code comments, some of the

3
docs/language-guide.md
View File

@@ -511,6 +511,9 @@ without making any changes. The `ExprVar` node naturally consumes scope's and
the `StmtProg` node cleverly passes the scope through in the order expected for the `StmtProg` node cleverly passes the scope through in the order expected for
the out-of-order bind logic to work. the out-of-order bind logic to work.
This step typically calls the ordering algorithm to determine the correct order
of statements in a program.
#### Type unification #### Type unification
Each expression must have a known type. The unpleasant option is to force the Each expression must have a known type. The unpleasant option is to force the

8
docs/on-the-web.md
View File

@@ -44,3 +44,11 @@ if we missed something that you think is relevant!
| James Shubin | blog | [Mgmt Configuration Language](https://purpleidea.com/blog/2018/02/05/mgmt-configuration-language/) | | James Shubin | blog | [Mgmt Configuration Language](https://purpleidea.com/blog/2018/02/05/mgmt-configuration-language/) |
| James Shubin | video | [Recording from CfgMgmtCamp.eu 2018](https://www.youtube.com/watch?v=NxObmwZDyrI) | | James Shubin | video | [Recording from CfgMgmtCamp.eu 2018](https://www.youtube.com/watch?v=NxObmwZDyrI) |
| Jonathan Gold | blog | [Go Netlink and Select](https://jonathangold.ca/blog/go-netlink-and-select/) | | Jonathan Gold | blog | [Go Netlink and Select](https://jonathangold.ca/blog/go-netlink-and-select/) |
| James Shubin | video | [Recording from DevOpsDays Montreal 2018](https://www.youtube.com/watch?v=1i38c5cooHo) |
| James Shubin | video | [Recording from FOSDEM Minimalistic Languages Devroom 2019](https://video.fosdem.org/2019/K.4.201/mgmtconfig.webm) |
| James Shubin | video | [Recording from FOSDEM Infra Management Devroom 2019](https://video.fosdem.org/2019/UB2.252A/mgmt.webm) |
| James Shubin | video | [Recording from FOSDEM Graph Processing Devroom 2019](https://video.fosdem.org/2019/H.1308/graph_mgmt_config.webm) |
| James Shubin | video | [Recording from FOSDEM Virtualization Devroom 2019](https://video.fosdem.org/2019/H.2213/vai_real_time_virtualization_automation.webm) |
| James Shubin | video | [Recording from FOSDEM Containers Devroom 2019](https://video.fosdem.org/2019/UA2.114/containers_mgmt.webm) |
| James Shubin | video | [Recording from FOSDEM Monitoring Devroom 2019](https://video.fosdem.org/2019/UB2.252A/real_time_merging_of_config_management_and_monitoring.webm) |
| James Shubin | blog | [Mgmt Configuration Language: Class and Include](https://purpleidea.com/blog/2019/07/26/class-and-include-in-mgmt/) |

2
docs/puppet-guide.md
View File

@@ -173,7 +173,7 @@ useful when you are in the process of replacing Puppet with mgmt. You
can translate your custom modules into mgmt's language one by one, can translate your custom modules into mgmt's language one by one,
and let mgmt run the current mix. and let mgmt run the current mix.
Instead of the usual `--puppet`, `--puppet-conf`, and `--lang` for mcl, Instead of the usual `--puppet-conf` flag and argv for `puppet` and `mcl` input,
you need to use alternative flags to make this work: you need to use alternative flags to make this work:
* `--lp-lang` to specify the mcl input * `--lp-lang` to specify the mcl input

214
docs/quick-start-guide.md
View File

@@ -2,65 +2,108 @@
## Introduction ## Introduction
This guide is intended for developers. Once `mgmt` is minimally viable, we'll This guide is intended for users and developers. If you're brand new to `mgmt`,
publish a quick start guide for users too. If you're brand new to `mgmt`, it's it's probably a good idea to start by reading an
probably a good idea to start by reading the [introductory article about the engine](https://purpleidea.com/blog/2016/01/18/next-generation-configuration-mgmt/)
[introductory article](https://purpleidea.com/blog/2016/01/18/next-generation-configuration-mgmt/) and an [introductory article about the language](https://purpleidea.com/blog/2018/02/05/mgmt-configuration-language/).
or to watch an [introductory video](https://www.youtube.com/watch?v=LkEtBVLfygE&html5=1). [There are other articles and videos available](on-the-web.md) if you'd like to
Once you're familiar with the general idea, please start hacking... learn more or prefer different formats. Once you're familiar with the general
idea, or if you prefer a hands-on approach, please start hacking...
## Quick start ## Getting mgmt
### Installing golang You can either build `mgmt` from source, or you can download a pre-built
release. There are also some distro repositories available, but they may not be
up to date. A pre-built release is the fastest option if there's one that's
available for your platform. If you are developing or testing a new patch to
`mgmt`, or there is not a release available for your platform, then you'll have
to build your own.
* You need golang version 1.10 or greater installed. ### Downloading a pre-built release:
The latest releases can be found [here](https://github.com/purpleidea/mgmt/releases/).
An alternate mirror is available [here](https://dl.fedoraproject.org/pub/alt/purpleidea/mgmt/releases/).
Make sure to verify the signatures of all packages before you use them. The
signing key can be downloaded from [https://purpleidea.com/contact/#pgp-key](https://purpleidea.com/contact/#pgp-key)
to verify the release.
If you've decided to install a pre-build release, you can skip to the
[Running mgmt](#running-mgmt) section below!
### Building a release:
You'll need some dependencies, including `golang`, and some associated tools.
#### Installing golang
* You need golang version 1.11 or greater installed.
* To install on rpm style systems: `sudo dnf install golang` * To install on rpm style systems: `sudo dnf install golang`
* To install on apt style systems: `sudo apt install golang` * To install on apt style systems: `sudo apt install golang`
* To install on macOS systems install [Homebrew](https://brew.sh) * To install on macOS systems install [Homebrew](https://brew.sh)
and run: `brew install go` and run: `brew install go`
* You can run `go version` to check the golang version. * You can run `go version` to check the golang version.
* If your distro is tool old, you may need to [download](https://golang.org/dl/) * If your distro is too old, you may need to [download](https://golang.org/dl/)
a newer golang version. a newer golang version.
### Setting up golang #### Setting up golang
* If you do not have a GOPATH yet, create one and export it: * You can skip this step, as your installation will default to using `~/go/`,
but if you do not have a `GOPATH` yet and want one in a custom location, create
one and export it:
``` ```shell
mkdir $HOME/gopath mkdir $HOME/gopath
export GOPATH=$HOME/gopath export GOPATH=$HOME/gopath
``` ```
* You might also want to add the GOPATH to your `~/.bashrc` or `~/.profile`. * You might also want to add the GOPATH to your `~/.bashrc` or `~/.profile`.
* For more information you can read the [GOPATH documentation](https://golang.org/cmd/go/#hdr-GOPATH_environment_variable). * For more information you can read the
[GOPATH documentation](https://golang.org/cmd/go/#hdr-GOPATH_environment_variable).
### Getting the mgmt code and dependencies #### Getting the mgmt code and associated dependencies
* Download the `mgmt` code into the GOPATH, and switch to that directory: * Download the `mgmt` code into the `GOPATH`, and switch to that directory:
``` ```shell
mkdir -p $GOPATH/src/github.com/purpleidea/ [ -z "$GOPATH" ] && mkdir ~/go/ || mkdir -p $GOPATH/src/github.com/purpleidea/
cd $GOPATH/src/github.com/purpleidea/ cd $GOPATH/src/github.com/purpleidea/ || cd ~/go/
git clone --recursive https://github.com/purpleidea/mgmt/ git clone --recursive https://github.com/purpleidea/mgmt/
cd $GOPATH/src/github.com/purpleidea/mgmt cd $GOPATH/src/github.com/purpleidea/mgmt/ || cd ~/go/src/github.com/purpleidea/mgmt/
``` ```
* Add $GOPATH/bin to $PATH * Add `$GOPATH/bin` to `$PATH`
``` ```shell
export PATH=$PATH:$GOPATH/bin export PATH=$PATH:$GOPATH/bin
``` ```
* Run `make deps` to install system and golang dependencies. Take a look at * Run `make deps` to install system and golang dependencies. Take a look at
`misc/make-deps.sh` for details. `misc/make-deps.sh` if you want to see the details of what it does.
* Run `make build` to get a freshly built `mgmt` binary.
### Running mgmt #### Building mgmt
* Run `time ./mgmt run --tmp-prefix lang --lang examples/lang/hello0.mcl` to try * Now run `make` to get a freshly built `mgmt` binary. If this succeeds, you can
out a very simple example! proceed to the [Running mgmt](#running-mgmt) section below!
### Installing a distro release
Installation of `mgmt` from distribution packages currently needs improvement.
They are not always up-to-date with git master and as such are not recommended.
At the moment we have:
* [COPR](https://copr.fedoraproject.org/coprs/purpleidea/mgmt/) (currently dead)
* [Arch](https://aur.archlinux.org/packages/mgmt/) (currently stale)
Please contribute more and help improve these! We'd especially like to see a
Debian package!
## Running mgmt
* Run `mgmt run --tmp-prefix lang examples/lang/hello0.mcl` to try out a very
simple example! If you built it from source, you'll need to use `./mgmt` from
the project directory.
* Look in that example file that you ran to see if you can figure out what it * Look in that example file that you ran to see if you can figure out what it
did! did! You can press `^C` to exit `mgmt`.
* Have fun hacking on our future technology and get involved to shape the * Have fun hacking on our future technology and get involved to shape the
project! project!
@@ -68,118 +111,3 @@ project!
Please look in the [examples/lang/](../examples/lang/) folder for some more Please look in the [examples/lang/](../examples/lang/) folder for some more
examples! examples!
## Vagrant
If you would like to avoid doing the above steps manually, we have prepared a
[Vagrant](https://www.vagrantup.com/) environment for your convenience. From the
project directory, run a `vagrant up`, and then a `vagrant status`. From there,
you can `vagrant ssh` into the `mgmt` machine. The MOTD will explain the rest.
## Using Docker
Alternatively, you can check out the [docker-guide](docs/docker-guide.md) in
order to develop or deploy using docker.
## Information about dependencies
Software projects have a few different kinds of dependencies. There are _build_
dependencies, _runtime_ dependencies, and additionally, a few extra dependencies
required for running the _test_ suite.
### Build
* `golang` 1.10 or higher (required, available in some distros and distributed
as a binary officially by [golang.org](https://golang.org/dl/))
### Runtime
A relatively modern GNU/Linux system should be able to run `mgmt` without any
problems. Since `mgmt` runs as a single statically compiled binary, all of the
library dependencies are included. It is expected, that certain advanced
resources require host specific facilities to work. These requirements are
listed below:
| Resource | Dependency | Version | Check version with |
|----------|-------------------|-----------------------------|-----------------------------------------------------------|
| augeas | augeas-devel | `augeas 1.6` or greater | `dnf info augeas-devel` or `apt-cache show libaugeas-dev` |
| file | inotify | `Linux 2.6.27` or greater | `uname -a` |
| hostname | systemd-hostnamed | `systemd 25` or greater | `systemctl --version` |
| nspawn | systemd-nspawn | `systemd ???` or greater | `systemctl --version` |
| pkg | packagekitd | `packagekit 1.x` or greater | `pkcon --version` |
| svc | systemd | `systemd ???` or greater | `systemctl --version` |
| virt | libvirt-devel | `libvirt 1.2.0` or greater | `dnf info libvirt-devel` or `apt-cache show libvirt-dev` |
| virt | libvirtd | `libvirt 1.2.0` or greater | `libvirtd --version` |
For building a visual representation of the graph, `graphviz` is required.
To build `mgmt` without augeas support please run:
`GOTAGS='noaugeas' make build`
To build `mgmt` without libvirt support please run:
`GOTAGS='novirt' make build`
To build `mgmt` without docker support please run:
`GOTAGS='nodocker' make build`
To build `mgmt` without augeas, libvirt or docker support please run:
`GOTAGS='noaugeas novirt nodocker' make build`
## Binary Package Installation
Installation of `mgmt` from distribution packages currently needs improvement.
They are not always up-to-date with git master and as such are not recommended.
At the moment we have:
* [COPR](https://copr.fedoraproject.org/coprs/purpleidea/mgmt/)
* [Arch](https://aur.archlinux.org/packages/mgmt/)
Please contribute more! We'd especially like to see a Debian package!
## OSX/macOS/Darwin development
Developing and running `mgmt` on macOS is currently not supported (but not
discouraged either). Meaning it might work but in the case it doesn't you would
have to provide your own patches to fix problems (the project maintainer and
community are glad to assist where needed).
There are currently some issues that make `mgmt` less suitable to run for provisioning
macOS. But as a client to provision remote servers it should run fine.
Since the primary supported systems are Linux and these are the environments
tested for it is wise to run these suites during macOS development as well. To
ease this Docker can be leveraged ([Docker for Mac](https://docs.docker.com/docker-for-mac/)).
Before running any of the commands below create the development Docker image:
```
docker/scripts/build-development
```
This image requires updating every time dependencies (`make-deps.sh`) change.
Then to run the test suite:
```
docker run --rm -ti \
-v $PWD:/go/src/github.com/purpleidea/mgmt/ \
-w /go/src/github.com/purpleidea/mgmt/ \
purpleidea/mgmt:development \
make test
```
For convenience this command is wrapped in `docker/scripts/exec-development`.
Basically any command can be executed this way. Because the repository source is
mounted into the Docker container invocation will be quick and allow rapid
testing, example:
```
docker/scripts/exec-development test/test-shell.sh load0.sh
```
Other examples:
```
docker/scripts/exec-development make build
docker/scripts/exec-development ./mgmt run --tmp-prefix lang --lang examples/lang/load0.mcl
```

86
docs/resource-guide.md
View File

@@ -96,8 +96,10 @@ Default() engine.Res
``` ```
This returns a populated resource struct as a `Res`. It shouldn't populate any This returns a populated resource struct as a `Res`. It shouldn't populate any
values which already have the correct default as the golang zero value. In values which already get a good default as the respective golang zero value. In
general it is preferable if the zero values make for the correct defaults. general it is preferable if the zero values make for the correct defaults.
(This is to say, resources are designed to behave safely and intuitively
when parameters take a zero value, whenever this is possible.)
#### Example #### Example
@@ -307,21 +309,18 @@ running.
The lifetime of most resources `Watch` method should be spent in an infinite The lifetime of most resources `Watch` method should be spent in an infinite
loop that is bounded by a `select` call. The `select` call is the point where loop that is bounded by a `select` call. The `select` call is the point where
our method hands back control to the engine (and the kernel) so that we can our method hands back control to the engine (and the kernel) so that we can
sleep until something of interest wakes us up. In this loop we must process sleep until something of interest wakes us up. In this loop we must wait until
events from the engine via the `<-obj.init.Events` channel, and receive events we get a shutdown event from the engine via the `<-obj.init.Done` channel, which
for our resource itself! closes when we'd like to shut everything down. At this point you should cleanup,
and let `Watch` close.
#### Events #### Events
If we receive an internal event from the `<-obj.init.Events` channel, we should If the `<-obj.init.Done` channel closes, we should shutdown our resource. When
read it with the `obj.init.Read` helper function. This function tells us if we When we want to send an event, we use the `Event` helper function. This
should shutdown our resource. It also handles pause functionality which blocks automatically marks the resource state as `dirty`. If you're unsure, it's not
our resource temporarily in this method. If this channel shuts down, then we harmful to send the event. This will ultimately cause `CheckApply` to run. This
should treat that as an exit signal. method can block if the resource is being paused.
When we want to send an event, we use the `Event` helper function. It is also
important to mark the resource state as `dirty` if we believe it might have
changed. We do this by calling the `obj.init.Dirty` function.
#### Startup #### Startup
@@ -330,8 +329,7 @@ to generate one event to notify the `mgmt` engine that we're now listening
successfully, so that it can run an initial `CheckApply` to ensure we're safely successfully, so that it can run an initial `CheckApply` to ensure we're safely
tracking a healthy state and that we didn't miss anything when `Watch` was down tracking a healthy state and that we didn't miss anything when `Watch` was down
or from before `mgmt` was running. You must do this by calling the or from before `mgmt` was running. You must do this by calling the
`obj.init.Running` method. If it returns an error, you must exit and return that `obj.init.Running` method.
error.
#### Converged #### Converged
@@ -358,41 +356,29 @@ func (obj *FooRes) Watch() error {
defer obj.whatever.CloseFoo() // shutdown our Foo defer obj.whatever.CloseFoo() // shutdown our Foo
// notify engine that we're running // notify engine that we're running
if err := obj.init.Running(); err != nil { obj.init.Running() // when started, notify engine that we're running
return err // exit if requested
}
var send = false // send event? var send = false // send event?
for { for {
select { select {
case event, ok := <-obj.init.Events:
if !ok {
// shutdown engine
// (it is okay if some `defer` code runs first)
return nil
}
if err := obj.init.Read(event); err != nil {
return err
}
// the actual events! // the actual events!
case event := <-obj.foo.Events: case event := <-obj.foo.Events:
if is_an_event { if is_an_event {
send = true send = true
obj.init.Dirty() // dirty
} }
// event errors // event errors
case err := <-obj.foo.Errors: case err := <-obj.foo.Errors:
return err // will cause a retry or permanent failure return err // will cause a retry or permanent failure
case <-obj.init.Done: // signal for shutdown request
return nil
} }
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event()
return err // exit if requested
}
} }
} }
} }
@@ -567,23 +553,10 @@ ready to detect changes.
Event sends an event notifying the engine of a possible state change. It is Event sends an event notifying the engine of a possible state change. It is
only called from within `Watch`. only called from within `Watch`.
### Events ### Done
Events is a channel that we must watch for messages from the engine. When it Done is a channel that closes when the engine wants us to shutdown. It is only
closes, this is a signal to shutdown. It is called from within `Watch`.
only called from within `Watch`.
### Read
Read processes messages that come in from the `Events` channel. It is a helper
method that knows how to handle the pause mechanism correctly. It is
only called from within `Watch`.
### Dirty
Dirty marks the resource state as dirty. This signals to the engine that
CheckApply will have some work to do in order to converge it. It is
only called from within `Watch`.
### Refresh ### Refresh
@@ -783,6 +756,23 @@ Feel free to use this pattern if you're convinced it's necessary. Alternatively,
if you think I got the `Res` API wrong and you have an improvement, please let if you think I got the `Res` API wrong and you have an improvement, please let
us know! us know!
### Why do resources have both a `Cmp` method and an `IFF` (on the UID) method?
The `Cmp()` methods are for determining if two resources are effectively the
same, which is used to make graph change delta's efficient. This is when we want
to change from the current running graph to a new graph, but preserve the common
vertices. Since we want to make this process efficient, we only update the parts
that are different, and leave everything else alone. This `Cmp()` method can
tell us if two resources are the same. In case it is not obvious, `cmp` is an
abbrev. for compare.
The `IFF()` method is part of the whole UID system, which is for discerning if a
resource meets the requirements another expects for an automatic edge. This is
because the automatic edge system assumes a unified UID pattern to test for
equality. In the future it might be helpful or sane to merge the two similar
comparison functions although for now they are separate because they are
actually answer different questions.
### What new resource primitives need writing? ### What new resource primitives need writing?
There are still many ideas for new resources that haven't been written yet. If There are still many ideas for new resources that haven't been written yet. If

17
docs/resources.md
View File

@@ -69,8 +69,8 @@ identified by a trailing slash in their path name. File have no such slash.
It has the following properties: It has the following properties:
* `path`: absolute file path (directories have a trailing slash here) * `path`: absolute file path (directories have a trailing slash here)
* `state`: either `exists`, `absent`, or undefined
* `content`: raw file content * `content`: raw file content
* `state`: either `exists` (the default value) or `absent`
* `mode`: octal unix file permissions * `mode`: octal unix file permissions
* `owner`: username or uid for the file owner * `owner`: username or uid for the file owner
* `group`: group name or gid for the file group * `group`: group name or gid for the file group
@@ -79,6 +79,16 @@ It has the following properties:
The path property specifies the file or directory that we are managing. The path property specifies the file or directory that we are managing.
### State
The state property describes the action we'd like to apply for the resource. The
possible values are: `exists` and `absent`. If you do not specify either of
these, it is undefined. Without specifying this value as `exists`, another param
cannot cause a file to get implicitly created. When specifying this value as
`absent`, you should not specify any other params that would normally change the
file. For example, if you specify `content` and this param is `absent`, then you
will get an engine validation error.
### Content ### Content
The content property is a string that specifies the desired file contents. The content property is a string that specifies the desired file contents.
@@ -88,11 +98,6 @@ The content property is a string that specifies the desired file contents.
The source property points to a source file or directory path that we wish to The source property points to a source file or directory path that we wish to
copy over and use as the desired contents for our resource. copy over and use as the desired contents for our resource.
### State
The state property describes the action we'd like to apply for the resource. The
possible values are: `exists` and `absent`.
### Recurse ### Recurse
The recurse property limits whether file resource operations should recurse into The recurse property limits whether file resource operations should recurse into

2
engine/autoedge.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

2
engine/autoedge_test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

2
engine/autogroup.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

26
engine/cmp.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -152,6 +152,18 @@ func ResCmp(r1, r2 Res) error {
} }
} }
// compare meta params for resources with reversible traits
r1v, ok1 := r1.(ReversibleRes)
r2v, ok2 := r2.(ReversibleRes)
if ok1 != ok2 {
return fmt.Errorf("reversible differs") // they must be different (optional)
}
if ok1 && ok2 {
if r1v.ReversibleMeta().Cmp(r2v.ReversibleMeta()) != nil {
return fmt.Errorf("reversible differs")
}
}
return nil return nil
} }
@@ -280,6 +292,18 @@ func AdaptCmp(r1, r2 CompatibleRes) error {
} }
} }
// compare meta params for resources with reversible traits
r1v, ok1 := r1.(ReversibleRes)
r2v, ok2 := r2.(ReversibleRes)
if ok1 != ok2 {
return fmt.Errorf("reversible differs") // they must be different (optional)
}
if ok1 && ok2 {
if r1v.ReversibleMeta().Cmp(r2v.ReversibleMeta()) != nil {
return fmt.Errorf("reversible differs")
}
}
return nil return nil
} }

14
engine/copy.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -20,7 +20,7 @@ package engine
import ( import (
"fmt" "fmt"
errwrap "github.com/pkg/errors" "github.com/purpleidea/mgmt/util/errwrap"
) )
// ResCopy copies a resource. This is the main entry point for copying a // ResCopy copies a resource. This is the main entry point for copying a
@@ -106,6 +106,16 @@ func ResCopy(r CopyableRes) (CopyableRes, error) {
} }
} }
// copy meta params for resources with reversible traits
if x, ok := r.(ReversibleRes); ok {
dst, ok := res.(ReversibleRes)
if !ok {
// programming error
panic("reversible interfaces are illogical")
}
dst.SetReversibleMeta(x.ReversibleMeta()) // no need to copy atm
}
return res, nil return res, nil
} }

2
engine/edge.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

9
engine/error.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -24,9 +24,6 @@ type Error string
func (e Error) Error() string { return string(e) } func (e Error) Error() string { return string(e) }
const ( const (
// ErrWatchExit represents an exit from the Watch loop via chan closure. // ErrClosed means we couldn't complete a task because we had closed.
ErrWatchExit = Error("watch exit") ErrClosed = Error("closed")
// ErrSignalExit represents an exit from the Watch loop via exit signal.
ErrSignalExit = Error("signal exit")
) )

83
engine/event/event.go
View File

@@ -1,83 +0,0 @@
// Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
// Package event provides some primitives that are used for message passing.
package event
//go:generate stringer -type=Kind -output=kind_stringer.go
// Kind represents the type of event being passed.
type Kind int
// The different event kinds are used in different contexts.
const (
KindNil Kind = iota
KindStart
KindPause
KindPoke
KindExit
)
// Pre-built messages so they can be used directly without having to use NewMsg.
// These are useful when we don't want a response via ACK().
var (
Start = &Msg{Kind: KindStart}
Pause = &Msg{Kind: KindPause} // probably unused b/c we want a resp
Poke = &Msg{Kind: KindPoke}
Exit = &Msg{Kind: KindExit}
)
// Msg is an event primitive that represents a kind of event, and optionally a
// request for an ACK.
type Msg struct {
Kind Kind
resp chan struct{}
}
// NewMsg builds a new message struct. It will want an ACK. If you don't want an
// ACK then use the pre-built messages in the package variable globals.
func NewMsg(kind Kind) *Msg {
return &Msg{
Kind: kind,
resp: make(chan struct{}),
}
}
// CanACK determines if an ACK is possible for this message. It does not say
// whether one has already been sent or not.
func (obj *Msg) CanACK() bool {
return obj.resp != nil
}
// ACK acknowledges the event. It must not be called more than once for the same
// event. It unblocks the past and future calls of Wait for this event.
func (obj *Msg) ACK() {
close(obj.resp)
}
// Wait on ACK for this event. It doesn't matter if this runs before or after
// the ACK. It will unblock either way.
// TODO: consider adding a context if it's ever useful.
func (obj *Msg) Wait() error {
select {
//case <-ctx.Done():
// return ctx.Err()
case <-obj.resp:
return nil
}
}

2
engine/fs.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

259
engine/graph/actions.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -24,11 +24,9 @@ import (
"time" "time"
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/event"
"github.com/purpleidea/mgmt/pgraph" "github.com/purpleidea/mgmt/pgraph"
"github.com/purpleidea/mgmt/util/errwrap"
//multierr "github.com/hashicorp/go-multierror"
errwrap "github.com/pkg/errors"
"golang.org/x/time/rate" "golang.org/x/time/rate"
) )
@@ -67,26 +65,24 @@ func (obj *Engine) Process(vertex pgraph.Vertex) error {
return fmt.Errorf("vertex is not a Res") return fmt.Errorf("vertex is not a Res")
} }
// Engine Guarantee: Do not allow CheckApply to run while we are paused.
// This makes the resource able to know that synchronous channel sending
// to the main loop select in Watch from within CheckApply, will succeed
// without blocking because the resource went into a paused state. If we
// are using the Poll metaparam, then Watch will (of course) not be run.
// FIXME: should this lock be here, or wrapped right around CheckApply ?
obj.state[vertex].eventsLock.Lock() // this lock is taken within Event()
defer obj.state[vertex].eventsLock.Unlock()
// backpoke! (can be async) // backpoke! (can be async)
if vs := obj.BadTimestamps(vertex); len(vs) > 0 { if vs := obj.BadTimestamps(vertex); len(vs) > 0 {
// back poke in parallel (sync b/c of waitgroup) // back poke in parallel (sync b/c of waitgroup)
wg := &sync.WaitGroup{}
for _, v := range obj.graph.IncomingGraphVertices(vertex) { for _, v := range obj.graph.IncomingGraphVertices(vertex) {
if !pgraph.VertexContains(v, vs) { // only poke what's needed if !pgraph.VertexContains(v, vs) { // only poke what's needed
continue continue
} }
go obj.state[v].Poke() // async // doesn't really need to be in parallel, but we can...
wg.Add(1)
go func(vv pgraph.Vertex) {
defer wg.Done()
obj.state[vv].Poke() // async
}(v)
} }
wg.Wait()
return nil // can't continue until timestamp is in sequence return nil // can't continue until timestamp is in sequence
} }
@@ -244,14 +240,22 @@ func (obj *Engine) Process(vertex pgraph.Vertex) error {
// Worker is the common run frontend of the vertex. It handles all of the retry // Worker is the common run frontend of the vertex. It handles all of the retry
// and retry delay common code, and ultimately returns the final status of this // and retry delay common code, and ultimately returns the final status of this
// vertex execution. // vertex execution. This function cannot be "re-run" for the same vertex. The
// retry mechanism stuff happens inside of this. To actually "re-run" you need
// to remove the vertex and build a new one. The engine guarantees that we do
// not allow CheckApply to run while we are paused. That is enforced here.
func (obj *Engine) Worker(vertex pgraph.Vertex) error { func (obj *Engine) Worker(vertex pgraph.Vertex) error {
res, isRes := vertex.(engine.Res) res, isRes := vertex.(engine.Res)
if !isRes { if !isRes {
return fmt.Errorf("vertex is not a resource") return fmt.Errorf("vertex is not a resource")
} }
defer close(obj.state[vertex].stopped) // done signal // bonus safety check
if res.MetaParams().Burst == 0 && !(res.MetaParams().Limit == rate.Inf) { // blocked
return fmt.Errorf("permanently limited (rate != Inf, burst = 0)")
}
//defer close(obj.state[vertex].stopped) // done signal
obj.state[vertex].cuid = obj.Converger.Register() obj.state[vertex].cuid = obj.Converger.Register()
obj.state[vertex].tuid = obj.Converger.Register() obj.state[vertex].tuid = obj.Converger.Register()
@@ -265,7 +269,28 @@ func (obj *Engine) Worker(vertex pgraph.Vertex) error {
obj.state[vertex].wg.Add(1) obj.state[vertex].wg.Add(1)
go func() { go func() {
defer obj.state[vertex].wg.Done() defer obj.state[vertex].wg.Done()
defer close(obj.state[vertex].outputChan) // we close this on behalf of res defer close(obj.state[vertex].eventsChan) // we close this on behalf of res
// This is a close reverse-multiplexer. If any of the channels
// close, then it will cause the doneChan to close. That way,
// multiple different folks can send a close signal, without
// every worrying about duplicate channel close panics.
obj.state[vertex].wg.Add(1)
go func() {
defer obj.state[vertex].wg.Done()
// reverse-multiplexer: any close, causes *the* close!
select {
case <-obj.state[vertex].processDone:
case <-obj.state[vertex].watchDone:
case <-obj.state[vertex].limitDone:
case <-obj.state[vertex].removeDone:
case <-obj.state[vertex].eventsDone:
}
// the main "done" signal gets activated here!
close(obj.state[vertex].doneChan)
}()
var err error var err error
var retry = res.MetaParams().Retry // lookup the retry value var retry = res.MetaParams().Retry // lookup the retry value
@@ -283,14 +308,9 @@ func (obj *Engine) Worker(vertex pgraph.Vertex) error {
case <-timer.C: // the wait is over case <-timer.C: // the wait is over
return errDelayExpired // special return errDelayExpired // special
case event, ok := <-obj.state[vertex].init.Events: case <-obj.state[vertex].init.Done:
if !ok {
return nil return nil
} }
if err := obj.state[vertex].init.Read(event); err != nil {
return err
}
}
} }
}() }()
if err == errDelayExpired { if err == errDelayExpired {
@@ -308,68 +328,121 @@ func (obj *Engine) Worker(vertex pgraph.Vertex) error {
obj.Logf("Watch(%s): Exited(%+v)", vertex, err) obj.Logf("Watch(%s): Exited(%+v)", vertex, err)
obj.state[vertex].cuid.StopTimer() // clean up nicely obj.state[vertex].cuid.StopTimer() // clean up nicely
} }
if err == nil || err == engine.ErrWatchExit || err == engine.ErrSignalExit { if err == nil { // || err == engine.ErrClosed
return // exited cleanly, we're done return // exited cleanly, we're done
} }
// we've got an error... // we've got an error...
delay = res.MetaParams().Delay delay = res.MetaParams().Delay
if retry < 0 { // infinite retries if retry < 0 { // infinite retries
obj.state[vertex].reset()
continue continue
} }
if retry > 0 { // don't decrement past 0 if retry > 0 { // don't decrement past 0
retry-- retry--
obj.state[vertex].init.Logf("retrying Watch after %.4f seconds (%d left)", float64(delay)/1000, retry) obj.state[vertex].init.Logf("retrying Watch after %.4f seconds (%d left)", float64(delay)/1000, retry)
obj.state[vertex].reset()
continue continue
} }
//if retry == 0 { // optional //if retry == 0 { // optional
// err = errwrap.Wrapf(err, "permanent watch error") // err = errwrap.Wrapf(err, "permanent watch error")
//} //}
break // break out of this and send the error break // break out of this and send the error
} } // for retry loop
// this section sends an error... // this section sends an error...
// If the CheckApply loop exits and THEN the Watch fails with an // If the CheckApply loop exits and THEN the Watch fails with an
// error, then we'd be stuck here if exit signal didn't unblock! // error, then we'd be stuck here if exit signal didn't unblock!
select { select {
case obj.state[vertex].outputChan <- errwrap.Wrapf(err, "watch failed"): case obj.state[vertex].eventsChan <- errwrap.Wrapf(err, "watch failed"):
// send // send
case <-obj.state[vertex].exit.Signal():
// pass
} }
}() }()
// bonus safety check // If this exits cleanly, we must unblock the reverse-multiplexer.
if res.MetaParams().Burst == 0 && !(res.MetaParams().Limit == rate.Inf) { // blocked // I think this additional close is unnecessary, but it's not harmful.
return fmt.Errorf("permanently limited (rate != Inf, burst = 0)") defer close(obj.state[vertex].eventsDone) // causes doneChan to close
} limiter := rate.NewLimiter(res.MetaParams().Limit, res.MetaParams().Burst)
var limiter = rate.NewLimiter(res.MetaParams().Limit, res.MetaParams().Burst) var reserv *rate.Reservation
// It is important that we shutdown the Watch loop if this exits. var reterr error
// Example, if Process errors permanently, we should ask Watch to exit. var failed bool // has Process permanently failed?
defer obj.state[vertex].Event(event.Exit) // signal an exit Loop:
for { for { // process loop
select { select {
case err, ok := <-obj.state[vertex].outputChan: // read from watch channel case err, ok := <-obj.state[vertex].eventsChan: // read from watch channel
if !ok { if !ok {
return nil return reterr // we only return when chan closes
} }
// If the Watch method exits with an error, then this
// channel will get that error propagated to it, which
// we then save so we can return it to the caller of us.
if err != nil { if err != nil {
return err // permanent failure failed = true
close(obj.state[vertex].watchDone) // causes doneChan to close
reterr = errwrap.Append(reterr, err) // permanent failure
continue
}
if obj.Debug {
obj.Logf("event received")
}
reserv = limiter.ReserveN(time.Now(), 1) // one event
// reserv.OK() seems to always be true here!
case _, ok := <-obj.state[vertex].pokeChan: // read from buffered poke channel
if !ok { // we never close it
panic("unexpected close of poke channel")
}
if obj.Debug {
obj.Logf("poke received")
}
reserv = nil // we didn't receive a real event here...
}
if failed { // don't Process anymore if we've already failed...
continue Loop
} }
// safe to go run the process... // drop redundant pokes
case <-obj.state[vertex].exit.Signal(): // TODO: is this needed? for len(obj.state[vertex].pokeChan) > 0 {
return nil select {
case <-obj.state[vertex].pokeChan:
default:
// race, someone else read one!
}
} }
now := time.Now() // pause if one was requested...
r := limiter.ReserveN(now, 1) // one event select {
// r.OK() seems to always be true here! case <-obj.state[vertex].pauseSignal: // channel closes
d := r.DelayFrom(now) // NOTE: If we allowed a doneChan below to let us out
if d > 0 { // delay // of the resumeSignal wait, then we could loop around
// and run this again, causing a panic. Instead of this
// being made safe with a sync.Once, we instead run a
// Resume() call inside of the vertexRemoveFn function,
// which should unblock it when we're going to need to.
obj.state[vertex].pausedAck.Ack() // send ack
// we are paused now, and waiting for resume or exit...
select {
case <-obj.state[vertex].resumeSignal: // channel closes
// resumed!
// pass through to allow a Process to try to run
// TODO: consider adding this fast pause here...
//if obj.fastPause {
// obj.Logf("fast pausing on resume")
// continue
//}
}
default:
// no pause requested, keep going...
}
if failed { // don't Process anymore if we've already failed...
continue Loop
}
// limit delay
d := time.Duration(0)
if reserv != nil {
d = reserv.DelayFrom(time.Now())
}
if reserv != nil && d > 0 { // delay
obj.state[vertex].init.Logf("limited (rate: %v/sec, burst: %d, next: %v)", res.MetaParams().Limit, res.MetaParams().Burst, d) obj.state[vertex].init.Logf("limited (rate: %v/sec, burst: %d, next: %v)", res.MetaParams().Limit, res.MetaParams().Burst, d)
var count int
timer := time.NewTimer(time.Duration(d) * time.Millisecond) timer := time.NewTimer(time.Duration(d) * time.Millisecond)
LimitWait: LimitWait:
for { for {
@@ -378,35 +451,38 @@ func (obj *Engine) Worker(vertex pgraph.Vertex) error {
break LimitWait break LimitWait
// consume other events while we're waiting... // consume other events while we're waiting...
case e, ok := <-obj.state[vertex].outputChan: // read from watch channel case e, ok := <-obj.state[vertex].eventsChan: // read from watch channel
if !ok { if !ok {
// FIXME: is this logic correct? return reterr // we only return when chan closes
if count == 0 {
return nil
}
// loop, because we have
// the previous event to
// run process on first!
continue
} }
if e != nil { if e != nil {
return e // permanent failure failed = true
close(obj.state[vertex].limitDone) // causes doneChan to close
reterr = errwrap.Append(reterr, e) // permanent failure
break LimitWait
} }
count++ // count the events... if obj.Debug {
obj.Logf("event received in limit")
}
// TODO: does this get added in properly?
limiter.ReserveN(time.Now(), 1) // one event limiter.ReserveN(time.Now(), 1) // one event
} }
} }
timer.Stop() // it's nice to cleanup timer.Stop() // it's nice to cleanup
obj.state[vertex].init.Logf("rate limiting expired!") obj.state[vertex].init.Logf("rate limiting expired!")
} }
if failed { // don't Process anymore if we've already failed...
continue Loop
}
// end of limit delay
// retry...
var err error var err error
var retry = res.MetaParams().Retry // lookup the retry value var retry = res.MetaParams().Retry // lookup the retry value
var delay uint64 var delay uint64
Loop: RetryLoop:
for { // retry loop for { // retry loop
if delay > 0 { if delay > 0 {
var count int
timer := time.NewTimer(time.Duration(delay) * time.Millisecond) timer := time.NewTimer(time.Duration(delay) * time.Millisecond)
RetryWait: RetryWait:
for { for {
@@ -415,22 +491,20 @@ func (obj *Engine) Worker(vertex pgraph.Vertex) error {
break RetryWait break RetryWait
// consume other events while we're waiting... // consume other events while we're waiting...
case e, ok := <-obj.state[vertex].outputChan: // read from watch channel case e, ok := <-obj.state[vertex].eventsChan: // read from watch channel
if !ok { if !ok {
// FIXME: is this logic correct? return reterr // we only return when chan closes
if count == 0 {
// last process error
return err
}
// loop, because we have
// the previous event to
// run process on first!
continue
} }
if e != nil { if e != nil {
return e // permanent failure failed = true
close(obj.state[vertex].limitDone) // causes doneChan to close
reterr = errwrap.Append(reterr, e) // permanent failure
break RetryWait
} }
count++ // count the events... if obj.Debug {
obj.Logf("event received in retry")
}
// TODO: does this get added in properly?
limiter.ReserveN(time.Now(), 1) // one event limiter.ReserveN(time.Now(), 1) // one event
} }
} }
@@ -438,6 +512,9 @@ func (obj *Engine) Worker(vertex pgraph.Vertex) error {
delay = 0 // reset delay = 0 // reset
obj.state[vertex].init.Logf("the CheckApply delay expired!") obj.state[vertex].init.Logf("the CheckApply delay expired!")
} }
if failed { // don't Process anymore if we've already failed...
continue Loop
}
if obj.Debug { if obj.Debug {
obj.Logf("Process(%s)", vertex) obj.Logf("Process(%s)", vertex)
@@ -447,7 +524,7 @@ func (obj *Engine) Worker(vertex pgraph.Vertex) error {
obj.Logf("Process(%s): Return(%+v)", vertex, err) obj.Logf("Process(%s): Return(%+v)", vertex, err)
} }
if err == nil { if err == nil {
break Loop break RetryLoop
} }
// we've got an error... // we've got an error...
delay = res.MetaParams().Delay delay = res.MetaParams().Delay
@@ -464,15 +541,23 @@ func (obj *Engine) Worker(vertex pgraph.Vertex) error {
// err = errwrap.Wrapf(err, "permanent process error") // err = errwrap.Wrapf(err, "permanent process error")
//} //}
// If this exits, defer calls: obj.Event(event.Exit), // It is important that we shutdown the Watch loop if
// which will cause the Watch loop to shutdown. Also, // this dies. If Process fails permanently, we ask it
// if the Watch loop shuts down, that will cause this // to exit right here... (It happens when we loop...)
// Process loop to shut down. Also the graph sync can failed = true
// run an: obj.Event(event.Exit) which causes this to close(obj.state[vertex].processDone) // causes doneChan to close
// shutdown as well. Lastly, it is possible that more reterr = errwrap.Append(reterr, err) // permanent failure
// that one of these scenarios happens simultaneously. continue
return err
} } // retry loop
}
// When this Process loop exits, it's because something has
// caused Watch() to shutdown (even if it's our permanent
// failure from Process), which caused this channel to close.
// On or more exit signals are possible, and more than one can
// happen simultaneously.
} // process loop
//return nil // unreachable //return nil // unreachable
} }

2
engine/graph/autoedge.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

11
engine/graph/autoedge/autoedge.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -22,9 +22,7 @@ import (
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/pgraph" "github.com/purpleidea/mgmt/pgraph"
"github.com/purpleidea/mgmt/util/errwrap"
multierr "github.com/hashicorp/go-multierror"
errwrap "github.com/pkg/errors"
) )
// AutoEdge adds the automatic edges to the graph. // AutoEdge adds the automatic edges to the graph.
@@ -49,7 +47,7 @@ func AutoEdge(graph *pgraph.Graph, debug bool, logf func(format string, v ...int
for _, res := range sorted { // for each vertexes autoedges for _, res := range sorted { // for each vertexes autoedges
autoEdgeObj, e := res.AutoEdges() autoEdgeObj, e := res.AutoEdges()
if e != nil { if e != nil {
err = multierr.Append(err, e) // collect all errors err = errwrap.Append(err, e) // collect all errors
continue continue
} }
if autoEdgeObj == nil { if autoEdgeObj == nil {
@@ -91,6 +89,9 @@ func AutoEdge(graph *pgraph.Graph, debug bool, logf func(format string, v ...int
} }
} }
} }
// It would be great to ensure we didn't add any loops here, but instead
// of checking now, we'll move the check into the main loop.
return nil return nil
} }

5
engine/graph/autogroup.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -24,8 +24,7 @@ import (
"github.com/purpleidea/mgmt/engine/graph/autogroup" "github.com/purpleidea/mgmt/engine/graph/autogroup"
"github.com/purpleidea/mgmt/pgraph" "github.com/purpleidea/mgmt/pgraph"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
) )
// AutoGroup runs the auto grouping on the loaded graph. // AutoGroup runs the auto grouping on the loaded graph.

8
engine/graph/autogroup/autogroup.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -22,8 +22,7 @@ import (
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/pgraph" "github.com/purpleidea/mgmt/pgraph"
"github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
) )
// AutoGroup is the mechanical auto group "runner" that runs the interface spec. // AutoGroup is the mechanical auto group "runner" that runs the interface spec.
@@ -67,5 +66,8 @@ func AutoGroup(ag engine.AutoGrouper, g *pgraph.Graph, debug bool, logf func(for
} }
} }
// It would be great to ensure we didn't add any loops here, but instead
// of checking now, we'll move the check into the main loop.
return nil return nil
} }

5
engine/graph/autogroup/autogroup_test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -31,8 +31,7 @@ import (
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
"github.com/purpleidea/mgmt/pgraph" "github.com/purpleidea/mgmt/pgraph"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
) )
func init() { func init() {

2
engine/graph/autogroup/base.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

5
engine/graph/autogroup/nonreachability.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -19,8 +19,7 @@ package autogroup
import ( import (
"github.com/purpleidea/mgmt/pgraph" "github.com/purpleidea/mgmt/pgraph"
"github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
) )
// NonReachabilityGrouper is the most straight-forward algorithm for grouping. // NonReachabilityGrouper is the most straight-forward algorithm for grouping.

19
engine/graph/autogroup/util.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -18,9 +18,11 @@
package autogroup package autogroup
import ( import (
"github.com/purpleidea/mgmt/pgraph" "fmt"
errwrap "github.com/pkg/errors" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/pgraph"
"github.com/purpleidea/mgmt/util/errwrap"
) )
// VertexMerge merges v2 into v1 by reattaching the edges where appropriate, // VertexMerge merges v2 into v1 by reattaching the edges where appropriate,
@@ -113,8 +115,17 @@ func VertexMerge(g *pgraph.Graph, v1, v2 pgraph.Vertex, vertexMergeFn func(pgrap
// note: This branch isn't used if the vertexMergeFn // note: This branch isn't used if the vertexMergeFn
// decides to just merge logically on its own instead // decides to just merge logically on its own instead
// of actually returning something that we then merge. // of actually returning something that we then merge.
v1 = v // TODO: ineffassign? v1 = v // XXX: ineffassign?
//*v1 = *v //*v1 = *v
// Ensure that everything still validates. (For safety!)
r, ok := v1.(engine.Res) // TODO: v ?
if !ok {
return fmt.Errorf("not a Res")
}
if err := engine.Validate(r); err != nil {
return errwrap.Wrapf(err, "the Res did not Validate")
}
} }
} }
g.DeleteVertex(v2) // remove grouped vertex g.DeleteVertex(v2) // remove grouped vertex

202
engine/graph/engine.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -25,12 +25,16 @@ import (
"github.com/purpleidea/mgmt/converger" "github.com/purpleidea/mgmt/converger"
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/event" engineUtil "github.com/purpleidea/mgmt/engine/util"
"github.com/purpleidea/mgmt/pgraph" "github.com/purpleidea/mgmt/pgraph"
"github.com/purpleidea/mgmt/util/errwrap"
"github.com/purpleidea/mgmt/util/semaphore" "github.com/purpleidea/mgmt/util/semaphore"
)
multierr "github.com/hashicorp/go-multierror" const (
errwrap "github.com/pkg/errors" // StateDir is the name of the sub directory where all the local
// resource state is stored.
StateDir = "state"
) )
// Engine encapsulates a generic graph and manages its operations. // Engine encapsulates a generic graph and manages its operations.
@@ -42,7 +46,7 @@ type Engine struct {
// Prefix is a unique directory prefix which can be used. It should be // Prefix is a unique directory prefix which can be used. It should be
// created if needed. // created if needed.
Prefix string Prefix string
Converger converger.Converger Converger *converger.Coordinator
Debug bool Debug bool
Logf func(format string, v ...interface{}) Logf func(format string, v ...interface{})
@@ -50,13 +54,15 @@ type Engine struct {
graph *pgraph.Graph graph *pgraph.Graph
nextGraph *pgraph.Graph nextGraph *pgraph.Graph
state map[pgraph.Vertex]*State state map[pgraph.Vertex]*State
waits map[pgraph.Vertex]*sync.WaitGroup waits map[pgraph.Vertex]*sync.WaitGroup // wg for the Worker func
wlock *sync.Mutex // lock around waits map
slock *sync.Mutex // semaphore lock slock *sync.Mutex // semaphore lock
semas map[string]*semaphore.Semaphore semas map[string]*semaphore.Semaphore
wg *sync.WaitGroup wg *sync.WaitGroup // wg for the whole engine (only used for close)
paused bool // are we paused?
fastPause bool fastPause bool
} }
@@ -64,6 +70,13 @@ type Engine struct {
// If the struct does not validate, or it cannot initialize, then this errors. // If the struct does not validate, or it cannot initialize, then this errors.
// Initially it will contain an empty graph. // Initially it will contain an empty graph.
func (obj *Engine) Init() error { func (obj *Engine) Init() error {
if obj.Program == "" {
return fmt.Errorf("the Program is empty")
}
if obj.Hostname == "" {
return fmt.Errorf("the Hostname is empty")
}
var err error var err error
if obj.graph, err = pgraph.NewGraph("graph"); err != nil { if obj.graph, err = pgraph.NewGraph("graph"); err != nil {
return err return err
@@ -78,12 +91,15 @@ func (obj *Engine) Init() error {
obj.state = make(map[pgraph.Vertex]*State) obj.state = make(map[pgraph.Vertex]*State)
obj.waits = make(map[pgraph.Vertex]*sync.WaitGroup) obj.waits = make(map[pgraph.Vertex]*sync.WaitGroup)
obj.wlock = &sync.Mutex{}
obj.slock = &sync.Mutex{} obj.slock = &sync.Mutex{}
obj.semas = make(map[string]*semaphore.Semaphore) obj.semas = make(map[string]*semaphore.Semaphore)
obj.wg = &sync.WaitGroup{} obj.wg = &sync.WaitGroup{}
obj.paused = true // start off true, so we can Resume after first Commit
return nil return nil
} }
@@ -137,6 +153,7 @@ func (obj *Engine) Apply(fn func(*pgraph.Graph) error) error {
func (obj *Engine) Commit() error { func (obj *Engine) Commit() error {
// TODO: Does this hurt performance or graph changes ? // TODO: Does this hurt performance or graph changes ?
start := []func() error{} // functions to run after graphsync to start...
vertexAddFn := func(vertex pgraph.Vertex) error { vertexAddFn := func(vertex pgraph.Vertex) error {
// some of these validation steps happen before this Commit step // some of these validation steps happen before this Commit step
// in Validate() to avoid erroring here. These are redundant. // in Validate() to avoid erroring here. These are redundant.
@@ -164,9 +181,9 @@ func (obj *Engine) Commit() error {
return errwrap.Wrapf(err, "the Res did not Validate") return errwrap.Wrapf(err, "the Res did not Validate")
} }
// FIXME: is res.Name() sufficiently unique to use as a UID here? pathUID := engineUtil.ResPathUID(res)
pathUID := fmt.Sprintf("%s-%s", res.Kind(), res.Name()) statePrefix := fmt.Sprintf("%s/", path.Join(obj.statePrefix(), pathUID))
statePrefix := fmt.Sprintf("%s/", path.Join(obj.Prefix, "state", pathUID))
// don't create this unless it *will* be used // don't create this unless it *will* be used
//if err := os.MkdirAll(statePrefix, 0770); err != nil { //if err := os.MkdirAll(statePrefix, 0770); err != nil {
// return errwrap.Wrapf(err, "can't create state prefix") // return errwrap.Wrapf(err, "can't create state prefix")
@@ -192,12 +209,45 @@ func (obj *Engine) Commit() error {
if err := obj.state[vertex].Init(); err != nil { if err := obj.state[vertex].Init(); err != nil {
return errwrap.Wrapf(err, "the Res did not Init") return errwrap.Wrapf(err, "the Res did not Init")
} }
fn := func() error {
// start the Worker
obj.wg.Add(1)
obj.wlock.Lock()
obj.waits[vertex].Add(1)
obj.wlock.Unlock()
go func(v pgraph.Vertex) {
defer obj.wg.Done()
defer func() {
// we need this lock, because this go
// routine could run when the next fn
// function above here is running...
obj.wlock.Lock()
obj.waits[v].Done()
obj.wlock.Unlock()
}()
obj.Logf("Worker(%s)", v)
// contains the Watch and CheckApply loops
err := obj.Worker(v)
obj.Logf("Worker(%s): Exited(%+v)", v, err)
obj.state[v].workerErr = err // store the error
// If the Rewatch metaparam is true, then this will get
// restarted if we do a graph cmp swap. This is why the
// graph cmp function runs the removes before the adds.
// XXX: This should feed into an $error var in the lang.
}(vertex)
return nil return nil
} }
start = append(start, fn) // do this at the end, if it's needed
return nil
}
free := []func() error{} // functions to run after graphsync to reset... free := []func() error{} // functions to run after graphsync to reset...
vertexRemoveFn := func(vertex pgraph.Vertex) error { vertexRemoveFn := func(vertex pgraph.Vertex) error {
// wait for exit before starting new graph! // wait for exit before starting new graph!
obj.state[vertex].Event(event.Exit) // signal an exit close(obj.state[vertex].removeDone) // causes doneChan to close
obj.state[vertex].Resume() // unblock from resume
obj.waits[vertex].Wait() // sync obj.waits[vertex].Wait() // sync
// close the state and resource // close the state and resource
@@ -216,15 +266,58 @@ func (obj *Engine) Commit() error {
return nil return nil
} }
// add the Worker swap (reload) on error decision into this vertexCmpFn
vertexCmpFn := func(v1, v2 pgraph.Vertex) (bool, error) {
r1, ok1 := v1.(engine.Res)
r2, ok2 := v2.(engine.Res)
if !ok1 || !ok2 { // should not happen, previously validated
return false, fmt.Errorf("not a Res")
}
m1 := r1.MetaParams()
m2 := r2.MetaParams()
swap1, swap2 := true, true // assume default of true
if m1 != nil {
swap1 = m1.Rewatch
}
if m2 != nil {
swap2 = m2.Rewatch
}
s1, ok1 := obj.state[v1]
s2, ok2 := obj.state[v2]
x1, x2 := false, false
if ok1 {
x1 = s1.workerErr != nil && swap1
}
if ok2 {
x2 = s2.workerErr != nil && swap2
}
if x1 || x2 {
// We swap, even if they're the same, so that we reload!
// This causes an add and remove of the "same" vertex...
return false, nil
}
return engine.VertexCmpFn(v1, v2) // do the normal cmp otherwise
}
// If GraphSync succeeds, it updates the receiver graph accordingly... // If GraphSync succeeds, it updates the receiver graph accordingly...
// Running the shutdown in vertexRemoveFn does not need to happen in a // Running the shutdown in vertexRemoveFn does not need to happen in a
// topologically sorted order because it already paused in that order. // topologically sorted order because it already paused in that order.
obj.Logf("graph sync...") obj.Logf("graph sync...")
if err := obj.graph.GraphSync(obj.nextGraph, engine.VertexCmpFn, vertexAddFn, vertexRemoveFn, engine.EdgeCmpFn); err != nil { if err := obj.graph.GraphSync(obj.nextGraph, vertexCmpFn, vertexAddFn, vertexRemoveFn, engine.EdgeCmpFn); err != nil {
return errwrap.Wrapf(err, "error running graph sync") return errwrap.Wrapf(err, "error running graph sync")
} }
// we run these afterwards, so that the state structs (that might get // We run these afterwards, so that we don't unnecessarily start anyone
// referenced) aren't destroyed while someone might poke or use one. // if GraphSync failed in some way. Otherwise we'd have to do clean up!
for _, fn := range start {
if err := fn(); err != nil {
return errwrap.Wrapf(err, "error running start fn")
}
}
// We run these afterwards, so that the state structs (that might get
// referenced) are not destroyed while someone might poke or use one.
for _, fn := range free { for _, fn := range free {
if err := fn(); err != nil { if err := fn(); err != nil {
return errwrap.Wrapf(err, "error running free fn") return errwrap.Wrapf(err, "error running free fn")
@@ -248,50 +341,28 @@ func (obj *Engine) Commit() error {
return nil return nil
} }
// Start runs the currently active graph. It also un-pauses the graph if it was // Resume runs the currently active graph. It also un-pauses the graph if it was
// paused. // paused. Very little that is interesting should happen here. It all happens in
func (obj *Engine) Start() error { // the Commit method. After Commit, new things are already started, but we still
// need to Resume any pre-existing resources.
func (obj *Engine) Resume() error {
if !obj.paused {
return fmt.Errorf("already resumed")
}
topoSort, err := obj.graph.TopologicalSort() topoSort, err := obj.graph.TopologicalSort()
if err != nil { if err != nil {
return err return err
} }
indegree := obj.graph.InDegree() // compute all of the indegree's //indegree := obj.graph.InDegree() // compute all of the indegree's
reversed := pgraph.Reverse(topoSort) reversed := pgraph.Reverse(topoSort)
for _, vertex := range reversed { for _, vertex := range reversed {
state := obj.state[vertex] //obj.state[vertex].starter = (indegree[vertex] == 0)
state.starter = (indegree[vertex] == 0) obj.state[vertex].Resume() // doesn't error
var unpause = true // assume true
if !state.working { // if not running...
state.working = true
unpause = false // doesn't need unpausing if starting
obj.wg.Add(1)
obj.waits[vertex].Add(1)
go func(v pgraph.Vertex) {
defer obj.wg.Done()
defer obj.waits[vertex].Done()
defer func() {
obj.state[v].working = false
}()
obj.Logf("Worker(%s)", v)
// contains the Watch and CheckApply loops
err := obj.Worker(v)
obj.Logf("Worker(%s): Exited(%+v)", v, err)
}(vertex)
}
select {
case <-state.started:
case <-state.stopped: // we failed on Watch start
}
if unpause { // unpause (if needed)
obj.state[vertex].Event(event.Start)
}
} }
// we wait for everyone to start before exiting! // we wait for everyone to start before exiting!
obj.paused = false
return nil return nil
} }
@@ -302,40 +373,46 @@ func (obj *Engine) Start() error {
// This is because once you've started a fast pause, some dependencies might // This is because once you've started a fast pause, some dependencies might
// have been skipped when fast pausing, and future resources might have missed a // have been skipped when fast pausing, and future resources might have missed a
// poke. In general this is only called when you're trying to hurry up the exit. // poke. In general this is only called when you're trying to hurry up the exit.
// XXX: Not implemented
func (obj *Engine) SetFastPause() { func (obj *Engine) SetFastPause() {
obj.fastPause = true obj.fastPause = true
} }
// Pause the active, running graph. At the moment this cannot error. // Pause the active, running graph.
func (obj *Engine) Pause(fastPause bool) { func (obj *Engine) Pause(fastPause bool) error {
if obj.paused {
return fmt.Errorf("already paused")
}
obj.fastPause = fastPause obj.fastPause = fastPause
topoSort, _ := obj.graph.TopologicalSort() topoSort, _ := obj.graph.TopologicalSort()
for _, vertex := range topoSort { // squeeze out the events... for _, vertex := range topoSort { // squeeze out the events...
// The Event is sent to an unbuffered channel, so this event is // The Event is sent to an unbuffered channel, so this event is
// synchronous, and as a result it blocks until it is received. // synchronous, and as a result it blocks until it is received.
obj.state[vertex].Event(event.Pause) if err := obj.state[vertex].Pause(); err != nil && err != engine.ErrClosed {
return err
} }
}
obj.paused = true
// we are now completely paused... // we are now completely paused...
obj.fastPause = false // reset obj.fastPause = false // reset
return nil
} }
// Close triggers a shutdown. Engine must be already paused before this is run. // Close triggers a shutdown. Engine must be already paused before this is run.
func (obj *Engine) Close() error { func (obj *Engine) Close() error {
var reterr error emptyGraph, reterr := pgraph.NewGraph("empty")
emptyGraph, err := pgraph.NewGraph("empty")
if err != nil {
reterr = multierr.Append(reterr, err) // list of errors
}
// this is a graph switch (graph sync) that switches to an empty graph! // this is a graph switch (graph sync) that switches to an empty graph!
if err := obj.Load(emptyGraph); err != nil { // copy in empty graph if err := obj.Load(emptyGraph); err != nil { // copy in empty graph
reterr = multierr.Append(reterr, err) reterr = errwrap.Append(reterr, err)
} }
// FIXME: Do we want to run commit if Load failed? Does this even work?
// the commit will cause the graph sync to shut things down cleverly... // the commit will cause the graph sync to shut things down cleverly...
if err := obj.Commit(); err != nil { if err := obj.Commit(); err != nil {
reterr = multierr.Append(reterr, err) reterr = errwrap.Append(reterr, err)
} }
obj.wg.Wait() // for now, this doesn't need to be a separate Wait() method obj.wg.Wait() // for now, this doesn't need to be a separate Wait() method
@@ -346,3 +423,8 @@ func (obj *Engine) Close() error {
func (obj *Engine) Graph() *pgraph.Graph { func (obj *Engine) Graph() *pgraph.Graph {
return obj.graph return obj.graph
} }
// statePrefix returns the dir where all the resource state is stored locally.
func (obj *Engine) statePrefix() string {
return fmt.Sprintf("%s/", path.Join(obj.Prefix, StateDir))
}

37
engine/graph/graph_test.go Normal file
View File

@@ -0,0 +1,37 @@
// Mgmt
// Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
// +build !root
package graph
import (
"fmt"
"testing"
"github.com/purpleidea/mgmt/util/errwrap"
)
func TestMultiErr(t *testing.T) {
var err error
e := fmt.Errorf("some error")
err = errwrap.Append(err, e) // build an error from a nil base
// ensure that this lib allows us to append to a nil
if err == nil {
t.Errorf("missing error")
}
}

2
engine/graph/refresh.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

295
engine/graph/reverse.go Normal file
View File

@@ -0,0 +1,295 @@
// Mgmt
// Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package graph
import (
"fmt"
"io/ioutil"
"os"
"path"
"sort"
"github.com/purpleidea/mgmt/engine"
engineUtil "github.com/purpleidea/mgmt/engine/util"
"github.com/purpleidea/mgmt/pgraph"
"github.com/purpleidea/mgmt/util/errwrap"
)
const (
// ReverseFile is the file name in the resource state dir where any
// reversal information is stored.
ReverseFile = "reverse"
// ReversePerm is the permissions mode used to create the ReverseFile.
ReversePerm = 0600
)
// Reversals adds the reversals onto the loaded graph. This should happen last,
// and before Commit.
func (obj *Engine) Reversals() error {
if obj.nextGraph == nil {
return fmt.Errorf("there is no active graph to add reversals to")
}
// Initially get all of the reversals to seek out all possible errors.
// XXX: The engine needs to know where data might have been stored if we
// XXX: want to potentially allow alternate read/write paths, like etcd.
// XXX: In this scenario, we'd have to store a token somewhere to let us
// XXX: know to look elsewhere for the special ReversalList read method.
data, err := obj.ReversalList() // (map[string]string, error)
if err != nil {
return errwrap.Wrapf(err, "the reversals had errors")
}
if len(data) == 0 {
return nil // end early
}
resMatch := func(r1, r2 engine.Res) bool { // simple match on UID only!
if r1.Kind() != r2.Kind() {
return false
}
if r1.Name() != r2.Name() {
return false
}
return true
}
resInList := func(needle engine.Res, haystack []engine.Res) bool {
for _, res := range haystack {
if resMatch(needle, res) {
return true
}
}
return false
}
if obj.Debug {
obj.Logf("decoding %d reversals...", len(data))
}
resources := []engine.Res{}
// do this in a sorted order so that it errors deterministically
sorted := []string{}
for key := range data {
sorted = append(sorted, key)
}
sort.Strings(sorted)
for _, key := range sorted {
val := data[key]
// XXX: replace this ResToB64 method with one that stores it in
// a human readable format, in case someone wants to hack and
// edit it manually.
// XXX: we probably want this to be YAML, it works with the diff
// too...
r, err := engineUtil.B64ToRes(val)
if err != nil {
return errwrap.Wrapf(err, "error decoding res with UID: `%s`", key)
}
res, ok := r.(engine.ReversibleRes)
if !ok {
// this requirement is here to keep things simpler...
return errwrap.Wrapf(err, "decoded res with UID: `%s` was not reversible", key)
}
matchFn := func(vertex pgraph.Vertex) (bool, error) {
r, ok := vertex.(engine.Res)
if !ok {
return false, fmt.Errorf("not a Res")
}
if !resMatch(r, res) {
return false, nil
}
return true, nil
}
// FIXME: not efficient, we could build a cache-map first
vertex, err := obj.nextGraph.VertexMatchFn(matchFn) // (Vertex, error)
if err != nil {
return errwrap.Wrapf(err, "error searching graph for match")
}
if vertex != nil { // found one!
continue // it doesn't need reversing yet
}
// TODO: check for (incompatible?) duplicates instead
if resInList(res, resources) { // we've already got this one...
continue
}
// We set this in two different places to be safe. It ensures
// that we erase the reversal state file after we've used it.
res.ReversibleMeta().Reversal = true // set this for later...
resources = append(resources, res)
}
if len(resources) == 0 {
return nil // end early
}
// Now that we've passed the chance of any errors, we modify the graph.
obj.Logf("adding %d reversals...", len(resources))
for _, res := range resources {
obj.nextGraph.AddVertex(res)
}
// TODO: Do we want a way for stored reversals to add edges too?
// It would be great to ensure we didn't add any loops here, but instead
// of checking now, we'll move the check into the main loop.
return nil
}
// ReversalList returns all the available pending reversal data on this host. It
// can then be decoded by whatever method is appropriate for.
func (obj *Engine) ReversalList() (map[string]string, error) {
result := make(map[string]string) // some key to contents
dir := obj.statePrefix() // loop through this dir...
files, err := ioutil.ReadDir(dir)
if err != nil && !os.IsNotExist(err) {
return nil, errwrap.Wrapf(err, "error reading list of state dirs")
} else if err != nil {
return result, nil // nothing found, no state dir exists yet
}
for _, x := range files {
key := x.Name() // some uid for the resource
file := path.Join(dir, key, ReverseFile)
content, err := ioutil.ReadFile(file)
if err != nil && !os.IsNotExist(err) {
return nil, errwrap.Wrapf(err, "could not read reverse file: %s", file)
} else if err != nil {
continue // file does not exist, skip
}
// file exists!
str := string(content)
result[key] = str // save
}
return result, nil
}
// ReversalInit performs the reversal initialization steps if necessary for this
// resource.
func (obj *State) ReversalInit() error {
res, ok := obj.Vertex.(engine.ReversibleRes)
if !ok {
return nil // nothing to do
}
if res.ReversibleMeta().Disabled {
return nil // nothing to do, reversal isn't enabled
}
// If the reversal is enabled, but we are the result of a previous
// reversal, then this will overwrite that older reversal request, and
// our resource should be designed to deal with that. This happens if we
// return a reversible resource as the reverse of a resource that was
// reversed. It's probably fairly rare.
if res.ReversibleMeta().Reversal {
obj.Logf("triangle reversal") // warn!
}
r, err := res.Reversed()
if err != nil {
return errwrap.Wrapf(err, "could not reverse: %s", res.String())
}
if r == nil {
return nil // this can't be reversed, or isn't implemented here
}
// We set this in two different places to be safe. It ensures that we
// erase the reversal state file after we've used it.
r.ReversibleMeta().Reversal = true // set this for later...
// XXX: replace this ResToB64 method with one that stores it in a human
// readable format, in case someone wants to hack and edit it manually.
// XXX: we probably want this to be YAML, it works with the diff too...
str, err := engineUtil.ResToB64(r)
if err != nil {
return errwrap.Wrapf(err, "could not encode: %s", res.String())
}
// TODO: put this method on traits.Reversible as part of the interface?
return obj.ReversalWrite(str, res.ReversibleMeta().Overwrite) // Store!
}
// ReversalClose performs the reversal shutdown steps if necessary for this
// resource.
func (obj *State) ReversalClose() error {
res, ok := obj.Vertex.(engine.ReversibleRes)
if !ok {
return nil // nothing to do
}
// Don't check res.ReversibleMeta().Disabled because we're removing the
// previous one. That value only applies if we're doing a new reversal.
if !res.ReversibleMeta().Reversal {
return nil // nothing to erase, we're not a reversal resource
}
if !obj.isStateOK { // did we successfully reverse?
obj.Logf("did not complete reversal") // warn
return nil
}
// TODO: put this method on traits.Reversible as part of the interface?
return obj.ReversalDelete() // Erase our reversal instructions.
}
// ReversalWrite stores the reversal state information for this resource.
func (obj *State) ReversalWrite(str string, overwrite bool) error {
dir, err := obj.varDir("") // private version
if err != nil {
return errwrap.Wrapf(err, "could not get VarDir for reverse")
}
file := path.Join(dir, ReverseFile) // return a unique file
content, err := ioutil.ReadFile(file)
if err != nil && !os.IsNotExist(err) {
return errwrap.Wrapf(err, "could not read reverse file: %s", file)
}
// file exists and we shouldn't overwrite if different
if err == nil && !overwrite {
// compare to existing file
oldStr := string(content)
if str != oldStr {
obj.Logf("existing, pending, reversible resource exists")
//obj.Logf("diff:")
//obj.Logf("") // TODO: print the diff w/o and secret values
return fmt.Errorf("existing, pending, reversible resource exists")
}
}
return ioutil.WriteFile(file, []byte(str), ReversePerm)
}
// ReversalDelete removes the reversal state information for this resource.
func (obj *State) ReversalDelete() error {
dir, err := obj.varDir("") // private version
if err != nil {
return errwrap.Wrapf(err, "could not get VarDir for reverse")
}
file := path.Join(dir, ReverseFile) // return a unique file
return errwrap.Wrapf(os.Remove(file), "could not remove reverse state file")
}

15
engine/graph/semaphore.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -23,9 +23,8 @@ import (
"strconv" "strconv"
"strings" "strings"
"github.com/purpleidea/mgmt/util/errwrap"
"github.com/purpleidea/mgmt/util/semaphore" "github.com/purpleidea/mgmt/util/semaphore"
multierr "github.com/hashicorp/go-multierror"
) )
// SemaSep is the trailing separator to split the semaphore id from the size. // SemaSep is the trailing separator to split the semaphore id from the size.
@@ -46,9 +45,8 @@ func (obj *Engine) semaLock(semas []string) error {
} }
obj.slock.Unlock() obj.slock.Unlock()
if err := sema.P(1); err != nil { // lock! err := sema.P(1) // lock!
reterr = multierr.Append(reterr, err) // list of errors reterr = errwrap.Append(reterr, err) // list of errors
}
} }
return reterr return reterr
} }
@@ -65,9 +63,8 @@ func (obj *Engine) semaUnlock(semas []string) error {
panic(fmt.Sprintf("graph: sema: %s does not exist", id)) panic(fmt.Sprintf("graph: sema: %s does not exist", id))
} }
if err := sema.V(1); err != nil { // unlock! err := sema.V(1) // unlock!
reterr = multierr.Append(reterr, err) // list of errors reterr = errwrap.Append(reterr, err) // list of errors
}
} }
return reterr return reterr
} }

2
engine/graph/semaphore_test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

54
engine/graph/sendrecv.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -22,9 +22,8 @@ import (
"reflect" "reflect"
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
engineUtil "github.com/purpleidea/mgmt/engine/util"
multierr "github.com/hashicorp/go-multierror" "github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
) )
// SendRecv pulls in the sent values into the receive slots. It is called by the // SendRecv pulls in the sent values into the receive slots. It is called by the
@@ -47,16 +46,51 @@ func (obj *Engine) SendRecv(res engine.RecvableRes) (map[string]bool, error) {
st = v.Res.Sent() st = v.Res.Sent()
} }
if st == nil {
e := fmt.Errorf("received nil value from: %s", v.Res)
err = errwrap.Append(err, e) // list of errors
continue
}
if e := engineUtil.StructFieldCompat(st, v.Key, res, k); e != nil {
err = errwrap.Append(err, e) // list of errors
continue
}
// send // send
m1, e := engineUtil.StructTagToFieldName(st)
if e != nil {
err = errwrap.Append(err, e) // list of errors
continue
}
key1, exists := m1[v.Key]
if !exists {
e := fmt.Errorf("requested key of `%s` not found in send struct", v.Key)
err = errwrap.Append(err, e) // list of errors
continue
}
obj1 := reflect.Indirect(reflect.ValueOf(st)) obj1 := reflect.Indirect(reflect.ValueOf(st))
type1 := obj1.Type() type1 := obj1.Type()
value1 := obj1.FieldByName(v.Key) value1 := obj1.FieldByName(key1)
kind1 := value1.Kind() kind1 := value1.Kind()
// recv // recv
m2, e := engineUtil.StructTagToFieldName(res)
if e != nil {
err = errwrap.Append(err, e) // list of errors
continue
}
key2, exists := m2[k]
if !exists {
e := fmt.Errorf("requested key of `%s` not found in recv struct", k)
err = errwrap.Append(err, e) // list of errors
continue
}
obj2 := reflect.Indirect(reflect.ValueOf(res)) // pass in full struct obj2 := reflect.Indirect(reflect.ValueOf(res)) // pass in full struct
type2 := obj2.Type() type2 := obj2.Type()
value2 := obj2.FieldByName(k) value2 := obj2.FieldByName(key2)
kind2 := value2.Kind() kind2 := value2.Kind()
if obj.Debug { if obj.Debug {
@@ -67,7 +101,7 @@ func (obj *Engine) SendRecv(res engine.RecvableRes) (map[string]bool, error) {
// i think we probably want the same kind, at least for now... // i think we probably want the same kind, at least for now...
if kind1 != kind2 { if kind1 != kind2 {
e := fmt.Errorf("kind mismatch between %s: %s and %s: %s", v.Res, kind1, res, kind2) e := fmt.Errorf("kind mismatch between %s: %s and %s: %s", v.Res, kind1, res, kind2)
err = multierr.Append(err, e) // list of errors err = errwrap.Append(err, e) // list of errors
continue continue
} }
@@ -75,21 +109,21 @@ func (obj *Engine) SendRecv(res engine.RecvableRes) (map[string]bool, error) {
// FIXME: do we want to relax this for string -> *string ? // FIXME: do we want to relax this for string -> *string ?
if e := TypeCmp(value1, value2); e != nil { if e := TypeCmp(value1, value2); e != nil {
e := errwrap.Wrapf(e, "type mismatch between %s and %s", v.Res, res) e := errwrap.Wrapf(e, "type mismatch between %s and %s", v.Res, res)
err = multierr.Append(err, e) // list of errors err = errwrap.Append(err, e) // list of errors
continue continue
} }
// if we can't set, then well this is pointless! // if we can't set, then well this is pointless!
if !value2.CanSet() { if !value2.CanSet() {
e := fmt.Errorf("can't set %s.%s", res, k) e := fmt.Errorf("can't set %s.%s", res, k)
err = multierr.Append(err, e) // list of errors err = errwrap.Append(err, e) // list of errors
continue continue
} }
// if we can't interface, we can't compare... // if we can't interface, we can't compare...
if !value1.CanInterface() || !value2.CanInterface() { if !value1.CanInterface() || !value2.CanInterface() {
e := fmt.Errorf("can't interface %s.%s", res, k) e := fmt.Errorf("can't interface %s.%s", res, k)
err = multierr.Append(err, e) // list of errors err = errwrap.Append(err, e) // list of errors
continue continue
} }

381
engine/graph/state.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -19,18 +19,14 @@ package graph
import ( import (
"fmt" "fmt"
"os"
"path"
"sync" "sync"
"time" "time"
"github.com/purpleidea/mgmt/converger" "github.com/purpleidea/mgmt/converger"
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/event"
"github.com/purpleidea/mgmt/pgraph" "github.com/purpleidea/mgmt/pgraph"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
) )
// State stores some state about the resource it is mapped to. // State stores some state about the resource it is mapped to.
@@ -51,7 +47,7 @@ type State struct {
// created if needed. // created if needed.
Prefix string Prefix string
//Converger converger.Converger //Converger *converger.Coordinator
// Debug turns on additional output and behaviours. // Debug turns on additional output and behaviours.
Debug bool Debug bool
@@ -61,49 +57,62 @@ type State struct {
timestamp int64 // last updated timestamp timestamp int64 // last updated timestamp
isStateOK bool // is state OK or do we need to run CheckApply ? isStateOK bool // is state OK or do we need to run CheckApply ?
workerErr error // did the Worker error?
// events is a channel of incoming events which is read by the Watch // doneChan closes when Watch should shut down. When any of the
// loop for that resource. It receives events like pause, start, and // following channels close, it causes this to close.
// poke. The channel shuts down to signal for Watch to exit. doneChan chan struct{}
eventsChan chan *event.Msg // incoming to resource
eventsLock *sync.Mutex // lock around sending and closing of events channel
eventsDone bool // is channel closed?
// outputChan is the channel that the engine listens on for events from // processDone is closed when the Process/CheckApply function fails
// permanently, and wants to cause Watch to exit.
processDone chan struct{}
// watchDone is closed when the Watch function fails permanently, and we
// close this to signal we should definitely exit. (Often redundant.)
watchDone chan struct{} // could be shared with limitDone
// limitDone is closed when the Watch function fails permanently, and we
// close this to signal we should definitely exit. This happens inside
// of the limit loop of the Process section of Worker.
limitDone chan struct{} // could be shared with watchDone
// removeDone is closed when the vertexRemoveFn method asks for an exit.
// This happens when we're switching graphs. The switch to an "empty" is
// the equivalent of asking for a final shutdown.
removeDone chan struct{}
// eventsDone is closed when we shutdown the Process loop because we
// closed without error. In theory this shouldn't happen, but it could
// if Watch returns without error for some reason.
eventsDone chan struct{}
// eventsChan is the channel that the engine listens on for events from
// the Watch loop for that resource. The event is nil normally, except // the Watch loop for that resource. The event is nil normally, except
// when events are sent on this channel from the engine. This only // when events are sent on this channel from the engine. This only
// happens as a signaling mechanism when Watch has shutdown and we want // happens as a signaling mechanism when Watch has shutdown and we want
// to notify the Process loop which reads from this. // to notify the Process loop which reads from this.
outputChan chan error // outgoing from resource eventsChan chan error // outgoing from resource
wg *sync.WaitGroup // pokeChan is a separate channel that the Process loop listens on to
exit *util.EasyExit // know when we might need to run Process. It never closes, and is safe
// to send on since it is buffered.
pokeChan chan struct{} // outgoing from resource
started chan struct{} // closes when it's started // paused represents if this particular res is paused or not.
stopped chan struct{} // closes when it's stopped paused bool
// pauseSignal closes to request a pause of this resource.
pauseSignal chan struct{}
// resumeSignal closes to request a resume of this resource.
resumeSignal chan struct{}
// pausedAck is used to send an ack message saying that we've paused.
pausedAck *util.EasyAck
starter bool // do we have an indegree of 0 ? wg *sync.WaitGroup // used for all vertex specific processes
working bool // is the Main() loop running ?
cuid converger.UID // primary converger cuid *converger.UID // primary converger
tuid converger.UID // secondary converger tuid *converger.UID // secondary converger
init *engine.Init // a copy of the init struct passed to res Init init *engine.Init // a copy of the init struct passed to res Init
} }
// Init initializes structures like channels. // Init initializes structures like channels.
func (obj *State) Init() error { func (obj *State) Init() error {
obj.eventsChan = make(chan *event.Msg)
obj.eventsLock = &sync.Mutex{}
obj.outputChan = make(chan error)
obj.wg = &sync.WaitGroup{}
obj.exit = util.NewEasyExit()
obj.started = make(chan struct{})
obj.stopped = make(chan struct{})
res, isRes := obj.Vertex.(engine.Res) res, isRes := obj.Vertex.(engine.Res)
if !isRes { if !isRes {
return fmt.Errorf("vertex is not a Res") return fmt.Errorf("vertex is not a Res")
@@ -121,6 +130,25 @@ func (obj *State) Init() error {
return fmt.Errorf("the Logf function is missing") return fmt.Errorf("the Logf function is missing")
} }
obj.doneChan = make(chan struct{})
obj.processDone = make(chan struct{})
obj.watchDone = make(chan struct{})
obj.limitDone = make(chan struct{})
obj.removeDone = make(chan struct{})
obj.eventsDone = make(chan struct{})
obj.eventsChan = make(chan error)
obj.pokeChan = make(chan struct{}, 1) // must be buffered
//obj.paused = false // starts off as started
obj.pauseSignal = make(chan struct{})
//obj.resumeSignal = make(chan struct{}) // happens on pause
//obj.pausedAck = util.NewEasyAck() // happens on pause
obj.wg = &sync.WaitGroup{}
//obj.cuid = obj.Converger.Register() // gets registered in Worker() //obj.cuid = obj.Converger.Register() // gets registered in Worker()
//obj.tuid = obj.Converger.Register() // gets registered in Worker() //obj.tuid = obj.Converger.Register() // gets registered in Worker()
@@ -129,24 +157,9 @@ func (obj *State) Init() error {
Hostname: obj.Hostname, Hostname: obj.Hostname,
// Watch: // Watch:
Running: func() error { Running: obj.event,
obj.tuid.StopTimer()
close(obj.started) // this is reset in the reset func
obj.isStateOK = false // assume we're initially dirty
// optimization: skip the initial send if not a starter
// because we'll get poked from a starter soon anyways!
if !obj.starter {
return nil
}
return obj.event()
},
Event: obj.event, Event: obj.event,
Events: obj.eventsChan, Done: obj.doneChan,
Read: obj.read,
Dirty: func() { // TODO: should we rename this SetDirty?
obj.tuid.StopTimer()
obj.isStateOK = false
},
// CheckApply: // CheckApply:
Refresh: func() bool { Refresh: func() bool {
@@ -190,6 +203,12 @@ func (obj *State) Init() error {
if obj.Debug { if obj.Debug {
obj.Logf("Init(%s)", res) obj.Logf("Init(%s)", res)
} }
// write the reverse request to the disk...
if err := obj.ReversalInit(); err != nil {
return err // TODO: test this code path...
}
err := res.Init(obj.init) err := res.Init(obj.init)
if obj.Debug { if obj.Debug {
obj.Logf("Init(%s): Return(%+v)", res, err) obj.Logf("Init(%s): Return(%+v)", res, err)
@@ -223,195 +242,110 @@ func (obj *State) Close() error {
if obj.Debug { if obj.Debug {
obj.Logf("Close(%s)", res) obj.Logf("Close(%s)", res)
} }
err := res.Close()
var reverr error
// clear the reverse request from the disk...
if err := obj.ReversalClose(); err != nil {
// TODO: test this code path...
// TODO: should this be an error or a warning?
reverr = err
}
reterr := res.Close()
if obj.Debug { if obj.Debug {
obj.Logf("Close(%s): Return(%+v)", res, err) obj.Logf("Close(%s): Return(%+v)", res, reterr)
} }
return err reterr = errwrap.Append(reterr, reverr)
return reterr
} }
// reset is run to reset the state so that Watch can run a second time. Thus is // Poke sends a notification on the poke channel. This channel is used to notify
// needed for the Watch retry in particular. // the Worker to run the Process/CheckApply when it can. This is used when there
func (obj *State) reset() { // is a need to schedule or reschedule some work which got postponed or dropped.
obj.started = make(chan struct{}) // This doesn't contain any internal synchronization primitives or wait groups,
obj.stopped = make(chan struct{}) // callers are expected to make sure that they don't leave any of these running
} // by the time the Worker() shuts down.
// Poke sends a nil message on the outputChan. This channel is used by the
// resource to signal a possible change. This will cause the Process loop to
// run if it can.
func (obj *State) Poke() { func (obj *State) Poke() {
// add a wait group on the vertex we're poking! // redundant
obj.wg.Add(1) //if len(obj.pokeChan) > 0 {
defer obj.wg.Done() // return
//}
// now that we've added to the wait group, obj.outputChan won't close...
// so see if there's an exit signal before we release the wait group!
// XXX: i don't think this is necessarily happening, but maybe it is?
// XXX: re-write some of the engine to ensure that: "the sender closes"!
select {
case <-obj.exit.Signal():
return // skip sending the poke b/c we're closing
default:
}
select { select {
case obj.outputChan <- nil: case obj.pokeChan <- struct{}{}:
default: // if chan is now full because more than one poke happened...
case <-obj.exit.Signal():
} }
} }
// Event sends a Pause or Start event to the resource. It can also be used to // Pause pauses this resource. It should not be called on any already paused
// send Poke events, but it's much more efficient to send them directly instead // resource. It will block until the resource pauses with an acknowledgment, or
// of passing them through the resource. // until an exit for that resource is seen. If the latter happens it will error.
func (obj *State) Event(msg *event.Msg) { // It is NOT thread-safe with the Resume() method so only call either one at a
// TODO: should these happen after the lock? // time.
obj.wg.Add(1) func (obj *State) Pause() error {
defer obj.wg.Done() if obj.paused {
return fmt.Errorf("already paused")
obj.eventsLock.Lock()
defer obj.eventsLock.Unlock()
if obj.eventsDone { // closing, skip events...
return
} }
if msg.Kind == event.KindExit { // set this so future events don't deadlock obj.pausedAck = util.NewEasyAck()
obj.Logf("exit event...") obj.resumeSignal = make(chan struct{}) // build the resume signal
obj.eventsDone = true close(obj.pauseSignal)
close(obj.eventsChan) // causes resource Watch loop to close obj.Poke() // unblock and notice the pause if necessary
obj.exit.Done(nil) // trigger exit signal to unblock some cases
return
}
// wait for ack (or exit signal)
select { select {
case obj.eventsChan <- msg: case <-obj.pausedAck.Wait(): // we got it!
// we're paused
case <-obj.exit.Signal(): case <-obj.doneChan:
return engine.ErrClosed
} }
} obj.paused = true
// read is a helper function used inside the main select statement of resources.
// If it returns an error, then this is a signal for the resource to exit.
func (obj *State) read(msg *event.Msg) error {
switch msg.Kind {
case event.KindPoke:
return obj.event() // a poke needs to cause an event...
case event.KindStart:
return fmt.Errorf("unexpected start")
case event.KindPause:
// pass
case event.KindExit:
return engine.ErrSignalExit
default:
return fmt.Errorf("unhandled event: %+v", msg.Kind)
}
// we're paused now
select {
case msg, ok := <-obj.eventsChan:
if !ok {
return engine.ErrWatchExit
}
switch msg.Kind {
case event.KindPoke:
return fmt.Errorf("unexpected poke")
case event.KindPause:
return fmt.Errorf("unexpected pause")
case event.KindStart:
// resumed
return nil return nil
case event.KindExit:
return engine.ErrSignalExit
default:
return fmt.Errorf("unhandled event: %+v", msg.Kind)
}
}
} }
// event is a helper function to send an event from the resource Watch loop. It // Resume unpauses this resource. It can be safely called on a brand-new
// can be used for the initial `running` event, or any regular event. If it // resource that has just started running without incident. It is NOT
// returns an error, then the Watch loop must return this error and shutdown. // thread-safe with the Pause() method, so only call either one at a time.
func (obj *State) event() error { func (obj *State) Resume() {
// loop until we sent on obj.outputChan or exit with error // TODO: do we need a mutex around Resume?
for { if !obj.paused { // no need to unpause brand-new resources
select { return
// send "activity" event
case obj.outputChan <- nil:
return nil // sent event!
// make sure to keep handling incoming
case msg, ok := <-obj.eventsChan:
if !ok {
return engine.ErrWatchExit
}
switch msg.Kind {
case event.KindPoke:
// we're trying to send an event, so swallow the
// poke: it's what we wanted to have happen here
continue
case event.KindStart:
return fmt.Errorf("unexpected start")
case event.KindPause:
// pass
case event.KindExit:
return engine.ErrSignalExit
default:
return fmt.Errorf("unhandled event: %+v", msg.Kind)
}
} }
// we're paused now obj.pauseSignal = make(chan struct{}) // rebuild for next pause
select { close(obj.resumeSignal)
case msg, ok := <-obj.eventsChan: //obj.Poke() // not needed, we're already waiting for resume
if !ok {
return engine.ErrWatchExit
}
switch msg.Kind {
case event.KindPoke:
return fmt.Errorf("unexpected poke")
case event.KindPause:
return fmt.Errorf("unexpected pause")
case event.KindStart:
// resumed
case event.KindExit:
return engine.ErrSignalExit
default: obj.paused = false
return fmt.Errorf("unhandled event: %+v", msg.Kind)
} // no need to wait for it to resume
} //return // implied
}
} }
// varDir returns the path to a working directory for the resource. It will try // event is a helper function to send an event to the CheckApply process loop.
// and create the directory first, and return an error if this failed. The dir // It can be used for the initial `running` event, or any regular event. You
// should be cleaned up by the resource on Close if it wishes to discard the // should instead use Poke() to "schedule" a new Process/CheckApply loop when
// contents. If it does not, then a future resource with the same kind and name // one might be needed. This method will block until we're unpaused and ready to
// may see those contents in that directory. The resource should clean up the // receive on the events channel.
// contents before use if it is important that nothing exist. It is always func (obj *State) event() {
// possible that contents could remain after an abrupt crash, so do not store obj.setDirty() // assume we're initially dirty
// overly sensitive data unless you're aware of the risks.
func (obj *State) varDir(extra string) (string, error) { select {
// Using extra adds additional dirs onto our namespace. An empty extra case obj.eventsChan <- nil:
// adds no additional directories. // send!
if obj.Prefix == "" { // safety
return "", fmt.Errorf("the VarDir prefix is empty")
} }
// an empty string at the end has no effect //return // implied
p := fmt.Sprintf("%s/", path.Join(obj.Prefix, extra)) }
if err := os.MkdirAll(p, 0770); err != nil {
return "", errwrap.Wrapf(err, "can't create prefix in: %s", p)
}
// returns with a trailing slash as per the mgmt file res convention // setDirty marks the resource state as dirty. This signals to the engine that
return p, nil // CheckApply will have some work to do in order to converge it.
func (obj *State) setDirty() {
obj.tuid.StopTimer()
obj.isStateOK = false
} }
// poll is a replacement for Watch when the Poll metaparameter is used. // poll is a replacement for Watch when the Poll metaparameter is used.
@@ -420,34 +354,17 @@ func (obj *State) poll(interval uint32) error {
ticker := time.NewTicker(time.Duration(interval) * time.Second) ticker := time.NewTicker(time.Duration(interval) * time.Second)
defer ticker.Stop() defer ticker.Stop()
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event?
for { for {
select { select {
case <-ticker.C: // received the timer event case <-ticker.C: // received the timer event
obj.init.Logf("polling...") obj.init.Logf("polling...")
send = true
obj.init.Dirty() // dirty
case event, ok := <-obj.init.Events: case <-obj.init.Done: // signal for shutdown request
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs obj.init.Event() // notify engine of an event (this can block)
if send {
send = false
if err := obj.init.Event(); err != nil {
return err // exit if requested
}
}
} }
} }

51
engine/graph/vardir.go Normal file
View File

@@ -0,0 +1,51 @@
// Mgmt
// Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package graph
import (
"fmt"
"os"
"path"
"github.com/purpleidea/mgmt/util/errwrap"
)
// varDir returns the path to a working directory for the resource. It will try
// and create the directory first, and return an error if this failed. The dir
// should be cleaned up by the resource on Close if it wishes to discard the
// contents. If it does not, then a future resource with the same kind and name
// may see those contents in that directory. The resource should clean up the
// contents before use if it is important that nothing exist. It is always
// possible that contents could remain after an abrupt crash, so do not store
// overly sensitive data unless you're aware of the risks.
func (obj *State) varDir(extra string) (string, error) {
// Using extra adds additional dirs onto our namespace. An empty extra
// adds no additional directories.
if obj.Prefix == "" { // safety
return "", fmt.Errorf("the VarDir prefix is empty")
}
// an empty string at the end has no effect
p := fmt.Sprintf("%s/", path.Join(obj.Prefix, extra))
if err := os.MkdirAll(p, 0770); err != nil {
return "", errwrap.Wrapf(err, "can't create prefix in: %s", p)
}
// returns with a trailing slash as per the mgmt file res convention
return p, nil
}

33
engine/metaparams.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -22,8 +22,8 @@ import (
"strconv" "strconv"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
"golang.org/x/time/rate" "golang.org/x/time/rate"
) )
@@ -37,6 +37,8 @@ var DefaultMetaParams = &MetaParams{
Limit: rate.Inf, // defaults to no limit Limit: rate.Inf, // defaults to no limit
Burst: 0, // no burst needed on an infinite rate Burst: 0, // no burst needed on an infinite rate
//Sema: []string{}, //Sema: []string{},
Rewatch: true,
Realize: false, // true would be more awesome, but unexpected for users
} }
// MetaRes is the interface a resource must implement to support meta params. // MetaRes is the interface a resource must implement to support meta params.
@@ -81,6 +83,24 @@ type MetaParams struct {
// has a count equal to 1, is different from a sema named `foo:1` which // has a count equal to 1, is different from a sema named `foo:1` which
// also has a count equal to 1, but is a different semaphore. // also has a count equal to 1, but is a different semaphore.
Sema []string `yaml:"sema"` Sema []string `yaml:"sema"`
// Rewatch specifies whether we re-run the Watch worker during a swap
// if it has errored. When doing a GraphCmp to swap the graphs, if this
// is true, and this particular worker has errored, then we'll remove it
// and add it back as a new vertex, thus causing it to run again. This
// is different from the Retry metaparam which applies during the normal
// execution. It is only when this is exhausted that we're in permanent
// worker failure, and only then can we rely on this metaparam.
Rewatch bool `yaml:"rewatch"`
// Realize ensures that the resource is guaranteed to converge at least
// once before a potential graph swap removes or changes it. This
// guarantee is useful for fast changing graphs, to ensure that the
// brief creation of a resource is seen. This guarantee does not prevent
// against the engine quitting normally, and it can't guarantee it if
// the resource is blocked because of a failed pre-requisite resource.
// XXX: Not implemented!
Realize bool `yaml:"realize"`
} }
// Cmp compares two AutoGroupMeta structs and determines if they're equivalent. // Cmp compares two AutoGroupMeta structs and determines if they're equivalent.
@@ -118,6 +138,13 @@ func (obj *MetaParams) Cmp(meta *MetaParams) error {
return errwrap.Wrapf(err, "values for Sema are different") return errwrap.Wrapf(err, "values for Sema are different")
} }
if obj.Rewatch != meta.Rewatch {
return fmt.Errorf("values for Rewatch are different")
}
if obj.Realize != meta.Realize {
return fmt.Errorf("values for Realize are different")
}
return nil return nil
} }
@@ -154,6 +181,8 @@ func (obj *MetaParams) Copy() *MetaParams {
Limit: obj.Limit, // FIXME: can we copy this type like this? test me! Limit: obj.Limit, // FIXME: can we copy this type like this? test me!
Burst: obj.Burst, Burst: obj.Burst,
Sema: sema, Sema: sema,
Rewatch: obj.Rewatch,
Realize: obj.Realize,
} }
} }

2
engine/metaparams_test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

2
engine/refresh.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

23
engine/resources.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -21,9 +21,8 @@ import (
"encoding/gob" "encoding/gob"
"fmt" "fmt"
"github.com/purpleidea/mgmt/engine/event" "github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
"gopkg.in/yaml.v2" "gopkg.in/yaml.v2"
) )
@@ -93,22 +92,14 @@ type Init struct {
// Called from within Watch: // Called from within Watch:
// Running must be called after your watches are all started and ready. // Running must be called after your watches are all started and ready.
Running func() error Running func()
// Event sends an event notifying the engine of a possible state change. // Event sends an event notifying the engine of a possible state change.
Event func() error Event func()
// Events returns a channel that we must watch for messages from the // Done returns a channel that will close to signal to us that it's time
// engine. When it closes, this is a signal to shutdown. // for us to shutdown.
Events chan *event.Msg Done chan struct{}
// Read processes messages that come in from the Events channel. It is a
// helper method that knows how to handle the pause mechanism correctly.
Read func(*event.Msg) error
// Dirty marks the resource state as dirty. This signals to the engine
// that CheckApply will have some work to do in order to converge it.
Dirty func()
// Called from within CheckApply: // Called from within CheckApply:

21
engine/resources/augeas.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -27,8 +27,8 @@ import (
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
"github.com/purpleidea/mgmt/recwatch" "github.com/purpleidea/mgmt/recwatch"
"github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
// FIXME: we vendor go/augeas because master requires augeas 1.6.0 // FIXME: we vendor go/augeas because master requires augeas 1.6.0
// and libaugeas-dev-1.6.0 is not yet available in a PPA. // and libaugeas-dev-1.6.0 is not yet available in a PPA.
"honnef.co/go/augeas" "honnef.co/go/augeas"
@@ -135,10 +135,7 @@ func (obj *AugeasRes) Watch() error {
} }
defer obj.recWatcher.Close() defer obj.recWatcher.Close()
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? var send = false // send event?
for { for {
@@ -158,23 +155,15 @@ func (obj *AugeasRes) Watch() error {
obj.init.Logf("Event(%s): %v", event.Body.Name, event.Body.Op) obj.init.Logf("Event(%s): %v", event.Body.Name, event.Body.Op)
} }
send = true send = true
obj.init.Dirty() // dirty
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }

94
engine/resources/aws_ec2.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -34,6 +34,7 @@ import (
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
"github.com/purpleidea/mgmt/util/errwrap"
"github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/awserr"
@@ -42,8 +43,6 @@ import (
cwe "github.com/aws/aws-sdk-go/service/cloudwatchevents" cwe "github.com/aws/aws-sdk-go/service/cloudwatchevents"
"github.com/aws/aws-sdk-go/service/ec2" "github.com/aws/aws-sdk-go/service/ec2"
"github.com/aws/aws-sdk-go/service/sns" "github.com/aws/aws-sdk-go/service/sns"
multierr "github.com/hashicorp/go-multierror"
errwrap "github.com/pkg/errors"
) )
func init() { func init() {
@@ -393,17 +392,14 @@ func (obj *AwsEc2Res) Close() error {
// clean up sns objects created by Init/snsWatch // clean up sns objects created by Init/snsWatch
if obj.snsClient != nil { if obj.snsClient != nil {
// delete the topic and associated subscriptions // delete the topic and associated subscriptions
if err := obj.snsDeleteTopic(obj.snsTopicArn); err != nil { e1 := obj.snsDeleteTopic(obj.snsTopicArn)
errList = multierr.Append(errList, err) errList = errwrap.Append(errList, e1)
}
// remove the target // remove the target
if err := obj.cweRemoveTarget(CweTargetID, CweRuleName); err != nil { e2 := obj.cweRemoveTarget(CweTargetID, CweRuleName)
errList = multierr.Append(errList, err) errList = errwrap.Append(errList, e2)
}
// delete the cloudwatch rule // delete the cloudwatch rule
if err := obj.cweDeleteRule(CweRuleName); err != nil { e3 := obj.cweDeleteRule(CweRuleName)
errList = multierr.Append(errList, err) errList = errwrap.Append(errList, e3)
}
} }
return errList return errList
@@ -423,9 +419,7 @@ func (obj *AwsEc2Res) longpollWatch() error {
// We tell the engine that we're running right away. This is not correct, // We tell the engine that we're running right away. This is not correct,
// but the api doesn't have a way to signal when the waiters are ready. // but the api doesn't have a way to signal when the waiters are ready.
if err := obj.init.Running(); err != nil { obj.init.Running() // when started, notify engine that we're running
return err // exit if requested
}
// cancellable context used for exiting cleanly // cancellable context used for exiting cleanly
ctx, cancel := context.WithCancel(context.TODO()) ctx, cancel := context.WithCancel(context.TODO())
@@ -488,14 +482,6 @@ func (obj *AwsEc2Res) longpollWatch() error {
// process events from the goroutine // process events from the goroutine
for { for {
select { select {
case event, ok := <-obj.init.Events:
if !ok {
return nil
}
if err := obj.init.Read(event); err != nil {
return err
}
case msg, ok := <-obj.awsChan: case msg, ok := <-obj.awsChan:
if !ok { if !ok {
return nil return nil
@@ -509,15 +495,16 @@ func (obj *AwsEc2Res) longpollWatch() error {
continue continue
default: default:
obj.init.Logf("State: %v", msg.state) obj.init.Logf("State: %v", msg.state)
obj.init.Dirty() // dirty
send = true send = true
} }
case <-obj.init.Done: // closed by the engine to signal shutdown
return nil
} }
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
@@ -587,14 +574,6 @@ func (obj *AwsEc2Res) snsWatch() error {
// process events // process events
for { for {
select { select {
case event, ok := <-obj.init.Events:
if !ok {
return nil
}
if err := obj.init.Read(event); err != nil {
return err
}
case msg, ok := <-obj.awsChan: case msg, ok := <-obj.awsChan:
if !ok { if !ok {
return nil return nil
@@ -607,26 +586,25 @@ func (obj *AwsEc2Res) snsWatch() error {
// is confirmed, we are ready to receive events, so we // is confirmed, we are ready to receive events, so we
// can notify the engine that we're running. // can notify the engine that we're running.
if msg.event == awsEc2EventWatchReady { if msg.event == awsEc2EventWatchReady {
if err := obj.init.Running(); err != nil { obj.init.Running() // when started, notify engine that we're running
return err // exit if requested
}
continue continue
} }
obj.init.Logf("State: %v", msg.event) obj.init.Logf("State: %v", msg.event)
obj.init.Dirty() // dirty
send = true send = true
case <-obj.init.Done: // closed by the engine to signal shutdown
return nil
} }
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
// CheckApply method for AwsEc2 resource. // CheckApply method for AwsEc2 resource.
func (obj *AwsEc2Res) CheckApply(apply bool) (checkOK bool, err error) { func (obj *AwsEc2Res) CheckApply(apply bool) (bool, error) {
obj.init.Logf("CheckApply(%t)", apply) obj.init.Logf("CheckApply(%t)", apply)
// find the instance we need to check // find the instance we need to check
@@ -773,45 +751,37 @@ func (obj *AwsEc2Res) CheckApply(apply bool) (checkOK bool, err error) {
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *AwsEc2Res) Cmp(r engine.Res) error { func (obj *AwsEc2Res) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *AwsEc2Res) Compare(r engine.Res) bool {
// we can only compare AwsEc2Res to others of the same resource kind // we can only compare AwsEc2Res to others of the same resource kind
res, ok := r.(*AwsEc2Res) res, ok := r.(*AwsEc2Res)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.State != res.State { if obj.State != res.State {
return false return fmt.Errorf("the State differs")
} }
if obj.Region != res.Region { if obj.Region != res.Region {
return false return fmt.Errorf("the Region differs")
} }
if obj.Type != res.Type { if obj.Type != res.Type {
return false return fmt.Errorf("the Type differs")
} }
if obj.ImageID != res.ImageID { if obj.ImageID != res.ImageID {
return false return fmt.Errorf("the ImageID differs")
} }
if obj.WatchEndpoint != res.WatchEndpoint { if obj.WatchEndpoint != res.WatchEndpoint {
return false return fmt.Errorf("the WatchEndpoint differs")
} }
if obj.WatchListenAddr != res.WatchListenAddr { if obj.WatchListenAddr != res.WatchListenAddr {
return false return fmt.Errorf("the WatchListenAddr differs")
} }
if obj.ErrorOnMalformedPost != res.ErrorOnMalformedPost { if obj.ErrorOnMalformedPost != res.ErrorOnMalformedPost {
return false return fmt.Errorf("the ErrorOnMalformedPost differs")
} }
if obj.UserData != res.UserData { if obj.UserData != res.UserData {
return false return fmt.Errorf("the UserData differs")
} }
return true return nil
} }
func (obj *AwsEc2Res) prependName() string { func (obj *AwsEc2Res) prependName() string {
@@ -1047,7 +1017,7 @@ func (obj *AwsEc2Res) snsMakeTopic() (string, error) {
} }
obj.init.Logf("Created SNS Topic") obj.init.Logf("Created SNS Topic")
if topic.TopicArn == nil { if topic.TopicArn == nil {
return "", fmt.Errorf("TopicArn is nil") return "", fmt.Errorf("the TopicArn is nil")
} }
return *topic.TopicArn, nil return *topic.TopicArn, nil
} }

250
engine/resources/config_etcd.go Normal file
View File

@@ -0,0 +1,250 @@
// Mgmt
// Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package resources
import (
"context"
"fmt"
"sync"
"time"
"github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/traits"
"github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
)
func init() {
engine.RegisterResource("config:etcd", func() engine.Res { return &ConfigEtcdRes{} })
}
const (
sizeCheckApplyTimeout = 5 * time.Second
)
// ConfigEtcdRes is a resource that sets mgmt's etcd configuration.
type ConfigEtcdRes struct {
traits.Base // add the base methods without re-implementation
init *engine.Init
// IdealClusterSize is the requested minimum size of the cluster. If you
// set this to zero, it will cause a cluster wide shutdown if
// AllowSizeShutdown is true. If it's not true, then it will cause a
// validation error.
IdealClusterSize uint16 `lang:"idealclustersize"`
// AllowSizeShutdown is a required safety flag that you must set to true
// if you want to allow causing a cluster shutdown by setting
// IdealClusterSize to zero.
AllowSizeShutdown bool `lang:"allow_size_shutdown"`
// sizeFlag determines whether sizeCheckApply already ran or not.
sizeFlag bool
interruptChan chan struct{}
wg *sync.WaitGroup
}
// Default returns some sensible defaults for this resource.
func (obj *ConfigEtcdRes) Default() engine.Res {
return &ConfigEtcdRes{}
}
// Validate if the params passed in are valid data.
func (obj *ConfigEtcdRes) Validate() error {
if obj.IdealClusterSize < 0 {
return fmt.Errorf("the IdealClusterSize param must be positive")
}
if obj.IdealClusterSize == 0 && !obj.AllowSizeShutdown {
return fmt.Errorf("the IdealClusterSize can't be zero if AllowSizeShutdown is false")
}
return nil
}
// Init runs some startup code for this resource.
func (obj *ConfigEtcdRes) Init(init *engine.Init) error {
obj.init = init // save for later
obj.interruptChan = make(chan struct{})
obj.wg = &sync.WaitGroup{}
return nil
}
// Close is run by the engine to clean up after the resource is done.
func (obj *ConfigEtcdRes) Close() error {
obj.wg.Wait() // bonus
return nil
}
// Watch is the primary listener for this resource and it outputs events.
func (obj *ConfigEtcdRes) Watch() error {
obj.wg.Add(1)
defer obj.wg.Done()
// FIXME: add timeout to context
// The obj.init.Done channel is closed by the engine to signal shutdown.
ctx, cancel := util.ContextWithCloser(context.Background(), obj.init.Done)
defer cancel()
ch, err := obj.init.World.IdealClusterSizeWatch(util.CtxWithWg(ctx, obj.wg))
if err != nil {
return errwrap.Wrapf(err, "could not watch ideal cluster size")
}
obj.init.Running() // when started, notify engine that we're running
Loop:
for {
select {
case event, ok := <-ch:
if !ok {
break Loop
}
if obj.init.Debug {
obj.init.Logf("event: %+v", event)
}
// pass through and send an event
case <-obj.init.Done: // closed by the engine to signal shutdown
}
obj.init.Event() // notify engine of an event (this can block)
}
return nil
}
// sizeCheckApply sets the IdealClusterSize parameter. If it sees a value change
// to zero, then it *won't* try and change it away from zero, because it assumes
// that someone has requested a shutdown. If the value is seen on first startup,
// then it will change it, because it might be a zero from the previous cluster.
func (obj *ConfigEtcdRes) sizeCheckApply(apply bool) (bool, error) {
wg := &sync.WaitGroup{}
defer wg.Wait() // this must be above the defer cancel() call
ctx, cancel := context.WithTimeout(context.Background(), sizeCheckApplyTimeout)
defer cancel()
wg.Add(1)
go func() {
defer wg.Done()
select {
case <-obj.interruptChan:
cancel()
case <-ctx.Done():
// let this exit
}
}()
val, err := obj.init.World.IdealClusterSizeGet(ctx)
if err != nil {
return false, errwrap.Wrapf(err, "could not get ideal cluster size")
}
// if we got a value of zero, and we've already run before, then it's ok
if obj.IdealClusterSize != 0 && val == 0 && obj.sizeFlag {
obj.init.Logf("impending cluster shutdown, not setting ideal cluster size")
return true, nil // impending shutdown, don't try and cancel it.
}
obj.sizeFlag = true
// must be done after setting the above flag
if obj.IdealClusterSize == val { // state is correct
return true, nil
}
if !apply {
return false, nil
}
// set!
// This is run as a transaction so we detect if we needed to change it.
changed, err := obj.init.World.IdealClusterSizeSet(ctx, obj.IdealClusterSize)
if err != nil {
return false, errwrap.Wrapf(err, "could not set ideal cluster size")
}
if !changed {
return true, nil // we lost a race, which means no change needed
}
obj.init.Logf("set dynamic cluster size to: %d", obj.IdealClusterSize)
return false, nil
}
// CheckApply method for Noop resource. Does nothing, returns happy!
func (obj *ConfigEtcdRes) CheckApply(apply bool) (bool, error) {
checkOK := true
if c, err := obj.sizeCheckApply(apply); err != nil {
return false, err
} else if !c {
checkOK = false
}
// TODO: add more config settings management here...
//if c, err := obj.TODOCheckApply(apply); err != nil {
// return false, err
//} else if !c {
// checkOK = false
//}
return checkOK, nil // w00t
}
// Cmp compares two resources and returns an error if they are not equivalent.
func (obj *ConfigEtcdRes) Cmp(r engine.Res) error {
// we can only compare ConfigEtcdRes to others of the same resource kind
res, ok := r.(*ConfigEtcdRes)
if !ok {
return fmt.Errorf("not a %s", obj.Kind())
}
if obj.IdealClusterSize != res.IdealClusterSize {
return fmt.Errorf("the IdealClusterSize param differs")
}
if obj.AllowSizeShutdown != res.AllowSizeShutdown {
return fmt.Errorf("the AllowSizeShutdown param differs")
}
return nil
}
// Interrupt is called to ask the execution of this resource to end early.
func (obj *ConfigEtcdRes) Interrupt() error {
close(obj.interruptChan)
return nil
}
// UnmarshalYAML is the custom unmarshal handler for this struct.
// It is primarily useful for setting the defaults.
func (obj *ConfigEtcdRes) UnmarshalYAML(unmarshal func(interface{}) error) error {
type rawRes ConfigEtcdRes // indirection to avoid infinite recursion
def := obj.Default() // get the default
res, ok := def.(*ConfigEtcdRes) // put in the right format
if !ok {
return fmt.Errorf("could not convert to ConfigEtcdRes")
}
raw := rawRes(*res) // convert; the defaults go here
if err := unmarshal(&raw); err != nil {
return err
}
*obj = ConfigEtcdRes(raw) // restore from indirection with type conversion!
return nil
}

37
engine/resources/cron.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -31,12 +31,12 @@ import (
engineUtil "github.com/purpleidea/mgmt/engine/util" engineUtil "github.com/purpleidea/mgmt/engine/util"
"github.com/purpleidea/mgmt/recwatch" "github.com/purpleidea/mgmt/recwatch"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
sdbus "github.com/coreos/go-systemd/dbus" sdbus "github.com/coreos/go-systemd/dbus"
"github.com/coreos/go-systemd/unit" "github.com/coreos/go-systemd/unit"
systemdUtil "github.com/coreos/go-systemd/util" systemdUtil "github.com/coreos/go-systemd/util"
"github.com/godbus/dbus" "github.com/godbus/dbus"
errwrap "github.com/pkg/errors"
) )
const ( const (
@@ -271,10 +271,7 @@ func (obj *CronRes) Watch() error {
} }
defer obj.recWatcher.Close() defer obj.recWatcher.Close()
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? var send = false // send event?
for { for {
@@ -285,7 +282,7 @@ func (obj *CronRes) Watch() error {
obj.init.Logf("%+v", event) obj.init.Logf("%+v", event)
} }
send = true send = true
obj.init.Dirty() // dirty
case event, ok := <-obj.recWatcher.Events(): case event, ok := <-obj.recWatcher.Events():
// process unit file recwatch events // process unit file recwatch events
if !ok { // channel shutdown if !ok { // channel shutdown
@@ -298,21 +295,14 @@ func (obj *CronRes) Watch() error {
obj.init.Logf("Event(%s): %v", event.Body.Name, event.Body.Op) obj.init.Logf("Event(%s): %v", event.Body.Name, event.Body.Op)
} }
send = true send = true
obj.init.Dirty() // dirty
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
@@ -320,28 +310,29 @@ func (obj *CronRes) Watch() error {
// CheckApply is run to check the state and, if apply is true, to apply the // CheckApply is run to check the state and, if apply is true, to apply the
// necessary changes to reach the desired state. This is run before Watch and // necessary changes to reach the desired state. This is run before Watch and
// again if Watch finds a change occurring to the state. // again if Watch finds a change occurring to the state.
func (obj *CronRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *CronRes) CheckApply(apply bool) (bool, error) {
ok := true checkOK := true
// use the embedded file resource to apply the correct state // use the embedded file resource to apply the correct state
if c, err := obj.file.CheckApply(apply); err != nil { if c, err := obj.file.CheckApply(apply); err != nil {
return false, errwrap.Wrapf(err, "nested file failed") return false, errwrap.Wrapf(err, "nested file failed")
} else if !c { } else if !c {
ok = false checkOK = false
} }
// check timer state and apply the defined state if needed // check timer state and apply the defined state if needed
if c, err := obj.unitCheckApply(apply); err != nil { if c, err := obj.unitCheckApply(apply); err != nil {
return false, errwrap.Wrapf(err, "unitCheckApply error") return false, errwrap.Wrapf(err, "unitCheckApply error")
} else if !c { } else if !c {
ok = false checkOK = false
} }
return ok, nil return checkOK, nil
} }
// unitCheckApply checks the state of the systemd-timer unit and, if apply is // unitCheckApply checks the state of the systemd-timer unit and, if apply is
// true, applies the defined state. // true, applies the defined state.
func (obj *CronRes) unitCheckApply(apply bool) (checkOK bool, err error) { func (obj *CronRes) unitCheckApply(apply bool) (bool, error) {
var conn *sdbus.Conn var conn *sdbus.Conn
var godbusConn *dbus.Conn var godbusConn *dbus.Conn
var err error
// this resource depends on systemd to ensure that it's running // this resource depends on systemd to ensure that it's running
if !systemdUtil.IsRunningSystemd() { if !systemdUtil.IsRunningSystemd() {

25
engine/resources/docker_container.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -30,13 +30,13 @@ import (
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
"github.com/docker/docker/api/types" "github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/container" "github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/filters" "github.com/docker/docker/api/types/filters"
"github.com/docker/docker/client" "github.com/docker/docker/client"
"github.com/docker/go-connections/nat" "github.com/docker/go-connections/nat"
errwrap "github.com/pkg/errors"
) )
const ( const (
@@ -168,10 +168,7 @@ func (obj *DockerContainerRes) Watch() error {
eventChan, errChan := obj.client.Events(ctx, types.EventsOptions{}) eventChan, errChan := obj.client.Events(ctx, types.EventsOptions{})
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? var send = false // send event?
for { for {
@@ -184,33 +181,27 @@ func (obj *DockerContainerRes) Watch() error {
obj.init.Logf("%+v", event) obj.init.Logf("%+v", event)
} }
send = true send = true
obj.init.Dirty() // dirty
case err, ok := <-errChan: case err, ok := <-errChan:
if !ok { if !ok {
return nil return nil
} }
return err return err
case event, ok := <-obj.init.Events:
if !ok { case <-obj.init.Done: // closed by the engine to signal shutdown
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
// CheckApply method for Docker resource. // CheckApply method for Docker resource.
func (obj *DockerContainerRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *DockerContainerRes) CheckApply(apply bool) (bool, error) {
var id string var id string
var destroy bool var destroy bool

2
engine/resources/docker_container_test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

429
engine/resources/exec.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -20,19 +20,19 @@ package resources
import ( import (
"bufio" "bufio"
"bytes" "bytes"
"context"
"fmt" "fmt"
"os/exec" "os/exec"
"os/user" "os/user"
"strings" "strings"
"sync" "sync"
"syscall" "syscall"
"time"
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
engineUtil "github.com/purpleidea/mgmt/engine/util" engineUtil "github.com/purpleidea/mgmt/engine/util"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
) )
func init() { func init() {
@@ -43,22 +43,59 @@ func init() {
type ExecRes struct { type ExecRes struct {
traits.Base // add the base methods without re-implementation traits.Base // add the base methods without re-implementation
traits.Edgeable traits.Edgeable
traits.Sendable
init *engine.Init init *engine.Init
Cmd string `yaml:"cmd"` // the command to run // Cmd is the command to run. If this is not specified, we use the name.
Shell string `yaml:"shell"` // the (optional) shell to use to run the cmd Cmd string `yaml:"cmd"`
Timeout int `yaml:"timeout"` // the cmd timeout in seconds // Args is a list of args to pass to Cmd. This can be used *instead* of
WatchCmd string `yaml:"watchcmd"` // the watch command to run // passing the full command and args as a single string to Cmd. It can
WatchShell string `yaml:"watchshell"` // the (optional) shell to use to run the watch cmd // only be used when a Shell is *not* specified. The advantage of this
IfCmd string `yaml:"ifcmd"` // the if command to run // is that you don't have to worry about escape characters.
IfShell string `yaml:"ifshell"` // the (optional) shell to use to run the if cmd Args []string `yaml:"args"`
User string `yaml:"user"` // the (optional) user to use to execute the command // Cmd is the dir to run the command in. If empty, then this will use
Group string `yaml:"group"` // the (optional) group to use to execute the command // the working directory of the calling process. (This process is mgmt,
Output *string // all cmd output, read only, do not set! // not the process being run here.)
Stdout *string // the cmd stdout, read only, do not set! Cwd string `yaml:"cwd"`
Stderr *string // the cmd stderr, read only, do not set! // Shell is the (optional) shell to use to run the cmd. If you specify
// this, then you can't use the Args parameter.
Shell string `yaml:"shell"`
// Timeout is the number of seconds to wait before sending a Kill to the
// running command. If the Kill is received before the process exits,
// then this be treated as an error.
Timeout uint64 `yaml:"timeout"`
// Watch is the command to run to detect event changes. Each line of
// output from this command is treated as an event.
WatchCmd string `yaml:"watchcmd"`
// WatchCwd is the Cwd for the WatchCmd. See the docs for Cwd.
WatchCwd string `yaml:"watchcwd"`
// WatchShell is the Shell for the WatchCmd. See the docs for Shell.
WatchShell string `yaml:"watchshell"`
// IfCmd is the command that runs to guard against running the Cmd. If
// this command succeeds, then Cmd *will* be run. If this command
// returns a non-zero result, then the Cmd will not be run. Any error
// scenario or timeout will cause the resource to error.
IfCmd string `yaml:"ifcmd"`
// IfCwd is the Cwd for the IfCmd. See the docs for Cwd.
IfCwd string `yaml:"ifcwd"`
// IfShell is the Shell for the IfCmd. See the docs for Shell.
IfShell string `yaml:"ifshell"`
// User is the (optional) user to use to execute the command. It is used
// for any command being run.
User string `yaml:"user"`
// Group is the (optional) group to use to execute the command. It is
// used for any command being run.
Group string `yaml:"group"`
output *string // all cmd output, read only, do not set!
stdout *string // the cmd stdout, read only, do not set!
stderr *string // the cmd stderr, read only, do not set!
interruptChan chan struct{}
wg *sync.WaitGroup wg *sync.WaitGroup
} }
@@ -67,10 +104,27 @@ func (obj *ExecRes) Default() engine.Res {
return &ExecRes{} return &ExecRes{}
} }
// getCmd returns the actual command to run. When Cmd is not specified, we use
// the Name.
func (obj *ExecRes) getCmd() string {
if obj.Cmd != "" {
return obj.Cmd
}
return obj.Name()
}
// Validate if the params passed in are valid data. // Validate if the params passed in are valid data.
func (obj *ExecRes) Validate() error { func (obj *ExecRes) Validate() error {
if obj.Cmd == "" { // this is the only thing that is really required if obj.getCmd() == "" { // this is the only thing that is really required
return fmt.Errorf("command can't be empty") return fmt.Errorf("the Cmd can't be empty")
}
split := strings.Fields(obj.getCmd())
if len(obj.Args) > 0 && obj.Shell != "" {
return fmt.Errorf("the Args param can't be used with a Shell")
}
if len(obj.Args) > 0 && len(split) > 1 {
return fmt.Errorf("the Args param can't be used when Cmd has args")
} }
// check that, if an user or a group is set, we're running as root // check that, if an user or a group is set, we're running as root
@@ -80,7 +134,7 @@ func (obj *ExecRes) Validate() error {
return errwrap.Wrapf(err, "error looking up current user") return errwrap.Wrapf(err, "error looking up current user")
} }
if currentUser.Uid != "0" { if currentUser.Uid != "0" {
return errwrap.Errorf("running as root is required if you want to use exec with a different user/group") return fmt.Errorf("running as root is required if you want to use exec with a different user/group")
} }
} }
@@ -91,6 +145,7 @@ func (obj *ExecRes) Validate() error {
func (obj *ExecRes) Init(init *engine.Init) error { func (obj *ExecRes) Init(init *engine.Init) error {
obj.init = init // save for later obj.init = init // save for later
obj.interruptChan = make(chan struct{})
obj.wg = &sync.WaitGroup{} obj.wg = &sync.WaitGroup{}
return nil return nil
@@ -103,7 +158,7 @@ func (obj *ExecRes) Close() error {
// Watch is the primary listener for this resource and it outputs events. // Watch is the primary listener for this resource and it outputs events.
func (obj *ExecRes) Watch() error { func (obj *ExecRes) Watch() error {
ioChan := make(chan *bufioOutput) ioChan := make(chan *cmdOutput)
defer obj.wg.Wait() defer obj.wg.Wait()
if obj.WatchCmd != "" { if obj.WatchCmd != "" {
@@ -121,8 +176,11 @@ func (obj *ExecRes) Watch() error {
cmdName = obj.WatchShell // usually bash, or sh cmdName = obj.WatchShell // usually bash, or sh
cmdArgs = []string{"-c", obj.WatchCmd} cmdArgs = []string{"-c", obj.WatchCmd}
} }
cmd := exec.Command(cmdName, cmdArgs...)
//cmd.Dir = "" // look for program in pwd ? ctx, cancel := context.WithCancel(context.Background())
defer cancel()
cmd := exec.CommandContext(ctx, cmdName, cmdArgs...)
cmd.Dir = obj.WatchCwd // run program in pwd if ""
// ignore signals sent to parent process (we're in our own group) // ignore signals sent to parent process (we're in our own group)
cmd.SysProcAttr = &syscall.SysProcAttr{ cmd.SysProcAttr = &syscall.SysProcAttr{
Setpgid: true, Setpgid: true,
@@ -135,29 +193,12 @@ func (obj *ExecRes) Watch() error {
return errwrap.Wrapf(err, "error while setting credential") return errwrap.Wrapf(err, "error while setting credential")
} }
cmdReader, err := cmd.StdoutPipe() if ioChan, err = obj.cmdOutputRunner(ctx, cmd); err != nil {
if err != nil { return errwrap.Wrapf(err, "error starting WatchCmd")
return errwrap.Wrapf(err, "error creating StdoutPipe for Cmd")
} }
scanner := bufio.NewScanner(cmdReader)
defer cmd.Wait() // wait for the command to exit before return!
defer func() {
// FIXME: without wrapping this in this func it panic's
// when running certain graphs... why?
cmd.Process.Kill() // shutdown the Watch command on exit
}()
if err := cmd.Start(); err != nil {
return errwrap.Wrapf(err, "error starting Cmd")
} }
ioChan = obj.bufioChanScanner(scanner) obj.init.Running() // when started, notify engine that we're running
}
// notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? var send = false // send event?
for { for {
@@ -169,32 +210,44 @@ func (obj *ExecRes) Watch() error {
return fmt.Errorf("reached EOF") return fmt.Errorf("reached EOF")
} }
if err := data.err; err != nil { if err := data.err; err != nil {
// error reading input? // error reading input or cmd failure
exitErr, ok := err.(*exec.ExitError) // embeds an os.ProcessState
if !ok {
// command failed in some bad way
return errwrap.Wrapf(err, "unknown error") return errwrap.Wrapf(err, "unknown error")
} }
pStateSys := exitErr.Sys() // (*os.ProcessState) Sys
wStatus, ok := pStateSys.(syscall.WaitStatus)
if !ok {
return errwrap.Wrapf(err, "error running cmd")
}
exitStatus := wStatus.ExitStatus()
obj.init.Logf("watchcmd exited with: %d", exitStatus)
if exitStatus != 0 {
return errwrap.Wrapf(err, "unexpected exit status of zero")
}
return err // i'm not sure if this could happen
}
// each time we get a line of output, we loop! // each time we get a line of output, we loop!
obj.init.Logf("watch output: %s", data.text) if s := data.text; s == "" {
obj.init.Logf("watch output is empty!")
} else {
obj.init.Logf("watch output is:")
obj.init.Logf(s)
}
if data.text != "" { if data.text != "" {
send = true send = true
obj.init.Dirty() // dirty
} }
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
@@ -208,7 +261,6 @@ func (obj *ExecRes) CheckApply(apply bool) (bool, error) {
// have a chance to execute, and all without the check of obj.Refresh()! // have a chance to execute, and all without the check of obj.Refresh()!
if obj.IfCmd != "" { // if there is no onlyif check, we should just run if obj.IfCmd != "" { // if there is no onlyif check, we should just run
var cmdName string var cmdName string
var cmdArgs []string var cmdArgs []string
if obj.IfShell == "" { if obj.IfShell == "" {
@@ -224,6 +276,7 @@ func (obj *ExecRes) CheckApply(apply bool) (bool, error) {
cmdArgs = []string{"-c", obj.IfCmd} cmdArgs = []string{"-c", obj.IfCmd}
} }
cmd := exec.Command(cmdName, cmdArgs...) cmd := exec.Command(cmdName, cmdArgs...)
cmd.Dir = obj.IfCwd // run program in pwd if ""
// ignore signals sent to parent process (we're in our own group) // ignore signals sent to parent process (we're in our own group)
cmd.SysProcAttr = &syscall.SysProcAttr{ cmd.SysProcAttr = &syscall.SysProcAttr{
Setpgid: true, Setpgid: true,
@@ -236,11 +289,42 @@ func (obj *ExecRes) CheckApply(apply bool) (bool, error) {
return false, errwrap.Wrapf(err, "error while setting credential") return false, errwrap.Wrapf(err, "error while setting credential")
} }
var out splitWriter
out.Init()
cmd.Stdout = out.Stdout
cmd.Stderr = out.Stderr
if err := cmd.Run(); err != nil { if err := cmd.Run(); err != nil {
// TODO: check exit value exitErr, ok := err.(*exec.ExitError) // embeds an os.ProcessState
return true, nil // don't run if !ok {
// command failed in some bad way
return false, err
}
pStateSys := exitErr.Sys() // (*os.ProcessState) Sys
wStatus, ok := pStateSys.(syscall.WaitStatus)
if !ok {
return false, errwrap.Wrapf(err, "error running cmd")
}
exitStatus := wStatus.ExitStatus()
if exitStatus == 0 {
return false, fmt.Errorf("unexpected exit status of zero")
} }
obj.init.Logf("ifcmd exited with: %d", exitStatus)
if s := out.String(); s == "" {
obj.init.Logf("ifcmd output is empty!")
} else {
obj.init.Logf("ifcmd output is:")
obj.init.Logf(s)
}
return true, nil // don't run
}
if s := out.String(); s == "" {
obj.init.Logf("ifcmd output is empty!")
} else {
obj.init.Logf("ifcmd output is:")
obj.init.Logf(s)
}
} }
// state is not okay, no work done, exit, but without error // state is not okay, no work done, exit, but without error
@@ -256,17 +340,34 @@ func (obj *ExecRes) CheckApply(apply bool) (bool, error) {
// call without a shell // call without a shell
// FIXME: are there still whitespace splitting issues? // FIXME: are there still whitespace splitting issues?
// TODO: we could make the split character user selectable...! // TODO: we could make the split character user selectable...!
split := strings.Fields(obj.Cmd) split := strings.Fields(obj.getCmd())
cmdName = split[0] cmdName = split[0]
//d, _ := os.Getwd() // TODO: how does this ever error ? //d, _ := os.Getwd() // TODO: how does this ever error ?
//cmdName = path.Join(d, cmdName) //cmdName = path.Join(d, cmdName)
cmdArgs = split[1:] cmdArgs = split[1:]
if len(obj.Args) > 0 {
if len(split) != 1 { // should not happen
return false, fmt.Errorf("validation error")
}
cmdArgs = obj.Args
}
} else { } else {
cmdName = obj.Shell // usually bash, or sh cmdName = obj.Shell // usually bash, or sh
cmdArgs = []string{"-c", obj.Cmd} cmdArgs = []string{"-c", obj.getCmd()}
} }
cmd := exec.Command(cmdName, cmdArgs...)
//cmd.Dir = "" // look for program in pwd ? wg := &sync.WaitGroup{}
defer wg.Wait() // this must be above the defer cancel() call
var ctx context.Context
var cancel context.CancelFunc
if obj.Timeout > 0 { // cmd.Process.Kill() is called on timeout
ctx, cancel = context.WithTimeout(context.Background(), time.Duration(obj.Timeout)*time.Second)
} else { // zero timeout means no timer
ctx, cancel = context.WithCancel(context.Background())
}
defer cancel()
cmd := exec.CommandContext(ctx, cmdName, cmdArgs...)
cmd.Dir = obj.Cwd // run program in pwd if ""
// ignore signals sent to parent process (we're in our own group) // ignore signals sent to parent process (we're in our own group)
cmd.SysProcAttr = &syscall.SysProcAttr{ cmd.SysProcAttr = &syscall.SysProcAttr{
Setpgid: true, Setpgid: true,
@@ -290,35 +391,32 @@ func (obj *ExecRes) CheckApply(apply bool) (bool, error) {
return false, errwrap.Wrapf(err, "error starting cmd") return false, errwrap.Wrapf(err, "error starting cmd")
} }
timeout := obj.Timeout wg.Add(1)
if timeout == 0 { // zero timeout means no timer, so disable it go func() {
timeout = -1 defer wg.Done()
}
done := make(chan error)
go func() { done <- cmd.Wait() }()
select { select {
case e := <-done: case <-obj.interruptChan:
err = e // store cancel()
case <-ctx.Done():
case <-util.TimeAfterOrBlock(timeout): // let this exit
cmd.Process.Kill() // TODO: check error?
return false, fmt.Errorf("timeout for cmd")
} }
}()
err = cmd.Wait() // we can unblock this with the timeout
// save in memory for send/recv // save in memory for send/recv
// we use pointers to strings to indicate if used or not // we use pointers to strings to indicate if used or not
if out.Stdout.Activity || out.Stderr.Activity { if out.Stdout.Activity || out.Stderr.Activity {
str := out.String() str := out.String()
obj.Output = &str obj.output = &str
} }
if out.Stdout.Activity { if out.Stdout.Activity {
str := out.Stdout.String() str := out.Stdout.String()
obj.Stdout = &str obj.stdout = &str
} }
if out.Stderr.Activity { if out.Stderr.Activity {
str := out.Stderr.String() str := out.Stderr.String()
obj.Stderr = &str obj.stderr = &str
} }
// process the err result from cmd, we process non-zero exits here too! // process the err result from cmd, we process non-zero exits here too!
@@ -329,7 +427,18 @@ func (obj *ExecRes) CheckApply(apply bool) (bool, error) {
if !ok { if !ok {
return false, errwrap.Wrapf(err, "error running cmd") return false, errwrap.Wrapf(err, "error running cmd")
} }
return false, fmt.Errorf("cmd error, exit status: %d", wStatus.ExitStatus()) exitStatus := wStatus.ExitStatus()
if !wStatus.Signaled() { // not a timeout or cancel (no signal)
return false, errwrap.Wrapf(err, "cmd error, exit status: %d", exitStatus)
}
sig := wStatus.Signal()
// we get this on timeout, because ctx calls cmd.Process.Kill()
if sig == syscall.SIGKILL {
return false, errwrap.Wrapf(err, "cmd timeout, exit status: %d", exitStatus)
}
return false, fmt.Errorf("unknown cmd error, signal: %s, exit status: %d", sig, exitStatus)
} else if err != nil { } else if err != nil {
return false, errwrap.Wrapf(err, "general cmd error") return false, errwrap.Wrapf(err, "general cmd error")
@@ -339,11 +448,18 @@ func (obj *ExecRes) CheckApply(apply bool) (bool, error) {
// would be nice, but it would require terminal log output that doesn't // would be nice, but it would require terminal log output that doesn't
// interleave all the parallel parts which would mix it all up... // interleave all the parallel parts which would mix it all up...
if s := out.String(); s == "" { if s := out.String(); s == "" {
obj.init.Logf("Command output is empty!") obj.init.Logf("command output is empty!")
} else { } else {
obj.init.Logf("Command output is:") obj.init.Logf("command output is:")
obj.init.Logf(out.String()) obj.init.Logf(s)
}
if err := obj.init.Send(&ExecSends{
Output: obj.output,
Stdout: obj.stdout,
Stderr: obj.stderr,
}); err != nil {
return false, err
} }
// The state tracking is for exec resources that can't "detect" their // The state tracking is for exec resources that can't "detect" their
@@ -356,49 +472,67 @@ func (obj *ExecRes) CheckApply(apply bool) (bool, error) {
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *ExecRes) Cmp(r engine.Res) error { func (obj *ExecRes) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *ExecRes) Compare(r engine.Res) bool {
// we can only compare ExecRes to others of the same resource kind // we can only compare ExecRes to others of the same resource kind
res, ok := r.(*ExecRes) res, ok := r.(*ExecRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.Cmd != res.Cmd { if obj.Cmd != res.Cmd {
return false return fmt.Errorf("the Cmd differs")
}
if len(obj.Args) != len(res.Args) {
return fmt.Errorf("the Args differ")
}
for i, a := range obj.Args {
if a != res.Args[i] {
return fmt.Errorf("the Args differ at index: %d", i)
}
}
if obj.Cwd != res.Cwd {
return fmt.Errorf("the Cwd differs")
} }
if obj.Shell != res.Shell { if obj.Shell != res.Shell {
return false return fmt.Errorf("the Shell differs")
} }
if obj.Timeout != res.Timeout { if obj.Timeout != res.Timeout {
return false return fmt.Errorf("the Timeout differs")
}
if obj.WatchCmd != res.WatchCmd {
return false
}
if obj.WatchShell != res.WatchShell {
return false
}
if obj.IfCmd != res.IfCmd {
return false
}
if obj.IfShell != res.IfShell {
return false
}
if obj.User != res.User {
return false
}
if obj.Group != res.Group {
return false
} }
return true if obj.WatchCmd != res.WatchCmd {
return fmt.Errorf("the WatchCmd differs")
}
if obj.WatchCwd != res.WatchCwd {
return fmt.Errorf("the WatchCwd differs")
}
if obj.WatchShell != res.WatchShell {
return fmt.Errorf("the WatchShell differs")
}
if obj.IfCmd != res.IfCmd {
return fmt.Errorf("the IfCmd differs")
}
if obj.IfCwd != res.IfCwd {
return fmt.Errorf("the IfCwd differs")
}
if obj.IfShell != res.IfShell {
return fmt.Errorf("the IfShell differs")
}
if obj.User != res.User {
return fmt.Errorf("the User differs")
}
if obj.Group != res.Group {
return fmt.Errorf("the Group differs")
}
return nil
}
// Interrupt is called to ask the execution of this resource to end early.
func (obj *ExecRes) Interrupt() error {
close(obj.interruptChan)
return nil
} }
// ExecUID is the UID struct for ExecRes. // ExecUID is the UID struct for ExecRes.
@@ -449,13 +583,32 @@ func (obj *ExecRes) AutoEdges() (engine.AutoEdge, error) {
func (obj *ExecRes) UIDs() []engine.ResUID { func (obj *ExecRes) UIDs() []engine.ResUID {
x := &ExecUID{ x := &ExecUID{
BaseUID: engine.BaseUID{Name: obj.Name(), Kind: obj.Kind()}, BaseUID: engine.BaseUID{Name: obj.Name(), Kind: obj.Kind()},
Cmd: obj.Cmd, Cmd: obj.getCmd(),
IfCmd: obj.IfCmd, IfCmd: obj.IfCmd,
// TODO: add more params here // TODO: add more params here
} }
return []engine.ResUID{x} return []engine.ResUID{x}
} }
// ExecSends is the struct of data which is sent after a successful Apply.
type ExecSends struct {
// Output is the combined stdout and stderr of the command.
Output *string `lang:"output"`
// Stdout is the stdout of the command.
Stdout *string `lang:"stdout"`
// Stderr is the stderr of the command.
Stderr *string `lang:"stderr"`
}
// Sends represents the default struct of values we can send using Send/Recv.
func (obj *ExecRes) Sends() interface{} {
return &ExecSends{
Output: nil,
Stdout: nil,
Stderr: nil,
}
}
// UnmarshalYAML is the custom unmarshal handler for this struct. // UnmarshalYAML is the custom unmarshal handler for this struct.
// It is primarily useful for setting the defaults. // It is primarily useful for setting the defaults.
func (obj *ExecRes) UnmarshalYAML(unmarshal func(interface{}) error) error { func (obj *ExecRes) UnmarshalYAML(unmarshal func(interface{}) error) error {
@@ -512,7 +665,7 @@ func (obj *ExecRes) cmdFiles() []string {
var paths []string var paths []string
if obj.Shell != "" { if obj.Shell != "" {
paths = append(paths, obj.Shell) paths = append(paths, obj.Shell)
} else if cmdSplit := strings.Fields(obj.Cmd); len(cmdSplit) > 0 { } else if cmdSplit := strings.Fields(obj.getCmd()); len(cmdSplit) > 0 {
paths = append(paths, cmdSplit[0]) paths = append(paths, cmdSplit[0])
} }
if obj.WatchShell != "" { if obj.WatchShell != "" {
@@ -528,28 +681,56 @@ func (obj *ExecRes) cmdFiles() []string {
return paths return paths
} }
// bufioOutput is the output struct of the bufioChanScanner channel output. // cmdOutput is the output struct of the cmdOutputRunner channel output. You
type bufioOutput struct { // should always check the error first. If it's nil, then you can assume the
// text data is good to use.
type cmdOutput struct {
text string text string
err error err error
} }
// bufioChanScanner wraps the scanner output in a channel. // cmdOutputRunner wraps the Cmd in with a StdoutPipe scanner and reads for
func (obj *ExecRes) bufioChanScanner(scanner *bufio.Scanner) chan *bufioOutput { // errors. It runs Start and Wait, and errors runtime things in the channel.
ch := make(chan *bufioOutput) // If it can't start up the command, it will fail early. Once it's running, it
// will return the channel which can be used for the duration of the process.
// Cancelling the context merely unblocks the sending on the output channel, it
// does not Kill the cmd process. For that you must do it yourself elsewhere.
func (obj *ExecRes) cmdOutputRunner(ctx context.Context, cmd *exec.Cmd) (chan *cmdOutput, error) {
cmdReader, err := cmd.StdoutPipe()
if err != nil {
return nil, errwrap.Wrapf(err, "error creating StdoutPipe for Cmd")
}
scanner := bufio.NewScanner(cmdReader)
if err := cmd.Start(); err != nil {
return nil, errwrap.Wrapf(err, "error starting Cmd")
}
ch := make(chan *cmdOutput)
obj.wg.Add(1) obj.wg.Add(1)
go func() { go func() {
defer obj.wg.Done() defer obj.wg.Done()
defer close(ch) defer close(ch)
for scanner.Scan() { for scanner.Scan() {
ch <- &bufioOutput{text: scanner.Text()} // blocks here ? select {
case ch <- &cmdOutput{text: scanner.Text()}: // blocks here ?
case <-ctx.Done():
return
} }
}
// on EOF, scanner.Err() will be nil // on EOF, scanner.Err() will be nil
if err := scanner.Err(); err != nil { reterr := scanner.Err()
ch <- &bufioOutput{err: err} // send any misc errors we encounter reterr = errwrap.Append(reterr, cmd.Wait()) // always run Wait()
// send any misc errors we encounter on the channel
if reterr != nil {
select {
case ch <- &cmdOutput{err: reterr}:
case <-ctx.Done():
return
}
} }
}() }()
return ch return ch, nil
} }
// splitWriter mimics what the ssh.CombinedOutput command does, but stores the // splitWriter mimics what the ssh.CombinedOutput command does, but stores the

157
engine/resources/exec_test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -20,23 +20,34 @@
package resources package resources
import ( import (
"context"
"fmt"
"os/exec"
"syscall"
"testing" "testing"
"time"
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
) )
func fakeInit(t *testing.T) *engine.Init { func fakeExecInit(t *testing.T) (*engine.Init, *ExecSends) {
debug := testing.Verbose() // set via the -test.v flag to `go test` debug := testing.Verbose() // set via the -test.v flag to `go test`
logf := func(format string, v ...interface{}) { logf := func(format string, v ...interface{}) {
t.Logf("test: "+format, v...) t.Logf("test: "+format, v...)
} }
execSends := &ExecSends{}
return &engine.Init{ return &engine.Init{
Running: func() error { Send: func(st interface{}) error {
x, ok := st.(*ExecSends)
if !ok {
return fmt.Errorf("unable to send")
}
*execSends = *x // set
return nil return nil
}, },
Debug: debug, Debug: debug,
Logf: logf, Logf: logf,
} }, execSends
} }
func TestExecSendRecv1(t *testing.T) { func TestExecSendRecv1(t *testing.T) {
@@ -53,7 +64,8 @@ func TestExecSendRecv1(t *testing.T) {
t.Errorf("close failed with: %v", err) t.Errorf("close failed with: %v", err)
} }
}() }()
if err := r1.Init(fakeInit(t)); err != nil { init, execSends := fakeExecInit(t)
if err := r1.Init(init); err != nil {
t.Errorf("init failed with: %v", err) t.Errorf("init failed with: %v", err)
} }
// run artificially without the entire engine // run artificially without the entire engine
@@ -61,23 +73,23 @@ func TestExecSendRecv1(t *testing.T) {
t.Errorf("checkapply failed with: %v", err) t.Errorf("checkapply failed with: %v", err)
} }
t.Logf("output is: %v", r1.Output) t.Logf("output is: %v", execSends.Output)
if r1.Output != nil { if execSends.Output != nil {
t.Logf("output is: %v", *r1.Output) t.Logf("output is: %v", *execSends.Output)
} }
t.Logf("stdout is: %v", r1.Stdout) t.Logf("stdout is: %v", execSends.Stdout)
if r1.Stdout != nil { if execSends.Stdout != nil {
t.Logf("stdout is: %v", *r1.Stdout) t.Logf("stdout is: %v", *execSends.Stdout)
} }
t.Logf("stderr is: %v", r1.Stderr) t.Logf("stderr is: %v", execSends.Stderr)
if r1.Stderr != nil { if execSends.Stderr != nil {
t.Logf("stderr is: %v", *r1.Stderr) t.Logf("stderr is: %v", *execSends.Stderr)
} }
if r1.Stdout == nil { if execSends.Stdout == nil {
t.Errorf("stdout is nil") t.Errorf("stdout is nil")
} else { } else {
if out := *r1.Stdout; out != "hello world\n" { if out := *execSends.Stdout; out != "hello world\n" {
t.Errorf("got wrong stdout(%d): %s", len(out), out) t.Errorf("got wrong stdout(%d): %s", len(out), out)
} }
} }
@@ -97,7 +109,8 @@ func TestExecSendRecv2(t *testing.T) {
t.Errorf("close failed with: %v", err) t.Errorf("close failed with: %v", err)
} }
}() }()
if err := r1.Init(fakeInit(t)); err != nil { init, execSends := fakeExecInit(t)
if err := r1.Init(init); err != nil {
t.Errorf("init failed with: %v", err) t.Errorf("init failed with: %v", err)
} }
// run artificially without the entire engine // run artificially without the entire engine
@@ -105,23 +118,23 @@ func TestExecSendRecv2(t *testing.T) {
t.Errorf("checkapply failed with: %v", err) t.Errorf("checkapply failed with: %v", err)
} }
t.Logf("output is: %v", r1.Output) t.Logf("output is: %v", execSends.Output)
if r1.Output != nil { if execSends.Output != nil {
t.Logf("output is: %v", *r1.Output) t.Logf("output is: %v", *execSends.Output)
} }
t.Logf("stdout is: %v", r1.Stdout) t.Logf("stdout is: %v", execSends.Stdout)
if r1.Stdout != nil { if execSends.Stdout != nil {
t.Logf("stdout is: %v", *r1.Stdout) t.Logf("stdout is: %v", *execSends.Stdout)
} }
t.Logf("stderr is: %v", r1.Stderr) t.Logf("stderr is: %v", execSends.Stderr)
if r1.Stderr != nil { if execSends.Stderr != nil {
t.Logf("stderr is: %v", *r1.Stderr) t.Logf("stderr is: %v", *execSends.Stderr)
} }
if r1.Stderr == nil { if execSends.Stderr == nil {
t.Errorf("stderr is nil") t.Errorf("stderr is nil")
} else { } else {
if out := *r1.Stderr; out != "hello world\n" { if out := *execSends.Stderr; out != "hello world\n" {
t.Errorf("got wrong stderr(%d): %s", len(out), out) t.Errorf("got wrong stderr(%d): %s", len(out), out)
} }
} }
@@ -141,7 +154,8 @@ func TestExecSendRecv3(t *testing.T) {
t.Errorf("close failed with: %v", err) t.Errorf("close failed with: %v", err)
} }
}() }()
if err := r1.Init(fakeInit(t)); err != nil { init, execSends := fakeExecInit(t)
if err := r1.Init(init); err != nil {
t.Errorf("init failed with: %v", err) t.Errorf("init failed with: %v", err)
} }
// run artificially without the entire engine // run artificially without the entire engine
@@ -149,42 +163,97 @@ func TestExecSendRecv3(t *testing.T) {
t.Errorf("checkapply failed with: %v", err) t.Errorf("checkapply failed with: %v", err)
} }
t.Logf("output is: %v", r1.Output) t.Logf("output is: %v", execSends.Output)
if r1.Output != nil { if execSends.Output != nil {
t.Logf("output is: %v", *r1.Output) t.Logf("output is: %v", *execSends.Output)
} }
t.Logf("stdout is: %v", r1.Stdout) t.Logf("stdout is: %v", execSends.Stdout)
if r1.Stdout != nil { if execSends.Stdout != nil {
t.Logf("stdout is: %v", *r1.Stdout) t.Logf("stdout is: %v", *execSends.Stdout)
} }
t.Logf("stderr is: %v", r1.Stderr) t.Logf("stderr is: %v", execSends.Stderr)
if r1.Stderr != nil { if execSends.Stderr != nil {
t.Logf("stderr is: %v", *r1.Stderr) t.Logf("stderr is: %v", *execSends.Stderr)
} }
if r1.Output == nil { if execSends.Output == nil {
t.Errorf("output is nil") t.Errorf("output is nil")
} else { } else {
// it looks like bash or golang race to the write, so whichever // it looks like bash or golang race to the write, so whichever
// order they come out in is ok, as long as they come out whole // order they come out in is ok, as long as they come out whole
if out := *r1.Output; out != "hello world\ngoodbye world\n" && out != "goodbye world\nhello world\n" { if out := *execSends.Output; out != "hello world\ngoodbye world\n" && out != "goodbye world\nhello world\n" {
t.Errorf("got wrong output(%d): %s", len(out), out) t.Errorf("got wrong output(%d): %s", len(out), out)
} }
} }
if r1.Stdout == nil { if execSends.Stdout == nil {
t.Errorf("stdout is nil") t.Errorf("stdout is nil")
} else { } else {
if out := *r1.Stdout; out != "hello world\n" { if out := *execSends.Stdout; out != "hello world\n" {
t.Errorf("got wrong stdout(%d): %s", len(out), out) t.Errorf("got wrong stdout(%d): %s", len(out), out)
} }
} }
if r1.Stderr == nil { if execSends.Stderr == nil {
t.Errorf("stderr is nil") t.Errorf("stderr is nil")
} else { } else {
if out := *r1.Stderr; out != "goodbye world\n" { if out := *execSends.Stderr; out != "goodbye world\n" {
t.Errorf("got wrong stderr(%d): %s", len(out), out) t.Errorf("got wrong stderr(%d): %s", len(out), out)
} }
} }
} }
func TestExecTimeoutBehaviour(t *testing.T) {
// cmd.Process.Kill() is called on timeout
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
cmdName := "/bin/sleep" // it's /usr/bin/sleep on modern distros
cmdArgs := []string{"300"} // 5 min in seconds
cmd := exec.CommandContext(ctx, cmdName, cmdArgs...)
// ignore signals sent to parent process (we're in our own group)
cmd.SysProcAttr = &syscall.SysProcAttr{
Setpgid: true,
Pgid: 0,
}
if err := cmd.Start(); err != nil {
t.Errorf("error starting cmd: %+v", err)
return
}
err := cmd.Wait() // we can unblock this with the timeout
if err == nil {
t.Errorf("expected error, got nil")
return
}
exitErr, ok := err.(*exec.ExitError) // embeds an os.ProcessState
if err != nil && ok {
pStateSys := exitErr.Sys() // (*os.ProcessState) Sys
wStatus, ok := pStateSys.(syscall.WaitStatus)
if !ok {
t.Errorf("error running cmd")
return
}
if !wStatus.Signaled() {
t.Errorf("did not get signal, exit status: %d", wStatus.ExitStatus())
return
}
// we get this on timeout, because ctx calls cmd.Process.Kill()
if sig := wStatus.Signal(); sig != syscall.SIGKILL {
t.Errorf("got wrong signal: %+v, exit status: %d", sig, wStatus.ExitStatus())
return
}
t.Logf("exit status: %d", wStatus.ExitStatus())
return
} else if err != nil {
t.Errorf("general cmd error")
return
}
// no error
}

767
engine/resources/file.go
View File

File diff suppressed because it is too large Load Diff

127
engine/resources/file_test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -78,7 +78,7 @@ func TestMiscEncodeDecode1(t *testing.T) {
e := gob.NewEncoder(&b1) e := gob.NewEncoder(&b1)
err = e.Encode(&input) // pass with & err = e.Encode(&input) // pass with &
if err != nil { if err != nil {
t.Errorf("Gob failed to Encode: %v", err) t.Errorf("gob failed to Encode: %v", err)
} }
str := base64.StdEncoding.EncodeToString(b1.Bytes()) str := base64.StdEncoding.EncodeToString(b1.Bytes())
@@ -86,27 +86,27 @@ func TestMiscEncodeDecode1(t *testing.T) {
var output interface{} var output interface{}
bb, err := base64.StdEncoding.DecodeString(str) bb, err := base64.StdEncoding.DecodeString(str)
if err != nil { if err != nil {
t.Errorf("Base64 failed to Decode: %v", err) t.Errorf("base64 failed to Decode: %v", err)
} }
b2 := bytes.NewBuffer(bb) b2 := bytes.NewBuffer(bb)
d := gob.NewDecoder(b2) d := gob.NewDecoder(b2)
err = d.Decode(&output) // pass with & err = d.Decode(&output) // pass with &
if err != nil { if err != nil {
t.Errorf("Gob failed to Decode: %v", err) t.Errorf("gob failed to Decode: %v", err)
} }
res1, ok := input.(engine.Res) res1, ok := input.(engine.Res)
if !ok { if !ok {
t.Errorf("Input %v is not a Res", res1) t.Errorf("input %v is not a Res", res1)
return return
} }
res2, ok := output.(engine.Res) res2, ok := output.(engine.Res)
if !ok { if !ok {
t.Errorf("Output %v is not a Res", res2) t.Errorf("output %v is not a Res", res2)
return return
} }
if err := res1.Cmp(res2); err != nil { if err := res1.Cmp(res2); err != nil {
t.Errorf("The input and output Res values do not match: %+v", err) t.Errorf("the input and output Res values do not match: %+v", err)
} }
} }
@@ -116,34 +116,135 @@ func TestMiscEncodeDecode2(t *testing.T) {
// encode // encode
input, err := engine.NewNamedResource("file", "file1") input, err := engine.NewNamedResource("file", "file1")
if err != nil { if err != nil {
t.Errorf("Can't create: %v", err) t.Errorf("can't create: %v", err)
return return
} }
// NOTE: Do not add this bit of code, because it would cause the path to
// get taken from the actual Path parameter, instead of using the name,
// and if we use the name, the Cmp function will detect if the name is
// stored properly or not.
//fileRes := input.(*FileRes) // must not panic
//fileRes.Path = "/tmp/whatever"
b64, err := engineUtil.ResToB64(input) b64, err := engineUtil.ResToB64(input)
if err != nil { if err != nil {
t.Errorf("Can't encode: %v", err) t.Errorf("can't encode: %v", err)
return return
} }
output, err := engineUtil.B64ToRes(b64) output, err := engineUtil.B64ToRes(b64)
if err != nil { if err != nil {
t.Errorf("Can't decode: %v", err) t.Errorf("can't decode: %v", err)
return return
} }
res1, ok := input.(engine.Res) res1, ok := input.(engine.Res)
if !ok { if !ok {
t.Errorf("Input %v is not a Res", res1) t.Errorf("input %v is not a Res", res1)
return return
} }
res2, ok := output.(engine.Res) res2, ok := output.(engine.Res)
if !ok { if !ok {
t.Errorf("Output %v is not a Res", res2) t.Errorf("output %v is not a Res", res2)
return return
} }
// this uses the standalone file cmp function
if err := res1.Cmp(res2); err != nil { if err := res1.Cmp(res2); err != nil {
t.Errorf("The input and output Res values do not match: %+v", err) t.Errorf("the input and output Res values do not match: %+v", err)
}
}
func TestMiscEncodeDecode3(t *testing.T) {
var err error
// encode
input, err := engine.NewNamedResource("file", "file1")
if err != nil {
t.Errorf("can't create: %v", err)
return
}
fileRes := input.(*FileRes) // must not panic
fileRes.Path = "/tmp/whatever"
// TODO: add other params/traits/etc here!
b64, err := engineUtil.ResToB64(input)
if err != nil {
t.Errorf("can't encode: %v", err)
return
}
output, err := engineUtil.B64ToRes(b64)
if err != nil {
t.Errorf("can't decode: %v", err)
return
}
res1, ok := input.(engine.Res)
if !ok {
t.Errorf("input %v is not a Res", res1)
return
}
res2, ok := output.(engine.Res)
if !ok {
t.Errorf("output %v is not a Res", res2)
return
}
// this uses the more complete, engine cmp function
if err := engine.ResCmp(res1, res2); err != nil {
t.Errorf("the input and output Res values do not match: %+v", err)
}
}
func TestMiscEncodeDecode4(t *testing.T) {
var err error
const (
Kind = "file"
Name = "file1"
)
// encode
input, err := engine.NewNamedResource(Kind, Name)
if err != nil {
t.Errorf("can't create: %v", err)
return
}
fileRes := input.(*FileRes) // must not panic
fileRes.Path = "/tmp/whatever"
// TODO: add other params/traits/etc here!
b64, err := engineUtil.ResToB64(input)
if err != nil {
t.Errorf("can't encode: %v", err)
return
}
output, err := engineUtil.B64ToRes(b64)
if err != nil {
t.Errorf("can't decode: %v", err)
return
}
res1, ok := input.(engine.Res)
if !ok {
t.Errorf("input %v is not a Res", res1)
return
}
res2, ok := output.(engine.Res)
if !ok {
t.Errorf("output %v is not a Res", res2)
return
}
// this uses the more complete, engine cmp function
if err := engine.ResCmp(res1, res2); err != nil {
t.Errorf("the input and output Res values do not match: %+v", err)
}
// ensure the kind and name are correctly decoded too!
if kind := res2.Kind(); kind != Kind {
t.Errorf("the output kind was `%s`, expected `%s`", kind, Kind)
}
if name := res2.Name(); name != Name {
t.Errorf("the output name was `%s`, expected `%s`", name, Name)
} }
} }

44
engine/resources/group.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -28,8 +28,7 @@ import (
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
"github.com/purpleidea/mgmt/recwatch" "github.com/purpleidea/mgmt/recwatch"
"github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
) )
func init() { func init() {
@@ -59,7 +58,7 @@ func (obj *GroupRes) Default() engine.Res {
// Validate if the params passed in are valid data. // Validate if the params passed in are valid data.
func (obj *GroupRes) Validate() error { func (obj *GroupRes) Validate() error {
if obj.State != "exists" && obj.State != "absent" { if obj.State != "exists" && obj.State != "absent" {
return fmt.Errorf("State must be 'exists' or 'absent'") return fmt.Errorf("state must be 'exists' or 'absent'")
} }
return nil return nil
} }
@@ -85,10 +84,7 @@ func (obj *GroupRes) Watch() error {
} }
defer obj.recWatcher.Close() defer obj.recWatcher.Close()
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? var send = false // send event?
for { for {
@@ -108,29 +104,21 @@ func (obj *GroupRes) Watch() error {
obj.init.Logf("Event(%s): %v", event.Body.Name, event.Body.Op) obj.init.Logf("Event(%s): %v", event.Body.Name, event.Body.Op)
} }
send = true send = true
obj.init.Dirty() // dirty
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
// CheckApply method for Group resource. // CheckApply method for Group resource.
func (obj *GroupRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *GroupRes) CheckApply(apply bool) (bool, error) {
obj.init.Logf("CheckApply(%t)", apply) obj.init.Logf("CheckApply(%t)", apply)
// check if the group exists // check if the group exists
@@ -232,32 +220,24 @@ func (obj *GroupRes) CheckApply(apply bool) (checkOK bool, err error) {
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *GroupRes) Cmp(r engine.Res) error { func (obj *GroupRes) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *GroupRes) Compare(r engine.Res) bool {
// we can only compare GroupRes to others of the same resource kind // we can only compare GroupRes to others of the same resource kind
res, ok := r.(*GroupRes) res, ok := r.(*GroupRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.State != res.State { if obj.State != res.State {
return false return fmt.Errorf("the State differs")
} }
if (obj.GID == nil) != (res.GID == nil) { if (obj.GID == nil) != (res.GID == nil) {
return false return fmt.Errorf("the GID differs")
} }
if obj.GID != nil && res.GID != nil { if obj.GID != nil && res.GID != nil {
if *obj.GID != *res.GID { if *obj.GID != *res.GID {
return false return fmt.Errorf("the GID differs")
} }
} }
return true return nil
} }
// GroupUID is the UID struct for GroupRes. // GroupUID is the UID struct for GroupRes.

55
engine/resources/hostname.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -25,9 +25,9 @@ import (
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
engineUtil "github.com/purpleidea/mgmt/engine/util" engineUtil "github.com/purpleidea/mgmt/engine/util"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
"github.com/godbus/dbus" "github.com/godbus/dbus"
errwrap "github.com/pkg/errors"
) )
func init() { func init() {
@@ -110,7 +110,7 @@ func (obj *HostnameRes) Watch() error {
// if we share the bus with others, we will get each others messages!! // if we share the bus with others, we will get each others messages!!
bus, err := util.SystemBusPrivateUsable() // don't share the bus connection! bus, err := util.SystemBusPrivateUsable() // don't share the bus connection!
if err != nil { if err != nil {
return errwrap.Wrap(err, "Failed to connect to bus") return errwrap.Wrapf(err, "failed to connect to bus")
} }
defer bus.Close() defer bus.Close()
// watch the PropertiesChanged signal on the hostname1 dbus path // watch the PropertiesChanged signal on the hostname1 dbus path
@@ -120,56 +120,45 @@ func (obj *HostnameRes) Watch() error {
dbusPropertiesIface, dbusPropertiesIface,
) )
if call := bus.BusObject().Call(engineUtil.DBusAddMatch, 0, args); call.Err != nil { if call := bus.BusObject().Call(engineUtil.DBusAddMatch, 0, args); call.Err != nil {
return errwrap.Wrap(call.Err, "Failed to subscribe to DBus events for hostname1") return errwrap.Wrapf(call.Err, "failed to subscribe to DBus events for hostname1")
} }
defer bus.BusObject().Call(engineUtil.DBusRemoveMatch, 0, args) // ignore the error defer bus.BusObject().Call(engineUtil.DBusRemoveMatch, 0, args) // ignore the error
signals := make(chan *dbus.Signal, 10) // closed by dbus package signals := make(chan *dbus.Signal, 10) // closed by dbus package
bus.Signal(signals) bus.Signal(signals)
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? var send = false // send event?
for { for {
select { select {
case <-signals: case <-signals:
send = true send = true
obj.init.Dirty() // dirty
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
func (obj *HostnameRes) updateHostnameProperty(object dbus.BusObject, expectedValue, property, setterName string, apply bool) (checkOK bool, err error) { func (obj *HostnameRes) updateHostnameProperty(object dbus.BusObject, expectedValue, property, setterName string, apply bool) (bool, error) {
propertyObject, err := object.GetProperty("org.freedesktop.hostname1." + property) propertyObject, err := object.GetProperty("org.freedesktop.hostname1." + property)
if err != nil { if err != nil {
return false, errwrap.Wrapf(err, "failed to get org.freedesktop.hostname1.%s", property) return false, errwrap.Wrapf(err, "failed to get org.freedesktop.hostname1.%s", property)
} }
if propertyObject.Value() == nil { if propertyObject.Value() == nil {
return false, errwrap.Errorf("Unexpected nil value received when reading property %s", property) return false, fmt.Errorf("unexpected nil value received when reading property %s", property)
} }
propertyValue, ok := propertyObject.Value().(string) propertyValue, ok := propertyObject.Value().(string)
if !ok { if !ok {
return false, fmt.Errorf("Received unexpected type as %s value, expected string got '%T'", property, propertyValue) return false, fmt.Errorf("received unexpected type as %s value, expected string got '%T'", property, propertyValue)
} }
// expected value and actual value match => checkOk // expected value and actual value match => checkOk
@@ -193,16 +182,16 @@ func (obj *HostnameRes) updateHostnameProperty(object dbus.BusObject, expectedVa
} }
// CheckApply method for Hostname resource. // CheckApply method for Hostname resource.
func (obj *HostnameRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *HostnameRes) CheckApply(apply bool) (bool, error) {
conn, err := util.SystemBusPrivateUsable() conn, err := util.SystemBusPrivateUsable()
if err != nil { if err != nil {
return false, errwrap.Wrap(err, "Failed to connect to the private system bus") return false, errwrap.Wrapf(err, "failed to connect to the private system bus")
} }
defer conn.Close() defer conn.Close()
hostnameObject := conn.Object(hostname1Iface, hostname1Path) hostnameObject := conn.Object(hostname1Iface, hostname1Path)
checkOK = true checkOK := true
if obj.PrettyHostname != "" { if obj.PrettyHostname != "" {
propertyCheckOK, err := obj.updateHostnameProperty(hostnameObject, obj.PrettyHostname, "PrettyHostname", "SetPrettyHostname", apply) propertyCheckOK, err := obj.updateHostnameProperty(hostnameObject, obj.PrettyHostname, "PrettyHostname", "SetPrettyHostname", apply)
if err != nil { if err != nil {
@@ -230,31 +219,23 @@ func (obj *HostnameRes) CheckApply(apply bool) (checkOK bool, err error) {
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *HostnameRes) Cmp(r engine.Res) error { func (obj *HostnameRes) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *HostnameRes) Compare(r engine.Res) bool {
// we can only compare HostnameRes to others of the same resource kind // we can only compare HostnameRes to others of the same resource kind
res, ok := r.(*HostnameRes) res, ok := r.(*HostnameRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.PrettyHostname != res.PrettyHostname { if obj.PrettyHostname != res.PrettyHostname {
return false return fmt.Errorf("the PrettyHostname differs")
} }
if obj.StaticHostname != res.StaticHostname { if obj.StaticHostname != res.StaticHostname {
return false return fmt.Errorf("the StaticHostname differs")
} }
if obj.TransientHostname != res.TransientHostname { if obj.TransientHostname != res.TransientHostname {
return false return fmt.Errorf("the TransientHostname differs")
} }
return true return nil
} }
// HostnameUID is the UID struct for HostnameRes. // HostnameUID is the UID struct for HostnameRes.

126
engine/resources/kv.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -18,13 +18,16 @@
package resources package resources
import ( import (
"context"
"fmt" "fmt"
"strconv" "strconv"
"sync"
"time"
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
"github.com/purpleidea/mgmt/util"
errwrap "github.com/pkg/errors" "github.com/purpleidea/mgmt/util/errwrap"
) )
func init() { func init() {
@@ -40,6 +43,10 @@ const (
SkipCmpStyleString SkipCmpStyleString
) )
const (
kvCheckApplyTimeout = 5 * time.Second
)
// KVRes is a resource which writes a key/value pair into cluster wide storage. // KVRes is a resource which writes a key/value pair into cluster wide storage.
// It will ensure that the key is set to the requested value. The one exception // It will ensure that the key is set to the requested value. The one exception
// is that if you use the SkipLessThan parameter, then it will only replace the // is that if you use the SkipLessThan parameter, then it will only replace the
@@ -56,14 +63,32 @@ type KVRes struct {
init *engine.Init init *engine.Init
// XXX: shouldn't the name be the key? // Key represents the key to set. If it is not specified, the Name value
Key string `yaml:"key"` // key to set // is used instead.
Value *string `yaml:"value"` // value to set (nil to delete) Key string `lang:"key" yaml:"key"`
SkipLessThan bool `yaml:"skiplessthan"` // skip updates as long as stored value is greater // Value represents the string value to set. If this value is nil or,
SkipCmpStyle KVResSkipCmpStyle `yaml:"skipcmpstyle"` // how to do the less than cmp // undefined, then this will delete that key.
Value *string `lang:"value" yaml:"value"`
// SkipLessThan causes the value to be updated as long as it is greater.
SkipLessThan bool `lang:"skiplessthan" yaml:"skiplessthan"`
// SkipCmpStyle is the type of compare function used when determining if
// the value is greater when using the SkipLessThan parameter.
SkipCmpStyle KVResSkipCmpStyle `lang:"skipcmpstyle" yaml:"skipcmpstyle"`
interruptChan chan struct{}
// TODO: does it make sense to have different backends here? (eg: local) // TODO: does it make sense to have different backends here? (eg: local)
} }
// getKey returns the key to be used for this resource. If the Key field is
// specified, it will use that, otherwise it uses the Name.
func (obj *KVRes) getKey() string {
if obj.Key != "" {
return obj.Key
}
return obj.Name()
}
// Default returns some sensible defaults for this resource. // Default returns some sensible defaults for this resource.
func (obj *KVRes) Default() engine.Res { func (obj *KVRes) Default() engine.Res {
return &KVRes{} return &KVRes{}
@@ -71,7 +96,7 @@ func (obj *KVRes) Default() engine.Res {
// Validate if the params passed in are valid data. // Validate if the params passed in are valid data.
func (obj *KVRes) Validate() error { func (obj *KVRes) Validate() error {
if obj.Key == "" { if obj.getKey() == "" {
return fmt.Errorf("key must not be empty") return fmt.Errorf("key must not be empty")
} }
if obj.SkipLessThan { if obj.SkipLessThan {
@@ -92,6 +117,8 @@ func (obj *KVRes) Validate() error {
func (obj *KVRes) Init(init *engine.Init) error { func (obj *KVRes) Init(init *engine.Init) error {
obj.init = init // save for later obj.init = init // save for later
obj.interruptChan = make(chan struct{})
return nil return nil
} }
@@ -102,13 +129,17 @@ func (obj *KVRes) Close() error {
// Watch is the primary listener for this resource and it outputs events. // Watch is the primary listener for this resource and it outputs events.
func (obj *KVRes) Watch() error { func (obj *KVRes) Watch() error {
// FIXME: add timeout to context
// The obj.init.Done channel is closed by the engine to signal shutdown.
ctx, cancel := util.ContextWithCloser(context.Background(), obj.init.Done)
defer cancel()
// notify engine that we're running ch, err := obj.init.World.StrMapWatch(ctx, obj.getKey()) // get possible events!
if err := obj.init.Running(); err != nil { if err != nil {
return err // exit if requested return err
} }
ch := obj.init.World.StrMapWatch(obj.Key) // get possible events! obj.init.Running() // when started, notify engine that we're running
var send = false // send event? var send = false // send event?
for { for {
@@ -122,32 +153,24 @@ func (obj *KVRes) Watch() error {
return errwrap.Wrapf(err, "unknown %s watcher error", obj) return errwrap.Wrapf(err, "unknown %s watcher error", obj)
} }
if obj.init.Debug { if obj.init.Debug {
obj.init.Logf("Event!") obj.init.Logf("event!")
} }
send = true send = true
obj.init.Dirty() // dirty
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
// lessThanCheck checks for less than validity. // lessThanCheck checks for less than validity.
func (obj *KVRes) lessThanCheck(value string) (checkOK bool, err error) { func (obj *KVRes) lessThanCheck(value string) (bool, error) {
v := *obj.Value v := *obj.Value
if value == v { // redundant check for safety if value == v { // redundant check for safety
return true, nil return true, nil
@@ -185,16 +208,31 @@ func (obj *KVRes) lessThanCheck(value string) (checkOK bool, err error) {
} }
// CheckApply method for Password resource. Does nothing, returns happy! // CheckApply method for Password resource. Does nothing, returns happy!
func (obj *KVRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *KVRes) CheckApply(apply bool) (bool, error) {
obj.init.Logf("CheckApply(%t)", apply) obj.init.Logf("CheckApply(%t)", apply)
wg := &sync.WaitGroup{}
defer wg.Wait() // this must be above the defer cancel() call
ctx, cancel := context.WithTimeout(context.Background(), kvCheckApplyTimeout)
defer cancel()
wg.Add(1)
go func() {
defer wg.Done()
select {
case <-obj.interruptChan:
cancel()
case <-ctx.Done():
// let this exit
}
}()
if val, exists := obj.init.Recv()["Value"]; exists && val.Changed { if val, exists := obj.init.Recv()["Value"]; exists && val.Changed {
// if we received on Value, and it changed, wooo, nothing to do. // if we received on Value, and it changed, wooo, nothing to do.
obj.init.Logf("CheckApply: `Value` was updated!") obj.init.Logf("CheckApply: `Value` was updated!")
} }
hostname := obj.init.Hostname // me hostname := obj.init.Hostname // me
keyMap, err := obj.init.World.StrMapGet(obj.Key) keyMap, err := obj.init.World.StrMapGet(ctx, obj.getKey())
if err != nil { if err != nil {
return false, errwrap.Wrapf(err, "check error during StrGet") return false, errwrap.Wrapf(err, "check error during StrGet")
} }
@@ -214,7 +252,7 @@ func (obj *KVRes) CheckApply(apply bool) (checkOK bool, err error) {
return true, nil // nothing to delete, we're good! return true, nil // nothing to delete, we're good!
} else if ok && obj.Value == nil { // delete } else if ok && obj.Value == nil { // delete
err := obj.init.World.StrMapDel(obj.Key) err := obj.init.World.StrMapDel(ctx, obj.getKey())
return false, errwrap.Wrapf(err, "apply error during StrDel") return false, errwrap.Wrapf(err, "apply error during StrDel")
} }
@@ -222,7 +260,7 @@ func (obj *KVRes) CheckApply(apply bool) (checkOK bool, err error) {
return false, nil return false, nil
} }
if err := obj.init.World.StrMapSet(obj.Key, *obj.Value); err != nil { if err := obj.init.World.StrMapSet(ctx, obj.getKey(), *obj.Value); err != nil {
return false, errwrap.Wrapf(err, "apply error during StrSet") return false, errwrap.Wrapf(err, "apply error during StrSet")
} }
@@ -231,39 +269,37 @@ func (obj *KVRes) CheckApply(apply bool) (checkOK bool, err error) {
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *KVRes) Cmp(r engine.Res) error { func (obj *KVRes) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *KVRes) Compare(r engine.Res) bool {
// we can only compare KVRes to others of the same resource kind // we can only compare KVRes to others of the same resource kind
res, ok := r.(*KVRes) res, ok := r.(*KVRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.Key != res.Key { if obj.getKey() != res.getKey() {
return false return fmt.Errorf("the Key differs")
} }
if (obj.Value == nil) != (res.Value == nil) { // xor if (obj.Value == nil) != (res.Value == nil) { // xor
return false return fmt.Errorf("the Value differs")
} }
if obj.Value != nil && res.Value != nil { if obj.Value != nil && res.Value != nil {
if *obj.Value != *res.Value { // compare the strings if *obj.Value != *res.Value { // compare the strings
return false return fmt.Errorf("the contents of Value differs")
} }
} }
if obj.SkipLessThan != res.SkipLessThan { if obj.SkipLessThan != res.SkipLessThan {
return false return fmt.Errorf("the SkipLessThan param differs")
} }
if obj.SkipCmpStyle != res.SkipCmpStyle { if obj.SkipCmpStyle != res.SkipCmpStyle {
return false return fmt.Errorf("the SkipCmpStyle param differs")
} }
return true return nil
}
// Interrupt is called to ask the execution of this resource to end early.
func (obj *KVRes) Interrupt() error {
close(obj.interruptChan)
return nil
} }
// KVUID is the UID struct for KVRes. // KVUID is the UID struct for KVRes.

42
engine/resources/mount.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -33,13 +33,13 @@ import (
engineUtil "github.com/purpleidea/mgmt/engine/util" engineUtil "github.com/purpleidea/mgmt/engine/util"
"github.com/purpleidea/mgmt/recwatch" "github.com/purpleidea/mgmt/recwatch"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
sdbus "github.com/coreos/go-systemd/dbus" sdbus "github.com/coreos/go-systemd/dbus"
"github.com/coreos/go-systemd/unit" "github.com/coreos/go-systemd/unit"
systemdUtil "github.com/coreos/go-systemd/util" systemdUtil "github.com/coreos/go-systemd/util"
fstab "github.com/deniswernert/go-fstab" fstab "github.com/deniswernert/go-fstab"
"github.com/godbus/dbus" "github.com/godbus/dbus"
errwrap "github.com/pkg/errors"
"golang.org/x/sys/unix" "golang.org/x/sys/unix"
) )
@@ -75,6 +75,8 @@ const (
// diskByLabel is the location of symlinks for partitions by label. // diskByLabel is the location of symlinks for partitions by label.
diskByPartLabel = devDisk + "by-partlabel/" diskByPartLabel = devDisk + "by-partlabel/"
// dbusSystemdService is the service to connect to systemd itself.
dbusSystemd1Service = "org.freedesktop.systemd1"
// dbusSystemd1Interface is the base systemd1 path. // dbusSystemd1Interface is the base systemd1 path.
dbusSystemd1Path = "/org/freedesktop/systemd1" dbusSystemd1Path = "/org/freedesktop/systemd1"
// dbusUnitPath is the dbus path where mount unit files are found. // dbusUnitPath is the dbus path where mount unit files are found.
@@ -88,6 +90,9 @@ const (
dbusManagerInterface = dbusSystemd1Interface + ".Manager" dbusManagerInterface = dbusSystemd1Interface + ".Manager"
// dbusRestartUnit is the dbus method for restarting systemd units. // dbusRestartUnit is the dbus method for restarting systemd units.
dbusRestartUnit = dbusManagerInterface + ".RestartUnit" dbusRestartUnit = dbusManagerInterface + ".RestartUnit"
// dbusReloadSystemd is the dbus method for reloading systemd settings.
// (i.e. systemctl daemon-reload)
dbusReloadSystemd = dbusManagerInterface + ".Reload"
// restartTimeout is the delay before restartUnit is assumed to have // restartTimeout is the delay before restartUnit is assumed to have
// failed. // failed.
dbusRestartCtxTimeout = 10 dbusRestartCtxTimeout = 10
@@ -224,10 +229,7 @@ func (obj *MountRes) Watch() error {
// close the recwatcher when we're done // close the recwatcher when we're done
defer recWatcher.Close() defer recWatcher.Close()
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // bubble up a NACK...
}
var send bool var send bool
var done bool var done bool
@@ -248,7 +250,6 @@ func (obj *MountRes) Watch() error {
obj.init.Logf("event(%s): %v", event.Body.Name, event.Body.Op) obj.init.Logf("event(%s): %v", event.Body.Name, event.Body.Op)
} }
obj.init.Dirty()
send = true send = true
case event, ok := <-ch: case event, ok := <-ch:
@@ -263,31 +264,23 @@ func (obj *MountRes) Watch() error {
obj.init.Logf("event: %+v", event) obj.init.Logf("event: %+v", event)
} }
obj.init.Dirty()
send = true send = true
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
// fstabCheckApply checks /etc/fstab for entries corresponding to the resource // fstabCheckApply checks /etc/fstab for entries corresponding to the resource
// definition, and adds or deletes the entry as needed. // definition, and adds or deletes the entry as needed.
func (obj *MountRes) fstabCheckApply(apply bool) (checkOK bool, err error) { func (obj *MountRes) fstabCheckApply(apply bool) (bool, error) {
exists, err := fstabEntryExists(fstabPath, obj.mount) exists, err := fstabEntryExists(fstabPath, obj.mount)
if err != nil { if err != nil {
return false, errwrap.Wrapf(err, "error checking if fstab entry exists") return false, errwrap.Wrapf(err, "error checking if fstab entry exists")
@@ -351,8 +344,8 @@ func (obj *MountRes) mountCheckApply(apply bool) (bool, error) {
// CheckApply is run to check the state and, if apply is true, to apply the // CheckApply is run to check the state and, if apply is true, to apply the
// necessary changes to reach the desired state. This is run before Watch and // necessary changes to reach the desired state. This is run before Watch and
// again if Watch finds a change occurring to the state. // again if Watch finds a change occurring to the state.
func (obj *MountRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *MountRes) CheckApply(apply bool) (bool, error) {
checkOK = true checkOK := true
if c, err := obj.fstabCheckApply(apply); err != nil { if c, err := obj.fstabCheckApply(apply); err != nil {
return false, err return false, err
@@ -588,7 +581,10 @@ func mountReload() error {
} }
defer conn.Close() defer conn.Close()
// systemctl daemon-reload // systemctl daemon-reload
conn.BusObject().Call("Reload", 0) call := conn.Object(dbusSystemd1Service, dbusSystemd1Path).Call(dbusReloadSystemd, 0)
if call.Err != nil {
return errwrap.Wrapf(call.Err, "error reloading systemd")
}
// systemctl restart local-fs.target // systemctl restart local-fs.target
if err := restartUnit(conn, "local-fs.target"); err != nil { if err := restartUnit(conn, "local-fs.target"); err != nil {
@@ -596,7 +592,7 @@ func mountReload() error {
} }
// systemctl restart remote-fs.target // systemctl restart remote-fs.target
if err := restartUnit(conn, "local-fs.target"); err != nil { if err := restartUnit(conn, "remote-fs.target"); err != nil {
return errwrap.Wrapf(err, "error restarting unit") return errwrap.Wrapf(err, "error restarting unit")
} }
@@ -631,7 +627,7 @@ func restartUnit(conn *dbus.Conn, unit string) error {
defer conn.RemoveSignal(ch) defer conn.RemoveSignal(ch)
// restart the unit // restart the unit
sd1 := conn.Object(dbusSystemd1Interface, dbus.ObjectPath(dbusSystemd1Path)) sd1 := conn.Object(dbusSystemd1Service, dbus.ObjectPath(dbusSystemd1Path))
if call := sd1.Call(dbusRestartUnit, 0, unit, "fail"); call.Err != nil { if call := sd1.Call(dbusRestartUnit, 0, unit, "fail"); call.Err != nil {
return errwrap.Wrapf(call.Err, "error restarting unit: %s", unit) return errwrap.Wrapf(call.Err, "error restarting unit: %s", unit)
} }

76
engine/resources/mount_linux_test.go Normal file
View File

@@ -0,0 +1,76 @@
// Mgmt
// Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
// +build !root !darwin
package resources
import (
"io/ioutil"
"os"
"testing"
fstab "github.com/deniswernert/go-fstab"
)
func TestMountExists(t *testing.T) {
const procMock1 = `/tmp/mount0 /mnt/proctest ext4 rw,seclabel,relatime,data=ordered 0 0` + "\n"
var mountExistsTests = []struct {
procMock []byte
in *fstab.Mount
out bool
}{
{
[]byte(procMock1),
&fstab.Mount{
Spec: "/tmp/mount0",
File: "/mnt/proctest",
VfsType: "ext4",
MntOps: map[string]string{"defaults": ""},
Freq: 1,
PassNo: 1,
},
true,
},
}
file, err := ioutil.TempFile("", "proc")
if err != nil {
t.Errorf("error creating temp file: %v", err)
return
}
defer os.Remove(file.Name())
for _, test := range mountExistsTests {
if err := ioutil.WriteFile(file.Name(), test.procMock, 0664); err != nil {
t.Errorf("error writing proc file: %s: %v", file.Name(), err)
return
}
if err := ioutil.WriteFile(test.in.Spec, []byte{}, 0664); err != nil {
t.Errorf("error writing fstab file: %s: %v", file.Name(), err)
return
}
result, err := mountExists(file.Name(), test.in)
if err != nil {
t.Errorf("error checking if fstab entry %s exists: %v", test.in.String(), err)
return
}
if result != test.out {
t.Errorf("mountExistsTests test wanted: %t, got: %t", test.out, result)
}
}
}

50
engine/resources/mount_test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -29,8 +29,6 @@ import (
const fstabMock1 = `UUID=ef5726f2-615c-4350-b0ab-f106e5fc90ad / ext4 defaults 1 1` + "\n" const fstabMock1 = `UUID=ef5726f2-615c-4350-b0ab-f106e5fc90ad / ext4 defaults 1 1` + "\n"
const procMock1 = `/tmp/mount0 /mnt/proctest ext4 rw,seclabel,relatime,data=ordered 0 0` + "\n"
var fstabWriteTests = []struct { var fstabWriteTests = []struct {
in fstab.Mounts in fstab.Mounts
}{ }{
@@ -295,49 +293,3 @@ func TestMountCompare(t *testing.T) {
} }
} }
} }
var mountExistsTests = []struct {
procMock []byte
in *fstab.Mount
out bool
}{
{
[]byte(procMock1),
&fstab.Mount{
Spec: "/tmp/mount0",
File: "/mnt/proctest",
VfsType: "ext4",
MntOps: map[string]string{"defaults": ""},
Freq: 1,
PassNo: 1,
},
true,
},
}
func TestMountExists(t *testing.T) {
file, err := ioutil.TempFile("", "proc")
if err != nil {
t.Errorf("error creating temp file: %v", err)
return
}
defer os.Remove(file.Name())
for _, test := range mountExistsTests {
if err := ioutil.WriteFile(file.Name(), test.procMock, 0664); err != nil {
t.Errorf("error writing proc file: %s: %v", file.Name(), err)
return
}
if err := ioutil.WriteFile(test.in.Spec, []byte{}, 0664); err != nil {
t.Errorf("error writing fstab file: %s: %v", file.Name(), err)
return
}
result, err := mountExists(file.Name(), test.in)
if err != nil {
t.Errorf("error checking if fstab entry %s exists: %v", test.in.String(), err)
return
}
if result != test.out {
t.Errorf("mountExistsTests test wanted: %t, got: %t", test.out, result)
}
}
}

48
engine/resources/msg.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -94,30 +94,20 @@ func (obj *MsgRes) Close() error {
// Watch is the primary listener for this resource and it outputs events. // Watch is the primary listener for this resource and it outputs events.
func (obj *MsgRes) Watch() error { func (obj *MsgRes) Watch() error {
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? //var send = false // send event?
for { for {
select { select {
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { //if send {
send = false // send = false
if err := obj.init.Event(); err != nil { // obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested //}
}
}
} }
} }
@@ -137,7 +127,7 @@ func (obj *MsgRes) isAllStateOK() bool {
func (obj *MsgRes) updateStateOK() { func (obj *MsgRes) updateStateOK() {
// XXX: this resource doesn't entirely make sense to me at the moment. // XXX: this resource doesn't entirely make sense to me at the moment.
if !obj.isAllStateOK() { if !obj.isAllStateOK() {
obj.init.Dirty() //obj.init.Dirty() // XXX: removed with API cleanup
} }
} }
@@ -210,36 +200,28 @@ func (obj *MsgRes) CheckApply(apply bool) (bool, error) {
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *MsgRes) Cmp(r engine.Res) error { func (obj *MsgRes) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *MsgRes) Compare(r engine.Res) bool {
// we can only compare MsgRes to others of the same resource kind // we can only compare MsgRes to others of the same resource kind
res, ok := r.(*MsgRes) res, ok := r.(*MsgRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.Body != res.Body { if obj.Body != res.Body {
return false return fmt.Errorf("the Body differs")
} }
if obj.Priority != res.Priority { if obj.Priority != res.Priority {
return false return fmt.Errorf("the Priority differs")
} }
if len(obj.Fields) != len(res.Fields) { if len(obj.Fields) != len(res.Fields) {
return false return fmt.Errorf("the length of Fields differs")
} }
for field, value := range obj.Fields { for field, value := range obj.Fields {
if res.Fields[field] != value { if res.Fields[field] != value {
return false return fmt.Errorf("the Fields differ")
} }
} }
return true return nil
} }
// MsgUID is a unique representation for a MsgRes object. // MsgUID is a unique representation for a MsgRes object.

2
engine/resources/msg_test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

190
engine/resources/net.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -34,9 +34,9 @@ import (
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
"github.com/purpleidea/mgmt/recwatch" "github.com/purpleidea/mgmt/recwatch"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
"github.com/purpleidea/mgmt/util/socketset"
multierr "github.com/hashicorp/go-multierror"
errwrap "github.com/pkg/errors"
// XXX: Do NOT use subscribe methods from this lib, as they are racey and // XXX: Do NOT use subscribe methods from this lib, as they are racey and
// do not clean up spawned goroutines. Should be replaced when a suitable // do not clean up spawned goroutines. Should be replaced when a suitable
// alternative is available. // alternative is available.
@@ -178,9 +178,7 @@ func (obj *NetRes) Close() error {
return fmt.Errorf("socket file should not be the root path") return fmt.Errorf("socket file should not be the root path")
} }
if obj.socketFile != "" { // safety if obj.socketFile != "" { // safety
if err := os.Remove(obj.socketFile); err != nil { errList = errwrap.Append(errList, os.Remove(obj.socketFile))
errList = multierr.Append(errList, err)
}
} }
return errList return errList
@@ -190,16 +188,20 @@ func (obj *NetRes) Close() error {
// TODO: currently gets events from ALL interfaces, would be nice to reject // TODO: currently gets events from ALL interfaces, would be nice to reject
// events from other interfaces. // events from other interfaces.
func (obj *NetRes) Watch() error { func (obj *NetRes) Watch() error {
// waitgroup for netlink receive goroutine
wg := &sync.WaitGroup{}
defer wg.Wait()
// create a netlink socket for receiving network interface events // create a netlink socket for receiving network interface events
conn, err := newSocketSet(rtmGrps, obj.socketFile) conn, err := socketset.NewSocketSet(rtmGrps, obj.socketFile, unix.NETLINK_ROUTE)
if err != nil { if err != nil {
return errwrap.Wrapf(err, "error creating socket set") return errwrap.Wrapf(err, "error creating socket set")
} }
defer conn.shutdown() // close the netlink socket and unblock conn.receive()
// waitgroup for netlink receive goroutine
wg := &sync.WaitGroup{}
defer conn.Close()
// We must wait for the Shutdown() AND the select inside of SocketSet to
// complete before we Close, since the unblocking in SocketSet is not a
// synchronous operation.
defer wg.Wait()
defer conn.Shutdown() // close the netlink socket and unblock conn.receive()
// watch the systemd-networkd configuration file // watch the systemd-networkd configuration file
recWatcher, err := recwatch.NewRecWatcher(obj.unitFilePath, false) recWatcher, err := recwatch.NewRecWatcher(obj.unitFilePath, false)
@@ -219,11 +221,10 @@ func (obj *NetRes) Watch() error {
wg.Add(1) wg.Add(1)
go func() { go func() {
defer wg.Done() defer wg.Done()
defer conn.close() // close the pipe when we're done with it
defer close(nlChan) defer close(nlChan)
for { for {
// receive messages from the socket set // receive messages from the socket set
msgs, err := conn.receive() msgs, err := conn.ReceiveNetlinkMessages()
if err != nil { if err != nil {
select { select {
case nlChan <- &nlChanStruct{ case nlChan <- &nlChanStruct{
@@ -243,10 +244,7 @@ func (obj *NetRes) Watch() error {
} }
}() }()
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? var send = false // send event?
var done bool var done bool
@@ -268,7 +266,6 @@ func (obj *NetRes) Watch() error {
} }
send = true send = true
obj.init.Dirty() // dirty
case event, ok := <-recWatcher.Events(): case event, ok := <-recWatcher.Events():
if !ok { if !ok {
@@ -286,23 +283,15 @@ func (obj *NetRes) Watch() error {
} }
send = true send = true
obj.init.Dirty() // dirty
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
@@ -471,8 +460,8 @@ func (obj *NetRes) fileCheckApply(apply bool) (bool, error) {
// CheckApply is run to check the state and, if apply is true, to apply the // CheckApply is run to check the state and, if apply is true, to apply the
// necessary changes to reach the desired state. This is run before Watch and // necessary changes to reach the desired state. This is run before Watch and
// again if Watch finds a change occurring to the state. // again if Watch finds a change occurring to the state.
func (obj *NetRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *NetRes) CheckApply(apply bool) (bool, error) {
checkOK = true checkOK := true
// check the network device // check the network device
if c, err := obj.ifaceCheckApply(apply); err != nil { if c, err := obj.ifaceCheckApply(apply); err != nil {
@@ -517,34 +506,26 @@ func (obj *NetRes) CheckApply(apply bool) (checkOK bool, err error) {
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *NetRes) Cmp(r engine.Res) error { func (obj *NetRes) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *NetRes) Compare(r engine.Res) bool {
// we can only compare NetRes to others of the same resource kind // we can only compare NetRes to others of the same resource kind
res, ok := r.(*NetRes) res, ok := r.(*NetRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.State != res.State { if obj.State != res.State {
return false return fmt.Errorf("the State differs")
} }
if (obj.Addrs == nil) != (res.Addrs == nil) { if (obj.Addrs == nil) != (res.Addrs == nil) {
return false return fmt.Errorf("the Addrs differ")
} }
if err := util.SortedStrSliceCompare(obj.Addrs, res.Addrs); err != nil { if err := util.SortedStrSliceCompare(obj.Addrs, res.Addrs); err != nil {
return false return fmt.Errorf("the Addrs differ")
} }
if obj.Gateway != res.Gateway { if obj.Gateway != res.Gateway {
return false return fmt.Errorf("the Gateway differs")
} }
return true return nil
} }
// NetUID is a unique resource identifier. // NetUID is a unique resource identifier.
@@ -768,122 +749,3 @@ func (obj *iface) addrApplyAdd(objAddrs []string) error {
} }
return nil return nil
} }
// socketSet is used to receive events from a socket and shut it down cleanly
// when asked. It contains a socket for events and a pipe socket to unblock
// receive on shutdown.
type socketSet struct {
fdEvents int
fdPipe int
pipeFile string
}
// newSocketSet returns a socketSet, initialized with the given parameters.
func newSocketSet(groups uint32, file string) (*socketSet, error) {
// make a netlink socket file descriptor
fdEvents, err := unix.Socket(unix.AF_NETLINK, unix.SOCK_RAW, unix.NETLINK_ROUTE)
if err != nil {
return nil, errwrap.Wrapf(err, "error creating netlink socket")
}
// bind to the socket and add add the netlink groups we need to get events
if err := unix.Bind(fdEvents, &unix.SockaddrNetlink{
Family: unix.AF_NETLINK,
Groups: groups,
}); err != nil {
return nil, errwrap.Wrapf(err, "error binding netlink socket")
}
// create a pipe socket to unblock unix.Select when we close
fdPipe, err := unix.Socket(unix.AF_UNIX, unix.SOCK_RAW, unix.PROT_NONE)
if err != nil {
return nil, errwrap.Wrapf(err, "error creating pipe socket")
}
// bind the pipe to a file
if err = unix.Bind(fdPipe, &unix.SockaddrUnix{
Name: file,
}); err != nil {
return nil, errwrap.Wrapf(err, "error binding pipe socket")
}
return &socketSet{
fdEvents: fdEvents,
fdPipe: fdPipe,
pipeFile: file,
}, nil
}
// shutdown closes the event file descriptor and unblocks receive by sending
// a message to the pipe file descriptor. It must be called before close, and
// should only be called once.
func (obj *socketSet) shutdown() error {
// close the event socket so no more events are produced
if err := unix.Close(obj.fdEvents); err != nil {
return err
}
// send a message to the pipe to unblock select
return unix.Sendto(obj.fdPipe, nil, 0, &unix.SockaddrUnix{
Name: path.Join(obj.pipeFile),
})
}
// close closes the pipe file descriptor. It must only be called after
// shutdown has closed fdEvents, and unblocked receive. It should only be
// called once.
func (obj *socketSet) close() error {
return unix.Close(obj.fdPipe)
}
// receive waits for bytes from fdEvents and parses them into a slice of
// netlink messages. It will block until an event is produced, or shutdown
// is called.
func (obj *socketSet) receive() ([]syscall.NetlinkMessage, error) {
// Select will return when any fd in fdSet (fdEvents and fdPipe) is ready
// to read.
_, err := unix.Select(obj.nfd(), obj.fdSet(), nil, nil, nil)
if err != nil {
// if a system interrupt is caught
if err == unix.EINTR { // signal interrupt
return nil, nil
}
return nil, errwrap.Wrapf(err, "error selecting on fd")
}
// receive the message from the netlink socket into b
b := make([]byte, os.Getpagesize())
n, _, err := unix.Recvfrom(obj.fdEvents, b, unix.MSG_DONTWAIT) // non-blocking receive
if err != nil {
// if fdEvents is closed
if err == unix.EBADF { // bad file descriptor
return nil, nil
}
return nil, errwrap.Wrapf(err, "error receiving messages")
}
// if we didn't get enough bytes for a header, something went wrong
if n < unix.NLMSG_HDRLEN {
return nil, fmt.Errorf("received short header")
}
b = b[:n] // truncate b to message length
// use syscall to parse, as func does not exist in x/sys/unix
return syscall.ParseNetlinkMessage(b)
}
// nfd returns one more than the highest fd value in the struct, for use as as
// the nfds parameter in select. It represents the file descriptor set maximum
// size. See man select for more info.
func (obj *socketSet) nfd() int {
if obj.fdEvents > obj.fdPipe {
return obj.fdEvents + 1
}
return obj.fdPipe + 1
}
// fdSet returns a bitmask representation of the integer values of fdEvents
// and fdPipe. See man select for more info.
func (obj *socketSet) fdSet() *unix.FdSet {
fdSet := &unix.FdSet{}
// Generate the bitmask representing the file descriptors in the socketSet.
// The rightmost bit corresponds to file descriptor zero, and each bit to
// the left represents the next file descriptor number in the sequence of
// all real numbers. E.g. the FdSet containing containing 0 and 4 is 10001.
fdSet.Bits[obj.fdEvents/64] |= 1 << uint(obj.fdEvents)
fdSet.Bits[obj.fdPipe/64] |= 1 << uint(obj.fdPipe)
return fdSet
}

88
engine/resources/net_test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -15,14 +15,14 @@
// You should have received a copy of the GNU General Public License // You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>. // along with this program. If not, see <http://www.gnu.org/licenses/>.
// +build !darwin
package resources package resources
import ( import (
"bytes" "bytes"
"strings" "strings"
"testing" "testing"
"golang.org/x/sys/unix"
) )
// test cases for NetRes.unitFileContents() // test cases for NetRes.unitFileContents()
@@ -82,85 +82,3 @@ func TestUnitFileContents(t *testing.T) {
} }
} }
} }
// test cases for socketSet.fdSet()
var fdSetTests = []struct {
in *socketSet
out *unix.FdSet
}{
{
&socketSet{
fdEvents: 3,
fdPipe: 4,
},
&unix.FdSet{
Bits: [16]int64{0x18}, // 11000
},
},
{
&socketSet{
fdEvents: 12,
fdPipe: 8,
},
&unix.FdSet{
Bits: [16]int64{0x1100}, // 1000100000000
},
},
{
&socketSet{
fdEvents: 9,
fdPipe: 21,
},
&unix.FdSet{
Bits: [16]int64{0x200200}, // 1000000000001000000000
},
},
}
// test socketSet.fdSet()
func TestFdSet(t *testing.T) {
for _, test := range fdSetTests {
result := test.in.fdSet()
if *result != *test.out {
t.Errorf("fdSet test wanted: %b, got: %b", *test.out, *result)
}
}
}
// test cases for socketSet.nfd()
var nfdTests = []struct {
in *socketSet
out int
}{
{
&socketSet{
fdEvents: 3,
fdPipe: 4,
},
5,
},
{
&socketSet{
fdEvents: 8,
fdPipe: 4,
},
9,
},
{
&socketSet{
fdEvents: 90,
fdPipe: 900,
},
901,
},
}
// test socketSet.nfd()
func TestNfd(t *testing.T) {
for _, test := range nfdTests {
result := test.in.nfd()
if result != test.out {
t.Errorf("nfd test wanted: %d, got: %d", test.out, result)
}
}
}

30
engine/resources/noop.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -63,35 +63,19 @@ func (obj *NoopRes) Close() error {
// Watch is the primary listener for this resource and it outputs events. // Watch is the primary listener for this resource and it outputs events.
func (obj *NoopRes) Watch() error { func (obj *NoopRes) Watch() error {
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event?
for {
select { select {
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil
}
if err := obj.init.Read(event); err != nil {
return err
}
} }
// do all our event sending all together to avoid duplicate msgs //obj.init.Event() // notify engine of an event (this can block)
if send {
send = false return nil
if err := obj.init.Event(); err != nil {
return err // exit if requested
}
}
}
} }
// CheckApply method for Noop resource. Does nothing, returns happy! // CheckApply method for Noop resource. Does nothing, returns happy!
func (obj *NoopRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *NoopRes) CheckApply(apply bool) (bool, error) {
if obj.init.Refresh() { if obj.init.Refresh() {
obj.init.Logf("received a notification!") obj.init.Logf("received a notification!")
} }

2
engine/resources/noop_test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

48
engine/resources/nspawn.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -21,18 +21,19 @@ import (
"errors" "errors"
"fmt" "fmt"
"strconv" "strconv"
"strings"
"unicode" "unicode"
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
engineUtil "github.com/purpleidea/mgmt/engine/util" engineUtil "github.com/purpleidea/mgmt/engine/util"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
systemdDbus "github.com/coreos/go-systemd/dbus" systemdDbus "github.com/coreos/go-systemd/dbus"
machined "github.com/coreos/go-systemd/machine1" machined "github.com/coreos/go-systemd/machine1"
systemdUtil "github.com/coreos/go-systemd/util" systemdUtil "github.com/coreos/go-systemd/util"
"github.com/godbus/dbus" "github.com/godbus/dbus"
errwrap "github.com/pkg/errors"
) )
const ( const (
@@ -167,10 +168,7 @@ func (obj *NspawnRes) Watch() error {
bus.Signal(busChan) bus.Signal(busChan)
defer bus.RemoveSignal(busChan) // not needed here, but nice for symmetry defer bus.RemoveSignal(busChan) // not needed here, but nice for symmetry
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? var send = false // send event?
for { for {
@@ -187,24 +185,16 @@ func (obj *NspawnRes) Watch() error {
return fmt.Errorf("unknown event: %s", event.Name) return fmt.Errorf("unknown event: %s", event.Name)
} }
send = true send = true
obj.init.Dirty() // dirty
} }
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
@@ -212,7 +202,7 @@ func (obj *NspawnRes) Watch() error {
// CheckApply is run to check the state and, if apply is true, to apply the // CheckApply is run to check the state and, if apply is true, to apply the
// necessary changes to reach the desired state. This is run before Watch and // necessary changes to reach the desired state. This is run before Watch and
// again if Watch finds a change occurring to the state. // again if Watch finds a change occurring to the state.
func (obj *NspawnRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *NspawnRes) CheckApply(apply bool) (bool, error) {
// this resource depends on systemd to ensure that it's running // this resource depends on systemd to ensure that it's running
if !systemdUtil.IsRunningSystemd() { if !systemdUtil.IsRunningSystemd() {
return false, errors.New("systemd is not running") return false, errors.New("systemd is not running")
@@ -271,35 +261,27 @@ func (obj *NspawnRes) CheckApply(apply bool) (checkOK bool, err error) {
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *NspawnRes) Cmp(r engine.Res) error { func (obj *NspawnRes) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *NspawnRes) Compare(r engine.Res) bool {
// we can only compare NspawnRes to others of the same resource kind // we can only compare NspawnRes to others of the same resource kind
res, ok := r.(*NspawnRes) res, ok := r.(*NspawnRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.State != res.State { if obj.State != res.State {
return false return fmt.Errorf("the State differs")
} }
// TODO: why is res.svc ever nil? // TODO: why is res.svc ever nil?
if (obj.svc == nil) != (res.svc == nil) { // xor if (obj.svc == nil) != (res.svc == nil) { // xor
return false return fmt.Errorf("the svc differs")
} }
if obj.svc != nil && res.svc != nil { if obj.svc != nil && res.svc != nil {
if !obj.svc.Compare(res.svc) { if err := obj.svc.Cmp(res.svc); err != nil {
return false return errwrap.Wrapf(err, "the svc differs")
} }
} }
return true return nil
} }
// NspawnUID is a unique resource identifier. // NspawnUID is a unique resource identifier.
@@ -369,10 +351,12 @@ func systemdVersion() (uint16, error) {
return 0, errwrap.Wrapf(err, "could not get version property") return 0, errwrap.Wrapf(err, "could not get version property")
} }
// lose the surrounding quotes // lose the surrounding quotes
verNum, err := strconv.Unquote(verString) verNumString, err := strconv.Unquote(verString)
if err != nil { if err != nil {
return 0, errwrap.Wrapf(err, "error unquoting version number") return 0, errwrap.Wrapf(err, "error unquoting version number")
} }
// trim possible version suffix like in "242.19-1"
verNum := strings.Split(verNumString, ".")[0]
// cast to uint16 // cast to uint16
ver, err := strconv.ParseUint(verNum, 10, 16) ver, err := strconv.ParseUint(verNum, 10, 16)
if err != nil { if err != nil {

60
engine/resources/packagekit/packagekit.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -27,10 +27,9 @@ import (
engineUtil "github.com/purpleidea/mgmt/engine/util" engineUtil "github.com/purpleidea/mgmt/engine/util"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
"github.com/godbus/dbus" "github.com/godbus/dbus"
multierr "github.com/hashicorp/go-multierror"
errwrap "github.com/pkg/errors"
) )
// global tweaks of verbosity and code path // global tweaks of verbosity and code path
@@ -198,9 +197,8 @@ func (obj *Conn) matchSignal(ch chan *dbus.Signal, path dbus.ObjectPath, iface s
removeSignals := func() error { removeSignals := func() error {
var errList error var errList error
for i := len(argsList) - 1; i >= 0; i-- { // last in first out for i := len(argsList) - 1; i >= 0; i-- { // last in first out
if call := bus.Call(engineUtil.DBusRemoveMatch, 0, argsList[i]); call.Err != nil { call := bus.Call(engineUtil.DBusRemoveMatch, 0, argsList[i])
errList = multierr.Append(errList, call.Err) errList = errwrap.Append(errList, call.Err)
}
} }
return errList return errList
} }
@@ -354,7 +352,7 @@ loop:
// should already be broken // should already be broken
break loop break loop
} else { } else {
return []string{}, fmt.Errorf("PackageKit: Error: %v", signal.Body) return []string{}, fmt.Errorf("error in body: %v", signal.Body)
} }
} }
} }
@@ -365,9 +363,9 @@ loop:
func (obj *Conn) IsInstalledList(packages []string) ([]bool, error) { func (obj *Conn) IsInstalledList(packages []string) ([]bool, error) {
var filter uint64 // initializes at the "zero" value of 0 var filter uint64 // initializes at the "zero" value of 0
filter += PkFilterEnumArch // always search in our arch filter += PkFilterEnumArch // always search in our arch
packageIDs, e := obj.ResolvePackages(packages, filter) packageIDs, err := obj.ResolvePackages(packages, filter)
if e != nil { if err != nil {
return nil, fmt.Errorf("ResolvePackages error: %v", e) return nil, errwrap.Wrapf(err, "error resolving packages")
} }
var m = make(map[string]int) var m = make(map[string]int)
@@ -445,7 +443,7 @@ loop:
} }
if signal.Name == FmtTransactionMethod("ErrorCode") { if signal.Name == FmtTransactionMethod("ErrorCode") {
return fmt.Errorf("PackageKit: Error: %v", signal.Body) return fmt.Errorf("error in body: %v", signal.Body)
} else if signal.Name == FmtTransactionMethod("Package") { } else if signal.Name == FmtTransactionMethod("Package") {
// a package was installed... // a package was installed...
// only start the timer once we're here... // only start the timer once we're here...
@@ -456,14 +454,14 @@ loop:
} else if signal.Name == FmtTransactionMethod("Destroy") { } else if signal.Name == FmtTransactionMethod("Destroy") {
return nil // success return nil // success
} else { } else {
return fmt.Errorf("PackageKit: Error: %v", signal.Body) return fmt.Errorf("error in body: %v", signal.Body)
} }
case <-util.TimeAfterOrBlock(timeout): case <-util.TimeAfterOrBlock(timeout):
if finished { if finished {
obj.Logf("Timeout: InstallPackages: Waiting for 'Destroy'") obj.Logf("Timeout: InstallPackages: Waiting for 'Destroy'")
return nil // got tired of waiting for Destroy return nil // got tired of waiting for Destroy
} }
return fmt.Errorf("PackageKit: Timeout: InstallPackages: %s", strings.Join(packageIDs, ", ")) return fmt.Errorf("timeout installing packages: %s", strings.Join(packageIDs, ", "))
} }
} }
} }
@@ -502,7 +500,7 @@ loop:
} }
if signal.Name == FmtTransactionMethod("ErrorCode") { if signal.Name == FmtTransactionMethod("ErrorCode") {
return fmt.Errorf("PackageKit: Error: %v", signal.Body) return fmt.Errorf("error in body: %v", signal.Body)
} else if signal.Name == FmtTransactionMethod("Package") { } else if signal.Name == FmtTransactionMethod("Package") {
// a package was installed... // a package was installed...
continue loop continue loop
@@ -513,7 +511,7 @@ loop:
// should already be broken // should already be broken
break loop break loop
} else { } else {
return fmt.Errorf("PackageKit: Error: %v", signal.Body) return fmt.Errorf("error in body: %v", signal.Body)
} }
} }
} }
@@ -551,7 +549,7 @@ loop:
} }
if signal.Name == FmtTransactionMethod("ErrorCode") { if signal.Name == FmtTransactionMethod("ErrorCode") {
return fmt.Errorf("PackageKit: Error: %v", signal.Body) return fmt.Errorf("error in body: %v", signal.Body)
} else if signal.Name == FmtTransactionMethod("Package") { } else if signal.Name == FmtTransactionMethod("Package") {
} else if signal.Name == FmtTransactionMethod("Finished") { } else if signal.Name == FmtTransactionMethod("Finished") {
// TODO: should we wait for the Destroy signal? // TODO: should we wait for the Destroy signal?
@@ -560,7 +558,7 @@ loop:
// should already be broken // should already be broken
break loop break loop
} else { } else {
return fmt.Errorf("PackageKit: Error: %v", signal.Body) return fmt.Errorf("error in body: %v", signal.Body)
} }
} }
} }
@@ -603,7 +601,7 @@ loop:
} }
if signal.Name == FmtTransactionMethod("ErrorCode") { if signal.Name == FmtTransactionMethod("ErrorCode") {
err = fmt.Errorf("PackageKit: Error: %v", signal.Body) err = fmt.Errorf("error in body: %v", signal.Body)
return return
// one signal returned per packageID found... // one signal returned per packageID found...
@@ -628,7 +626,7 @@ loop:
// should already be broken // should already be broken
break loop break loop
} else { } else {
err = fmt.Errorf("PackageKit: Error: %v", signal.Body) err = fmt.Errorf("error in body: %v", signal.Body)
return return
} }
} }
@@ -671,7 +669,7 @@ loop:
} }
if signal.Name == FmtTransactionMethod("ErrorCode") { if signal.Name == FmtTransactionMethod("ErrorCode") {
return nil, fmt.Errorf("PackageKit: Error: %v", signal.Body) return nil, fmt.Errorf("error in body: %v", signal.Body)
} else if signal.Name == FmtTransactionMethod("Package") { } else if signal.Name == FmtTransactionMethod("Package") {
//pkg_int, ok := signal.Body[0].(int) //pkg_int, ok := signal.Body[0].(int)
@@ -694,7 +692,7 @@ loop:
// should already be broken // should already be broken
break loop break loop
} else { } else {
return nil, fmt.Errorf("PackageKit: Error: %v", signal.Body) return nil, fmt.Errorf("error in body: %v", signal.Body)
} }
} }
} }
@@ -720,9 +718,9 @@ func (obj *Conn) PackagesToPackageIDs(packageMap map[string]string, filter uint6
if obj.Debug { if obj.Debug {
obj.Logf("PackagesToPackageIDs(): %s", strings.Join(packages, ", ")) obj.Logf("PackagesToPackageIDs(): %s", strings.Join(packages, ", "))
} }
resolved, e := obj.ResolvePackages(packages, filter) resolved, err := obj.ResolvePackages(packages, filter)
if e != nil { if err != nil {
return nil, fmt.Errorf("Resolve error: %v", e) return nil, errwrap.Wrapf(err, "error resolving")
} }
found := make([]bool, count) // default false found := make([]bool, count) // default false
@@ -760,7 +758,7 @@ func (obj *Conn) PackagesToPackageIDs(packageMap map[string]string, filter uint6
} }
state := packageMap[pkg] // lookup the requested state/version state := packageMap[pkg] // lookup the requested state/version
if state == "" { if state == "" {
return nil, fmt.Errorf("Empty package state for %v", pkg) return nil, fmt.Errorf("empty package state for: `%s`", pkg)
} }
found[index] = true found[index] = true
stateIsVersion := (state != "installed" && state != "uninstalled" && state != "newest") // must be a ver. string stateIsVersion := (state != "installed" && state != "uninstalled" && state != "newest") // must be a ver. string
@@ -796,9 +794,9 @@ func (obj *Conn) PackagesToPackageIDs(packageMap map[string]string, filter uint6
// to be done, and if so, anything that needs updating isn't newest! // to be done, and if so, anything that needs updating isn't newest!
// if something isn't installed, we can't verify it with this method // if something isn't installed, we can't verify it with this method
// FIXME: https://github.com/hughsie/PackageKit/issues/116 // FIXME: https://github.com/hughsie/PackageKit/issues/116
updates, e := obj.GetUpdates(filter) updates, err := obj.GetUpdates(filter)
if e != nil { if err != nil {
return nil, fmt.Errorf("Updates error: %v", e) return nil, errwrap.Wrapf(err, "updates error")
} }
for _, packageID := range updates { for _, packageID := range updates {
//obj.Logf("* %v", packageID) //obj.Logf("* %v", packageID)
@@ -846,9 +844,9 @@ func (obj *Conn) PackagesToPackageIDs(packageMap map[string]string, filter uint6
if obj.Debug { if obj.Debug {
obj.Logf("PackagesToPackageIDs(): Recurse: %s", strings.Join(checkPackages, ", ")) obj.Logf("PackagesToPackageIDs(): Recurse: %s", strings.Join(checkPackages, ", "))
} }
recursion, e = obj.PackagesToPackageIDs(filteredPackageMap, filter+PkFilterEnumNewest) recursion, err = obj.PackagesToPackageIDs(filteredPackageMap, filter+PkFilterEnumNewest)
if e != nil { if err != nil {
return nil, fmt.Errorf("Recursion error: %v", e) return nil, errwrap.Wrapf(err, "recursion error")
} }
} }
} }

44
engine/resources/password.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -29,8 +29,7 @@ import (
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
"github.com/purpleidea/mgmt/recwatch" "github.com/purpleidea/mgmt/recwatch"
"github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
) )
func init() { func init() {
@@ -182,10 +181,7 @@ func (obj *PasswordRes) Watch() error {
} }
defer obj.recWatcher.Close() defer obj.recWatcher.Close()
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? var send = false // send event?
for { for {
@@ -199,29 +195,21 @@ func (obj *PasswordRes) Watch() error {
return errwrap.Wrapf(err, "unknown %s watcher error", obj) return errwrap.Wrapf(err, "unknown %s watcher error", obj)
} }
send = true send = true
obj.init.Dirty() // dirty
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
// CheckApply method for Password resource. Does nothing, returns happy! // CheckApply method for Password resource. Does nothing, returns happy!
func (obj *PasswordRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *PasswordRes) CheckApply(apply bool) (bool, error) {
var refresh = obj.init.Refresh() // do we have a pending reload to apply? var refresh = obj.init.Refresh() // do we have a pending reload to apply?
var exists = true // does the file (aka the token) exist? var exists = true // does the file (aka the token) exist?
var generate bool // do we need to generate a new password? var generate bool // do we need to generate a new password?
@@ -307,33 +295,25 @@ func (obj *PasswordRes) CheckApply(apply bool) (checkOK bool, err error) {
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *PasswordRes) Cmp(r engine.Res) error { func (obj *PasswordRes) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *PasswordRes) Compare(r engine.Res) bool {
// we can only compare PasswordRes to others of the same resource kind // we can only compare PasswordRes to others of the same resource kind
res, ok := r.(*PasswordRes) res, ok := r.(*PasswordRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.Length != res.Length { if obj.Length != res.Length {
return false return fmt.Errorf("the Length differs")
} }
// TODO: we *could* optimize by allowing CheckApply to move from // TODO: we *could* optimize by allowing CheckApply to move from
// saved->!saved, by removing the file, but not likely worth it! // saved->!saved, by removing the file, but not likely worth it!
if obj.Saved != res.Saved { if obj.Saved != res.Saved {
return false return fmt.Errorf("the Saved differs")
} }
if obj.CheckRecovery != res.CheckRecovery { if obj.CheckRecovery != res.CheckRecovery {
return false return fmt.Errorf("the CheckRecovery differs")
} }
return true return nil
} }
// PasswordUID is the UID struct for PasswordRes. // PasswordUID is the UID struct for PasswordRes.
@@ -355,7 +335,7 @@ func (obj *PasswordRes) UIDs() []engine.ResUID {
// PasswordSends is the struct of data which is sent after a successful Apply. // PasswordSends is the struct of data which is sent after a successful Apply.
type PasswordSends struct { type PasswordSends struct {
// Password is the generated password being sent. // Password is the generated password being sent.
Password *string Password *string `lang:"password"`
// Hashing is the algorithm used for this password. Empty is plain text. // Hashing is the algorithm used for this password. Empty is plain text.
Hashing string // TODO: implement me Hashing string // TODO: implement me
} }

36
engine/resources/pkg.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -26,8 +26,7 @@ import (
"github.com/purpleidea/mgmt/engine/resources/packagekit" "github.com/purpleidea/mgmt/engine/resources/packagekit"
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
) )
func init() { func init() {
@@ -67,7 +66,7 @@ type PkgRes struct {
// Default returns some sensible defaults for this resource. // Default returns some sensible defaults for this resource.
func (obj *PkgRes) Default() engine.Res { func (obj *PkgRes) Default() engine.Res {
return &PkgRes{ return &PkgRes{
State: PkgStateInstalled, // i think this is preferable to "latest" State: PkgStateInstalled, // this *is* preferable to "newest"
} }
} }
@@ -76,6 +75,9 @@ func (obj *PkgRes) Validate() error {
if obj.State == "" { if obj.State == "" {
return fmt.Errorf("state cannot be empty") return fmt.Errorf("state cannot be empty")
} }
if obj.State == "latest" {
return fmt.Errorf("state is invalid, did you mean `newest` ?")
}
return nil return nil
} }
@@ -118,10 +120,7 @@ func (obj *PkgRes) Watch() error {
return errwrap.Wrapf(err, "error adding signal match") return errwrap.Wrapf(err, "error adding signal match")
} }
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? var send = false // send event?
for { for {
@@ -143,20 +142,15 @@ func (obj *PkgRes) Watch() error {
} }
send = true send = true
obj.init.Dirty() // dirty
case event := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if err := obj.init.Read(event); err != nil { return nil
return err
}
} }
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
@@ -203,7 +197,7 @@ func (obj *PkgRes) pkgMappingHelper(bus *packagekit.Conn) (map[string]*packageki
packageMap[obj.Name()] = obj.State // key is pkg name, value is pkg state packageMap[obj.Name()] = obj.State // key is pkg name, value is pkg state
var filter uint64 // initializes at the "zero" value of 0 var filter uint64 // initializes at the "zero" value of 0
filter += packagekit.PkFilterEnumArch // always search in our arch (optional!) filter += packagekit.PkFilterEnumArch // always search in our arch (optional!)
// we're requesting latest version, or to narrow down install choices! // we're requesting newest version, or to narrow down install choices!
if obj.State == PkgStateNewest || obj.State == PkgStateInstalled { if obj.State == PkgStateNewest || obj.State == PkgStateInstalled {
// if we add this, we'll still see older packages if installed // if we add this, we'll still see older packages if installed
// this is an optimization, and is *optional*, this logic is // this is an optimization, and is *optional*, this logic is
@@ -218,7 +212,7 @@ func (obj *PkgRes) pkgMappingHelper(bus *packagekit.Conn) (map[string]*packageki
} }
result, err := bus.PackagesToPackageIDs(packageMap, filter) result, err := bus.PackagesToPackageIDs(packageMap, filter)
if err != nil { if err != nil {
return nil, errwrap.Wrapf(err, "Can't run PackagesToPackageIDs") return nil, errwrap.Wrapf(err, "can't run PackagesToPackageIDs")
} }
return result, nil return result, nil
} }
@@ -249,6 +243,10 @@ func (obj *PkgRes) populateFileList() error {
if !ok || !data.Found { if !ok || !data.Found {
return fmt.Errorf("can't find package named '%s'", obj.Name()) return fmt.Errorf("can't find package named '%s'", obj.Name())
} }
if data.PackageID == "" {
// this can happen if you specify a bad version like "latest"
return fmt.Errorf("empty PackageID found for '%s'", obj.Name())
}
packageIDs := []string{data.PackageID} // just one for now packageIDs := []string{data.PackageID} // just one for now
filesMap, err := bus.GetFilesByPackageID(packageIDs) filesMap, err := bus.GetFilesByPackageID(packageIDs)
@@ -264,7 +262,7 @@ func (obj *PkgRes) populateFileList() error {
// CheckApply checks the resource state and applies the resource if the bool // CheckApply checks the resource state and applies the resource if the bool
// input is true. It returns error info and if the state check passed or not. // input is true. It returns error info and if the state check passed or not.
func (obj *PkgRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *PkgRes) CheckApply(apply bool) (bool, error) {
obj.init.Logf("Check: %s", obj.fmtNames(obj.getNames())) obj.init.Logf("Check: %s", obj.fmtNames(obj.getNames()))
bus := packagekit.NewBus() bus := packagekit.NewBus()

2
engine/resources/pkg_test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

64
engine/resources/print.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -40,6 +40,10 @@ type PrintRes struct {
init *engine.Init init *engine.Init
Msg string `lang:"msg" yaml:"msg"` // the message to display Msg string `lang:"msg" yaml:"msg"` // the message to display
// RefreshOnly is an option that causes the message to be printed only
// when notified by another resource. When set to true, this resource
// cannot be autogrouped.
RefreshOnly bool `lang:"refresh_only" yaml:"refresh_only"`
} }
// Default returns some sensible defaults for this resource. // Default returns some sensible defaults for this resource.
@@ -66,77 +70,61 @@ func (obj *PrintRes) Close() error {
// Watch is the primary listener for this resource and it outputs events. // Watch is the primary listener for this resource and it outputs events.
func (obj *PrintRes) Watch() error { func (obj *PrintRes) Watch() error {
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event?
for {
select { select {
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil
}
if err := obj.init.Read(event); err != nil {
return err
}
} }
// do all our event sending all together to avoid duplicate msgs //obj.init.Event() // notify engine of an event (this can block)
if send {
send = false return nil
if err := obj.init.Event(); err != nil {
return err // exit if requested
}
}
}
} }
// CheckApply method for Print resource. Does nothing, returns happy! // CheckApply method for Print resource. Does nothing, returns happy!
func (obj *PrintRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *PrintRes) CheckApply(apply bool) (bool, error) {
obj.init.Logf("CheckApply: %t", apply) obj.init.Logf("CheckApply: %t", apply)
if val, exists := obj.init.Recv()["Msg"]; exists && val.Changed { if val, exists := obj.init.Recv()["Msg"]; exists && val.Changed {
// if we received on Msg, and it changed, log message // if we received on Msg, and it changed, log message
obj.init.Logf("CheckApply: Received `Msg` of: %s", obj.Msg) obj.init.Logf("CheckApply: Received `Msg` of: %s", obj.Msg)
} }
if obj.init.Refresh() { var refresh = obj.init.Refresh()
// we output if not RefreshOnly, or we are in refresh mode and RefreshOnly
var display = refresh || !obj.RefreshOnly
if refresh {
obj.init.Logf("Received a notification!") obj.init.Logf("Received a notification!")
} }
if display {
obj.init.Logf("Msg: %s", obj.Msg) obj.init.Logf("Msg: %s", obj.Msg)
}
if g := obj.GetGroup(); len(g) > 0 { // add any grouped elements if g := obj.GetGroup(); len(g) > 0 { // add any grouped elements
for _, x := range g { for _, x := range g {
print, ok := x.(*PrintRes) // convert from Res print, ok := x.(*PrintRes) // convert from Res
if !ok { if !ok {
panic(fmt.Sprintf("grouped member %v is not a %s", x, obj.Kind())) panic(fmt.Sprintf("grouped member %v is not a %s", x, obj.Kind()))
} }
if display {
obj.init.Logf("%s: Msg: %s", print, print.Msg) obj.init.Logf("%s: Msg: %s", print, print.Msg)
} }
} }
}
return true, nil // state is always okay return true, nil // state is always okay
} }
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *PrintRes) Cmp(r engine.Res) error { func (obj *PrintRes) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *PrintRes) Compare(r engine.Res) bool {
// we can only compare PrintRes to others of the same resource kind // we can only compare PrintRes to others of the same resource kind
res, ok := r.(*PrintRes) res, ok := r.(*PrintRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.Msg != res.Msg { if obj.Msg != res.Msg {
return false return fmt.Errorf("the Msg differs")
} }
return true return nil
} }
// PrintUID is the UID struct for PrintRes. // PrintUID is the UID struct for PrintRes.
@@ -157,10 +145,14 @@ func (obj *PrintRes) UIDs() []engine.ResUID {
// GroupCmp returns whether two resources can be grouped together or not. // GroupCmp returns whether two resources can be grouped together or not.
func (obj *PrintRes) GroupCmp(r engine.GroupableRes) error { func (obj *PrintRes) GroupCmp(r engine.GroupableRes) error {
_, ok := r.(*PrintRes) res, ok := r.(*PrintRes)
if !ok { if !ok {
return fmt.Errorf("resource is not the same kind") return fmt.Errorf("resource is not the same kind")
} }
// we don't group if it's RefreshOnly: only the notifier may trigger
if obj.RefreshOnly || res.RefreshOnly {
return fmt.Errorf("resource uses RefreshOnly, it cannot be merged")
}
return nil // grouped together if we were asked to return nil // grouped together if we were asked to
} }

1182
engine/resources/resources_test.go Normal file
View File

File diff suppressed because it is too large Load Diff

55
engine/resources/svc.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -28,11 +28,11 @@ import (
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
engineUtil "github.com/purpleidea/mgmt/engine/util" engineUtil "github.com/purpleidea/mgmt/engine/util"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
systemd "github.com/coreos/go-systemd/dbus" // change namespace systemd "github.com/coreos/go-systemd/dbus" // change namespace
systemdUtil "github.com/coreos/go-systemd/util" systemdUtil "github.com/coreos/go-systemd/util"
"github.com/godbus/dbus" // namespace collides with systemd wrapper "github.com/godbus/dbus" // namespace collides with systemd wrapper
errwrap "github.com/pkg/errors"
) )
func init() { func init() {
@@ -120,10 +120,7 @@ func (obj *SvcRes) Watch() error {
bus.Signal(buschan) bus.Signal(buschan)
defer bus.RemoveSignal(buschan) // not needed here, but nice for symmetry defer bus.RemoveSignal(buschan) // not needed here, but nice for symmetry
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var svc = fmt.Sprintf("%s.service", obj.Name()) // systemd name var svc = fmt.Sprintf("%s.service", obj.Name()) // systemd name
var send = false // send event? var send = false // send event?
@@ -161,7 +158,6 @@ func (obj *SvcRes) Watch() error {
if previous != invalid { // if invalid changed, send signal if previous != invalid { // if invalid changed, send signal
send = true send = true
obj.init.Dirty() // dirty
} }
if invalid { if invalid {
@@ -176,10 +172,8 @@ func (obj *SvcRes) Watch() error {
// loop so that we can see the changed invalid signal // loop so that we can see the changed invalid signal
obj.init.Logf("daemon reload") obj.init.Logf("daemon reload")
case event := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if err := obj.init.Read(event); err != nil { return nil
return err
}
} }
} else { } else {
if !activeSet { if !activeSet {
@@ -217,35 +211,31 @@ func (obj *SvcRes) Watch() error {
obj.init.Logf("stopped") obj.init.Logf("stopped")
} }
send = true send = true
obj.init.Dirty() // dirty
case err := <-subErrors: case err := <-subErrors:
return errwrap.Wrapf(err, "unknown %s error", obj) return errwrap.Wrapf(err, "unknown %s error", obj)
case event := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if err := obj.init.Read(event); err != nil { return nil
return err
}
} }
} }
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
// CheckApply checks the resource state and applies the resource if the bool // CheckApply checks the resource state and applies the resource if the bool
// input is true. It returns error info and if the state check passed or not. // input is true. It returns error info and if the state check passed or not.
func (obj *SvcRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *SvcRes) CheckApply(apply bool) (bool, error) {
if !systemdUtil.IsRunningSystemd() { if !systemdUtil.IsRunningSystemd() {
return false, fmt.Errorf("systemd is not running") return false, fmt.Errorf("systemd is not running")
} }
var conn *systemd.Conn var conn *systemd.Conn
var err error
if obj.Session { if obj.Session {
conn, err = systemd.NewUserConnection() // user session conn, err = systemd.NewUserConnection() // user session
} else { } else {
@@ -343,7 +333,12 @@ func (obj *SvcRes) CheckApply(apply bool) (checkOK bool, err error) {
if &status == nil { if &status == nil {
return false, fmt.Errorf("systemd service action result is nil") return false, fmt.Errorf("systemd service action result is nil")
} }
if status != "done" { switch status {
case "done":
// pass
case "failed":
return false, fmt.Errorf("svc failed (selinux?)")
default:
return false, fmt.Errorf("unknown systemd return string: %v", status) return false, fmt.Errorf("unknown systemd return string: %v", status)
} }
@@ -359,31 +354,23 @@ func (obj *SvcRes) CheckApply(apply bool) (checkOK bool, err error) {
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *SvcRes) Cmp(r engine.Res) error { func (obj *SvcRes) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *SvcRes) Compare(r engine.Res) bool {
// we can only compare SvcRes to others of the same resource kind // we can only compare SvcRes to others of the same resource kind
res, ok := r.(*SvcRes) res, ok := r.(*SvcRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.State != res.State { if obj.State != res.State {
return false return fmt.Errorf("the State differs")
} }
if obj.Startup != res.Startup { if obj.Startup != res.Startup {
return false return fmt.Errorf("the Startup differs")
} }
if obj.Session != res.Session { if obj.Session != res.Session {
return false return fmt.Errorf("the Session differs")
} }
return true return nil
} }
// SvcUID is the UID struct for SvcRes. // SvcUID is the UID struct for SvcRes.

126
engine/resources/test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -125,35 +125,19 @@ func (obj *TestRes) Close() error {
// Watch is the primary listener for this resource and it outputs events. // Watch is the primary listener for this resource and it outputs events.
func (obj *TestRes) Watch() error { func (obj *TestRes) Watch() error {
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event?
for {
select { select {
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil
}
if err := obj.init.Read(event); err != nil {
return err
}
} }
// do all our event sending all together to avoid duplicate msgs //obj.init.Event() // notify engine of an event (this can block)
if send {
send = false return nil
if err := obj.init.Event(); err != nil {
return err // exit if requested
}
}
}
} }
// CheckApply method for Test resource. Does nothing, returns happy! // CheckApply method for Test resource. Does nothing, returns happy!
func (obj *TestRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *TestRes) CheckApply(apply bool) (bool, error) {
for key, val := range obj.init.Recv() { for key, val := range obj.init.Recv() {
obj.init.Logf("CheckApply: Received `%s`, changed: %t", key, val.Changed) obj.init.Logf("CheckApply: Received `%s`, changed: %t", key, val.Changed)
} }
@@ -215,25 +199,17 @@ func (obj *TestRes) CheckApply(apply bool) (checkOK bool, err error) {
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *TestRes) Cmp(r engine.Res) error { func (obj *TestRes) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *TestRes) Compare(r engine.Res) bool {
// we can only compare TestRes to others of the same resource kind // we can only compare TestRes to others of the same resource kind
res, ok := r.(*TestRes) res, ok := r.(*TestRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
//if obj.Name != res.Name { //if obj.Name != res.Name {
// return false // return false
//} //}
if obj.CompareFail || res.CompareFail { if obj.CompareFail || res.CompareFail {
return false return fmt.Errorf("the CompareFail is true")
} }
// TODO: yes, I know the long manual version is absurd, but I couldn't // TODO: yes, I know the long manual version is absurd, but I couldn't
@@ -244,145 +220,145 @@ func (obj *TestRes) Compare(r engine.Res) bool {
//} //}
if obj.Bool != res.Bool { if obj.Bool != res.Bool {
return false return fmt.Errorf("the Bool differs")
} }
if obj.Str != res.Str { if obj.Str != res.Str {
return false return fmt.Errorf("the Str differs")
} }
if obj.Int != res.Int { if obj.Int != res.Int {
return false return fmt.Errorf("the Str differs")
} }
if obj.Int8 != res.Int8 { if obj.Int8 != res.Int8 {
return false return fmt.Errorf("the Int8 differs")
} }
if obj.Int16 != res.Int16 { if obj.Int16 != res.Int16 {
return false return fmt.Errorf("the Int16 differs")
} }
if obj.Int32 != res.Int32 { if obj.Int32 != res.Int32 {
return false return fmt.Errorf("the Int32 differs")
} }
if obj.Int64 != res.Int64 { if obj.Int64 != res.Int64 {
return false return fmt.Errorf("the Int64 differs")
} }
if obj.Uint != res.Uint { if obj.Uint != res.Uint {
return false return fmt.Errorf("the Uint differs")
} }
if obj.Uint8 != res.Uint8 { if obj.Uint8 != res.Uint8 {
return false return fmt.Errorf("the Uint8 differs")
} }
if obj.Uint16 != res.Uint16 { if obj.Uint16 != res.Uint16 {
return false return fmt.Errorf("the Uint16 differs")
} }
if obj.Uint32 != res.Uint32 { if obj.Uint32 != res.Uint32 {
return false return fmt.Errorf("the Uint32 differs")
} }
if obj.Uint64 != res.Uint64 { if obj.Uint64 != res.Uint64 {
return false return fmt.Errorf("the Uint64 differs")
} }
//if obj.Uintptr //if obj.Uintptr
if obj.Byte != res.Byte { if obj.Byte != res.Byte {
return false return fmt.Errorf("the Byte differs")
} }
if obj.Rune != res.Rune { if obj.Rune != res.Rune {
return false return fmt.Errorf("the Rune differs")
} }
if obj.Float32 != res.Float32 { if obj.Float32 != res.Float32 {
return false return fmt.Errorf("the Float32 differs")
} }
if obj.Float64 != res.Float64 { if obj.Float64 != res.Float64 {
return false return fmt.Errorf("the Float64 differs")
} }
if obj.Complex64 != res.Complex64 { if obj.Complex64 != res.Complex64 {
return false return fmt.Errorf("the Complex64 differs")
} }
if obj.Complex128 != res.Complex128 { if obj.Complex128 != res.Complex128 {
return false return fmt.Errorf("the Complex128 differs")
} }
if (obj.BoolPtr == nil) != (res.BoolPtr == nil) { // xor if (obj.BoolPtr == nil) != (res.BoolPtr == nil) { // xor
return false return fmt.Errorf("the BoolPtr differs")
} }
if obj.BoolPtr != nil && res.BoolPtr != nil { if obj.BoolPtr != nil && res.BoolPtr != nil {
if *obj.BoolPtr != *res.BoolPtr { // compare if *obj.BoolPtr != *res.BoolPtr { // compare
return false return fmt.Errorf("the BoolPtr differs")
} }
} }
if (obj.StringPtr == nil) != (res.StringPtr == nil) { // xor if (obj.StringPtr == nil) != (res.StringPtr == nil) { // xor
return false return fmt.Errorf("the StringPtr differs")
} }
if obj.StringPtr != nil && res.StringPtr != nil { if obj.StringPtr != nil && res.StringPtr != nil {
if *obj.StringPtr != *res.StringPtr { // compare if *obj.StringPtr != *res.StringPtr { // compare
return false return fmt.Errorf("the StringPtr differs")
} }
} }
if (obj.Int64Ptr == nil) != (res.Int64Ptr == nil) { // xor if (obj.Int64Ptr == nil) != (res.Int64Ptr == nil) { // xor
return false return fmt.Errorf("the Int64Ptr differs")
} }
if obj.Int64Ptr != nil && res.Int64Ptr != nil { if obj.Int64Ptr != nil && res.Int64Ptr != nil {
if *obj.Int64Ptr != *res.Int64Ptr { // compare if *obj.Int64Ptr != *res.Int64Ptr { // compare
return false return fmt.Errorf("the Int64Ptr differs")
} }
} }
if (obj.Int8Ptr == nil) != (res.Int8Ptr == nil) { // xor if (obj.Int8Ptr == nil) != (res.Int8Ptr == nil) { // xor
return false return fmt.Errorf("the Int8Ptr differs")
} }
if obj.Int8Ptr != nil && res.Int8Ptr != nil { if obj.Int8Ptr != nil && res.Int8Ptr != nil {
if *obj.Int8Ptr != *res.Int8Ptr { // compare if *obj.Int8Ptr != *res.Int8Ptr { // compare
return false return fmt.Errorf("the Int8Ptr differs")
} }
} }
if (obj.Uint8Ptr == nil) != (res.Uint8Ptr == nil) { // xor if (obj.Uint8Ptr == nil) != (res.Uint8Ptr == nil) { // xor
return false return fmt.Errorf("the Uint8Ptr differs")
} }
if obj.Uint8Ptr != nil && res.Uint8Ptr != nil { if obj.Uint8Ptr != nil && res.Uint8Ptr != nil {
if *obj.Uint8Ptr != *res.Uint8Ptr { // compare if *obj.Uint8Ptr != *res.Uint8Ptr { // compare
return false return fmt.Errorf("the Uint8Ptr differs")
} }
} }
if !reflect.DeepEqual(obj.Int8PtrPtrPtr, res.Int8PtrPtrPtr) { if !reflect.DeepEqual(obj.Int8PtrPtrPtr, res.Int8PtrPtrPtr) {
return false return fmt.Errorf("the Int8PtrPtrPtr differs")
} }
if !reflect.DeepEqual(obj.SliceString, res.SliceString) { if !reflect.DeepEqual(obj.SliceString, res.SliceString) {
return false return fmt.Errorf("the SliceString differs")
} }
if !reflect.DeepEqual(obj.MapIntFloat, res.MapIntFloat) { if !reflect.DeepEqual(obj.MapIntFloat, res.MapIntFloat) {
return false return fmt.Errorf("the MapIntFloat differs")
} }
if !reflect.DeepEqual(obj.MixedStruct, res.MixedStruct) { if !reflect.DeepEqual(obj.MixedStruct, res.MixedStruct) {
return false return fmt.Errorf("the MixedStruct differs")
} }
if !reflect.DeepEqual(obj.Interface, res.Interface) { if !reflect.DeepEqual(obj.Interface, res.Interface) {
return false return fmt.Errorf("the Interface differs")
} }
if obj.AnotherStr != res.AnotherStr { if obj.AnotherStr != res.AnotherStr {
return false return fmt.Errorf("the AnotherStr differs")
} }
if obj.ValidateBool != res.ValidateBool { if obj.ValidateBool != res.ValidateBool {
return false return fmt.Errorf("the ValidateBool differs")
} }
if obj.ValidateError != res.ValidateError { if obj.ValidateError != res.ValidateError {
return false return fmt.Errorf("the ValidateError differs")
} }
if obj.AlwaysGroup != res.AlwaysGroup { if obj.AlwaysGroup != res.AlwaysGroup {
return false return fmt.Errorf("the AlwaysGroup differs")
} }
if obj.SendValue != res.SendValue { if obj.SendValue != res.SendValue {
return false return fmt.Errorf("the SendValue differs")
} }
if obj.Comment != res.Comment { if obj.Comment != res.Comment {
return false return fmt.Errorf("the Comment differs")
} }
return true return nil
} }
// TestUID is the UID struct for TestRes. // TestUID is the UID struct for TestRes.
@@ -417,8 +393,8 @@ func (obj *TestRes) GroupCmp(r engine.GroupableRes) error {
// TestSends is the struct of data which is sent after a successful Apply. // TestSends is the struct of data which is sent after a successful Apply.
type TestSends struct { type TestSends struct {
// Hello is some value being sent. // Hello is some value being sent.
Hello *string Hello *string `lang:"hello"`
Answer int // some other value being sent Answer int `lang:"answer"` // some other value being sent
} }
// Sends represents the default struct of values we can send using Send/Recv. // Sends represents the default struct of values we can send using Send/Recv.

2
engine/resources/test_test.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

32
engine/resources/timer.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -75,10 +75,7 @@ func (obj *TimerRes) Watch() error {
obj.ticker = obj.newTicker() obj.ticker = obj.newTicker()
defer obj.ticker.Stop() defer obj.ticker.Stop()
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? var send = false // send event?
for { for {
@@ -87,20 +84,13 @@ func (obj *TimerRes) Watch() error {
send = true send = true
obj.init.Logf("received tick") obj.init.Logf("received tick")
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
@@ -123,25 +113,17 @@ func (obj *TimerRes) CheckApply(apply bool) (bool, error) {
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *TimerRes) Cmp(r engine.Res) error { func (obj *TimerRes) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *TimerRes) Compare(r engine.Res) bool {
// we can only compare TimerRes to others of the same resource kind // we can only compare TimerRes to others of the same resource kind
res, ok := r.(*TimerRes) res, ok := r.(*TimerRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.Interval != res.Interval { if obj.Interval != res.Interval {
return false return fmt.Errorf("the Interval differs")
} }
return true return nil
} }
// TimerUID is the UID struct for TimerRes. // TimerUID is the UID struct for TimerRes.

60
engine/resources/user.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -30,8 +30,7 @@ import (
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
"github.com/purpleidea/mgmt/recwatch" "github.com/purpleidea/mgmt/recwatch"
"github.com/purpleidea/mgmt/util/errwrap"
errwrap "github.com/pkg/errors"
) )
func init() { func init() {
@@ -119,10 +118,7 @@ func (obj *UserRes) Watch() error {
} }
defer obj.recWatcher.Close() defer obj.recWatcher.Close()
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? var send = false // send event?
for { for {
@@ -142,29 +138,21 @@ func (obj *UserRes) Watch() error {
obj.init.Logf("Event(%s): %v", event.Body.Name, event.Body.Op) obj.init.Logf("Event(%s): %v", event.Body.Name, event.Body.Op)
} }
send = true send = true
obj.init.Dirty() // dirty
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
// CheckApply method for User resource. // CheckApply method for User resource.
func (obj *UserRes) CheckApply(apply bool) (checkOK bool, err error) { func (obj *UserRes) CheckApply(apply bool) (bool, error) {
obj.init.Logf("CheckApply(%t)", apply) obj.init.Logf("CheckApply(%t)", apply)
var exists = true var exists = true
@@ -285,45 +273,37 @@ func (obj *UserRes) CheckApply(apply bool) (checkOK bool, err error) {
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *UserRes) Cmp(r engine.Res) error { func (obj *UserRes) Cmp(r engine.Res) error {
if !obj.Compare(r) {
return fmt.Errorf("did not compare")
}
return nil
}
// Compare two resources and return if they are equivalent.
func (obj *UserRes) Compare(r engine.Res) bool {
// we can only compare UserRes to others of the same resource kind // we can only compare UserRes to others of the same resource kind
res, ok := r.(*UserRes) res, ok := r.(*UserRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.State != res.State { if obj.State != res.State {
return false return fmt.Errorf("the State differs")
} }
if (obj.UID == nil) != (res.UID == nil) { if (obj.UID == nil) != (res.UID == nil) {
return false return fmt.Errorf("the UID differs")
} }
if obj.UID != nil && res.UID != nil { if obj.UID != nil && res.UID != nil {
if *obj.UID != *res.UID { if *obj.UID != *res.UID {
return false return fmt.Errorf("the UID differs")
} }
} }
if (obj.GID == nil) != (res.GID == nil) { if (obj.GID == nil) != (res.GID == nil) {
return false return fmt.Errorf("the GID differs")
} }
if obj.GID != nil && res.GID != nil { if obj.GID != nil && res.GID != nil {
if *obj.GID != *res.GID { if *obj.GID != *res.GID {
return false return fmt.Errorf("the GID differs")
} }
} }
if (obj.Groups == nil) != (res.Groups == nil) { if (obj.Groups == nil) != (res.Groups == nil) {
return false return fmt.Errorf("the Group differs")
} }
if obj.Groups != nil && res.Groups != nil { if obj.Groups != nil && res.Groups != nil {
if len(obj.Groups) != len(res.Groups) { if len(obj.Groups) != len(res.Groups) {
return false return fmt.Errorf("the Group differs")
} }
objGroups := obj.Groups objGroups := obj.Groups
resGroups := res.Groups resGroups := res.Groups
@@ -331,22 +311,22 @@ func (obj *UserRes) Compare(r engine.Res) bool {
sort.Strings(resGroups) sort.Strings(resGroups)
for i := range objGroups { for i := range objGroups {
if objGroups[i] != resGroups[i] { if objGroups[i] != resGroups[i] {
return false return fmt.Errorf("the Group differs at index: %d", i)
} }
} }
} }
if (obj.HomeDir == nil) != (res.HomeDir == nil) { if (obj.HomeDir == nil) != (res.HomeDir == nil) {
return false return fmt.Errorf("the HomeDirs differs")
} }
if obj.HomeDir != nil && res.HomeDir != nil { if obj.HomeDir != nil && res.HomeDir != nil {
if *obj.HomeDir != *obj.HomeDir { if *obj.HomeDir != *res.HomeDir {
return false return fmt.Errorf("the HomeDir differs")
} }
} }
if obj.AllowDuplicateUID != res.AllowDuplicateUID { if obj.AllowDuplicateUID != res.AllowDuplicateUID {
return false return fmt.Errorf("the AllowDuplicateUID differs")
} }
return true return nil
} }
// UserUID is the UID struct for UserRes. // UserUID is the UID struct for UserRes.

431
engine/resources/virt.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -29,11 +29,12 @@ import (
"github.com/purpleidea/mgmt/engine" "github.com/purpleidea/mgmt/engine"
"github.com/purpleidea/mgmt/engine/traits" "github.com/purpleidea/mgmt/engine/traits"
engineUtil "github.com/purpleidea/mgmt/engine/util"
"github.com/purpleidea/mgmt/util" "github.com/purpleidea/mgmt/util"
"github.com/purpleidea/mgmt/util/errwrap"
"github.com/libvirt/libvirt-go" "github.com/libvirt/libvirt-go"
libvirtxml "github.com/libvirt/libvirt-go-xml" libvirtxml "github.com/libvirt/libvirt-go-xml"
errwrap "github.com/pkg/errors"
) )
func init() { func init() {
@@ -65,30 +66,53 @@ const (
// VirtRes is a libvirt resource. A transient virt resource, which has its state // VirtRes is a libvirt resource. A transient virt resource, which has its state
// set to `shutoff` is one which does not exist. The parallel equivalent is a // set to `shutoff` is one which does not exist. The parallel equivalent is a
// file resource which removes a particular path. // file resource which removes a particular path.
// TODO: some values inside here should be enum's!
type VirtRes struct { type VirtRes struct {
traits.Base // add the base methods without re-implementation traits.Base // add the base methods without re-implementation
traits.Refreshable traits.Refreshable
init *engine.Init init *engine.Init
URI string `yaml:"uri"` // connection uri, eg: qemu:///session // URI is the libvirt connection URI, eg: `qemu:///session`.
State string `yaml:"state"` // running, paused, shutoff URI string `lang:"uri" yaml:"uri"`
Transient bool `yaml:"transient"` // defined (false) or undefined (true) // State is the desired vm state. Possible values include: `running`,
CPUs uint `yaml:"cpus"` // `paused` and `shutoff`.
MaxCPUs uint `yaml:"maxcpus"` State string `lang:"state" yaml:"state"`
Memory uint64 `yaml:"memory"` // in KBytes // Transient is whether the vm is defined (false) or undefined (true).
OSInit string `yaml:"osinit"` // init used by lxc Transient bool `lang:"transient" yaml:"transient"`
Boot []string `yaml:"boot"` // boot order. values: fd, hd, cdrom, network
Disk []diskDevice `yaml:"disk"`
CDRom []cdRomDevice `yaml:"cdrom"`
Network []networkDevice `yaml:"network"`
Filesystem []filesystemDevice `yaml:"filesystem"`
Auth *VirtAuth `yaml:"auth"`
HotCPUs bool `yaml:"hotcpus"` // allow hotplug of cpus? // CPUs is the desired cpu count of the machine.
// FIXME: values here should be enum's! CPUs uint `lang:"cpus" yaml:"cpus"`
RestartOnDiverge string `yaml:"restartondiverge"` // restart policy: "ignore", "ifneeded", "error" // MaxCPUs is the maximum number of cpus allowed in the machine. You
RestartOnRefresh bool `yaml:"restartonrefresh"` // restart on refresh? // need to set this so that on boot the `hardware` knows how many cpu
// `slots` it might need to make room for.
MaxCPUs uint `lang:"maxcpus" yaml:"maxcpus"`
// HotCPUs specifies whether we can hot plug and unplug cpus.
HotCPUs bool `lang:"hotcpus" yaml:"hotcpus"`
// Memory is the size in KBytes of memory to include in the machine.
Memory uint64 `lang:"memory" yaml:"memory"`
// OSInit is the init used by lxc.
OSInit string `lang:"osinit" yaml:"osinit"`
// Boot is the boot order. Values are `fd`, `hd`, `cdrom` and `network`.
Boot []string `lang:"boot" yaml:"boot"`
// Disk is the list of disk devices to include.
Disk []*DiskDevice `lang:"disk" yaml:"disk"`
// CdRom is the list of cdrom devices to include.
CDRom []*CDRomDevice `lang:"cdrom" yaml:"cdrom"`
// Network is the list of network devices to include.
Network []*NetworkDevice `lang:"network" yaml:"network"`
// Filesystem is the list of file system devices to include.
Filesystem []*FilesystemDevice `lang:"filesystem" yaml:"filesystem"`
// Auth points to the libvirt credentials to use if any are necessary.
Auth *VirtAuth `lang:"auth" yaml:"auth"`
// RestartOnDiverge is the restart policy, and can be: `ignore`,
// `ifneeded` or `error`.
RestartOnDiverge string `lang:"restartondiverge" yaml:"restartondiverge"`
// RestartOnRefresh specifies if we restart on refresh signal.
RestartOnRefresh bool `lang:"restartonrefresh" yaml:"restartonrefresh"`
wg *sync.WaitGroup wg *sync.WaitGroup
conn *libvirt.Connect conn *libvirt.Connect
@@ -103,8 +127,26 @@ type VirtRes struct {
// VirtAuth is used to pass credentials to libvirt. // VirtAuth is used to pass credentials to libvirt.
type VirtAuth struct { type VirtAuth struct {
Username string `yaml:"username"` Username string `lang:"username" yaml:"username"`
Password string `yaml:"password"` Password string `lang:"password" yaml:"password"`
}
// Cmp compares two VirtAuth structs. It errors if they are not identical.
func (obj *VirtAuth) Cmp(auth *VirtAuth) error {
if (obj == nil) != (auth == nil) { // xor
return fmt.Errorf("the VirtAuth differs")
}
if obj == nil && auth == nil {
return nil
}
if obj.Username != auth.Username {
return fmt.Errorf("the Username differs")
}
if obj.Password != auth.Password {
return fmt.Errorf("the Password differs")
}
return nil
} }
// Default returns some sensible defaults for this resource. // Default returns some sensible defaults for this resource.
@@ -138,7 +180,7 @@ func (obj *VirtRes) Init(init *engine.Init) error {
var u *url.URL var u *url.URL
var err error var err error
if u, err = url.Parse(obj.URI); err != nil { if u, err = url.Parse(obj.URI); err != nil {
return errwrap.Wrapf(err, "%s: Parsing URI failed: %s", obj, obj.URI) return errwrap.Wrapf(err, "parsing URI (`%s`) failed", obj.URI)
} }
switch u.Scheme { switch u.Scheme {
case "lxc": case "lxc":
@@ -149,7 +191,7 @@ func (obj *VirtRes) Init(init *engine.Init) error {
obj.conn, err = obj.connect() // gets closed in Close method of Res API obj.conn, err = obj.connect() // gets closed in Close method of Res API
if err != nil { if err != nil {
return errwrap.Wrapf(err, "%s: Connection to libvirt failed in init", obj) return errwrap.Wrapf(err, "connection to libvirt failed in init")
} }
// check for hard to change properties // check for hard to change properties
@@ -157,14 +199,14 @@ func (obj *VirtRes) Init(init *engine.Init) error {
if err == nil { if err == nil {
defer dom.Free() defer dom.Free()
} else if !isNotFound(err) { } else if !isNotFound(err) {
return errwrap.Wrapf(err, "%s: Could not lookup on init", obj) return errwrap.Wrapf(err, "could not lookup on init")
} }
if err == nil { if err == nil {
// maxCPUs, err := dom.GetMaxVcpus() // maxCPUs, err := dom.GetMaxVcpus()
i, err := dom.GetVcpusFlags(libvirt.DOMAIN_VCPU_MAXIMUM) i, err := dom.GetVcpusFlags(libvirt.DOMAIN_VCPU_MAXIMUM)
if err != nil { if err != nil {
return errwrap.Wrapf(err, "%s: Could not lookup MaxCPUs on init", obj) return errwrap.Wrapf(err, "could not lookup MaxCPUs on init")
} }
maxCPUs := uint(i) maxCPUs := uint(i)
if obj.MaxCPUs != maxCPUs { // max cpu slots is hard to change if obj.MaxCPUs != maxCPUs { // max cpu slots is hard to change
@@ -177,11 +219,11 @@ func (obj *VirtRes) Init(init *engine.Init) error {
// event handlers so that we don't miss any events via race? // event handlers so that we don't miss any events via race?
xmlDesc, err := dom.GetXMLDesc(0) // 0 means no flags xmlDesc, err := dom.GetXMLDesc(0) // 0 means no flags
if err != nil { if err != nil {
return errwrap.Wrapf(err, "%s: Could not GetXMLDesc on init", obj) return errwrap.Wrapf(err, "could not GetXMLDesc on init")
} }
domXML := &libvirtxml.Domain{} domXML := &libvirtxml.Domain{}
if err := domXML.Unmarshal(xmlDesc); err != nil { if err := domXML.Unmarshal(xmlDesc); err != nil {
return errwrap.Wrapf(err, "%s: Could not unmarshal XML on init", obj) return errwrap.Wrapf(err, "could not unmarshal XML on init")
} }
// guest agent: domain->devices->channel->target->state == connected? // guest agent: domain->devices->channel->target->state == connected?
@@ -326,10 +368,7 @@ func (obj *VirtRes) Watch() error {
} }
defer obj.conn.DomainEventDeregister(gaCallbackID) defer obj.conn.DomainEventDeregister(gaCallbackID)
// notify engine that we're running obj.init.Running() // when started, notify engine that we're running
if err := obj.init.Running(); err != nil {
return err // exit if requested
}
var send = false // send event? var send = false // send event?
for { for {
@@ -340,31 +379,26 @@ func (obj *VirtRes) Watch() error {
switch event { switch event {
case libvirt.DOMAIN_EVENT_DEFINED: case libvirt.DOMAIN_EVENT_DEFINED:
if obj.Transient { if obj.Transient {
obj.init.Dirty() // dirty
send = true send = true
} }
case libvirt.DOMAIN_EVENT_UNDEFINED: case libvirt.DOMAIN_EVENT_UNDEFINED:
if !obj.Transient { if !obj.Transient {
obj.init.Dirty() // dirty
send = true send = true
} }
case libvirt.DOMAIN_EVENT_STARTED: case libvirt.DOMAIN_EVENT_STARTED:
fallthrough fallthrough
case libvirt.DOMAIN_EVENT_RESUMED: case libvirt.DOMAIN_EVENT_RESUMED:
if obj.State != "running" { if obj.State != "running" {
obj.init.Dirty() // dirty
send = true send = true
} }
case libvirt.DOMAIN_EVENT_SUSPENDED: case libvirt.DOMAIN_EVENT_SUSPENDED:
if obj.State != "paused" { if obj.State != "paused" {
obj.init.Dirty() // dirty
send = true send = true
} }
case libvirt.DOMAIN_EVENT_STOPPED: case libvirt.DOMAIN_EVENT_STOPPED:
fallthrough fallthrough
case libvirt.DOMAIN_EVENT_SHUTDOWN: case libvirt.DOMAIN_EVENT_SHUTDOWN:
if obj.State != "shutoff" { if obj.State != "shutoff" {
obj.init.Dirty() // dirty
send = true send = true
} }
processExited = true processExited = true
@@ -375,7 +409,6 @@ func (obj *VirtRes) Watch() error {
// verify, detect and patch appropriately! // verify, detect and patch appropriately!
fallthrough fallthrough
case libvirt.DOMAIN_EVENT_CRASHED: case libvirt.DOMAIN_EVENT_CRASHED:
obj.init.Dirty() // dirty
send = true send = true
processExited = true // FIXME: is this okay for PMSUSPENDED ? processExited = true // FIXME: is this okay for PMSUSPENDED ?
} }
@@ -390,40 +423,32 @@ func (obj *VirtRes) Watch() error {
if state == libvirt.CONNECT_DOMAIN_EVENT_AGENT_LIFECYCLE_STATE_CONNECTED { if state == libvirt.CONNECT_DOMAIN_EVENT_AGENT_LIFECYCLE_STATE_CONNECTED {
obj.guestAgentConnected = true obj.guestAgentConnected = true
obj.init.Dirty() // dirty
send = true send = true
obj.init.Logf("Guest agent connected") obj.init.Logf("guest agent connected")
} else if state == libvirt.CONNECT_DOMAIN_EVENT_AGENT_LIFECYCLE_STATE_DISCONNECTED { } else if state == libvirt.CONNECT_DOMAIN_EVENT_AGENT_LIFECYCLE_STATE_DISCONNECTED {
obj.guestAgentConnected = false obj.guestAgentConnected = false
// ignore CONNECT_DOMAIN_EVENT_AGENT_LIFECYCLE_REASON_DOMAIN_STARTED // ignore CONNECT_DOMAIN_EVENT_AGENT_LIFECYCLE_REASON_DOMAIN_STARTED
// events because they just tell you that guest agent channel was added // events because they just tell you that guest agent channel was added
if reason == libvirt.CONNECT_DOMAIN_EVENT_AGENT_LIFECYCLE_REASON_CHANNEL { if reason == libvirt.CONNECT_DOMAIN_EVENT_AGENT_LIFECYCLE_REASON_CHANNEL {
obj.init.Logf("Guest agent disconnected") obj.init.Logf("guest agent disconnected")
} }
} else { } else {
return fmt.Errorf("unknown %s guest agent state: %v", obj, state) return fmt.Errorf("unknown guest agent state: %v", state)
} }
case err := <-errorChan: case err := <-errorChan:
return fmt.Errorf("unknown %s libvirt error: %s", obj, err) return errwrap.Wrapf(err, "unknown libvirt error")
case event, ok := <-obj.init.Events: case <-obj.init.Done: // closed by the engine to signal shutdown
if !ok {
return nil return nil
} }
if err := obj.init.Read(event); err != nil {
return err
}
}
// do all our event sending all together to avoid duplicate msgs // do all our event sending all together to avoid duplicate msgs
if send { if send {
send = false send = false
if err := obj.init.Event(); err != nil { obj.init.Event() // notify engine of an event (this can block)
return err // exit if requested
}
} }
} }
} }
@@ -451,7 +476,7 @@ func (obj *VirtRes) domainCreate() (*libvirt.Domain, bool, error) {
if err != nil { if err != nil {
return dom, false, err // returned dom is invalid return dom, false, err // returned dom is invalid
} }
obj.init.Logf("Domain transient %s", state) obj.init.Logf("transient domain %s", state) // log the state
return dom, false, nil return dom, false, nil
} }
@@ -459,20 +484,20 @@ func (obj *VirtRes) domainCreate() (*libvirt.Domain, bool, error) {
if err != nil { if err != nil {
return dom, false, err // returned dom is invalid return dom, false, err // returned dom is invalid
} }
obj.init.Logf("Domain defined") obj.init.Logf("domain defined")
if obj.State == "running" { if obj.State == "running" {
if err := dom.Create(); err != nil { if err := dom.Create(); err != nil {
return dom, false, err return dom, false, err
} }
obj.init.Logf("Domain started") obj.init.Logf("domain started")
} }
if obj.State == "paused" { if obj.State == "paused" {
if err := dom.CreateWithFlags(libvirt.DOMAIN_START_PAUSED); err != nil { if err := dom.CreateWithFlags(libvirt.DOMAIN_START_PAUSED); err != nil {
return dom, false, err return dom, false, err
} }
obj.init.Logf("Domain created paused") obj.init.Logf("domain created paused")
} }
return dom, false, nil return dom, false, nil
@@ -500,7 +525,7 @@ func (obj *VirtRes) stateCheckApply(apply bool, dom *libvirt.Domain) (bool, erro
} }
if domInfo.State == libvirt.DOMAIN_BLOCKED { if domInfo.State == libvirt.DOMAIN_BLOCKED {
// TODO: what should happen? // TODO: what should happen?
return false, fmt.Errorf("domain %s is blocked", obj.Name()) return false, fmt.Errorf("domain is blocked")
} }
if !apply { if !apply {
return false, nil return false, nil
@@ -510,14 +535,14 @@ func (obj *VirtRes) stateCheckApply(apply bool, dom *libvirt.Domain) (bool, erro
return false, errwrap.Wrapf(err, "domain.Resume failed") return false, errwrap.Wrapf(err, "domain.Resume failed")
} }
checkOK = false checkOK = false
obj.init.Logf("Domain resumed") obj.init.Logf("domain resumed")
break break
} }
if err := dom.Create(); err != nil { if err := dom.Create(); err != nil {
return false, errwrap.Wrapf(err, "domain.Create failed") return false, errwrap.Wrapf(err, "domain.Create failed")
} }
checkOK = false checkOK = false
obj.init.Logf("Domain created") obj.init.Logf("domain created")
case "paused": case "paused":
if domInfo.State == libvirt.DOMAIN_PAUSED { if domInfo.State == libvirt.DOMAIN_PAUSED {
@@ -531,14 +556,14 @@ func (obj *VirtRes) stateCheckApply(apply bool, dom *libvirt.Domain) (bool, erro
return false, errwrap.Wrapf(err, "domain.Suspend failed") return false, errwrap.Wrapf(err, "domain.Suspend failed")
} }
checkOK = false checkOK = false
obj.init.Logf("Domain paused") obj.init.Logf("domain paused")
break break
} }
if err := dom.CreateWithFlags(libvirt.DOMAIN_START_PAUSED); err != nil { if err := dom.CreateWithFlags(libvirt.DOMAIN_START_PAUSED); err != nil {
return false, errwrap.Wrapf(err, "domain.CreateWithFlags failed") return false, errwrap.Wrapf(err, "domain.CreateWithFlags failed")
} }
checkOK = false checkOK = false
obj.init.Logf("Domain created paused") obj.init.Logf("domain created paused")
case "shutoff": case "shutoff":
if domInfo.State == libvirt.DOMAIN_SHUTOFF || domInfo.State == libvirt.DOMAIN_SHUTDOWN { if domInfo.State == libvirt.DOMAIN_SHUTOFF || domInfo.State == libvirt.DOMAIN_SHUTDOWN {
@@ -552,7 +577,7 @@ func (obj *VirtRes) stateCheckApply(apply bool, dom *libvirt.Domain) (bool, erro
return false, errwrap.Wrapf(err, "domain.Destroy failed") return false, errwrap.Wrapf(err, "domain.Destroy failed")
} }
checkOK = false checkOK = false
obj.init.Logf("Domain destroyed") obj.init.Logf("domain destroyed")
} }
return checkOK, nil return checkOK, nil
@@ -578,7 +603,7 @@ func (obj *VirtRes) attrCheckApply(apply bool, dom *libvirt.Domain) (bool, error
if err := dom.SetMemory(obj.Memory); err != nil { if err := dom.SetMemory(obj.Memory); err != nil {
return false, errwrap.Wrapf(err, "domain.SetMemory failed") return false, errwrap.Wrapf(err, "domain.SetMemory failed")
} }
obj.init.Logf("Memory changed to %d", obj.Memory) obj.init.Logf("memory changed to: %d", obj.Memory)
} }
// check cpus // check cpus
@@ -617,7 +642,7 @@ func (obj *VirtRes) attrCheckApply(apply bool, dom *libvirt.Domain) (bool, error
return false, errwrap.Wrapf(err, "domain.SetVcpus failed") return false, errwrap.Wrapf(err, "domain.SetVcpus failed")
} }
checkOK = false checkOK = false
obj.init.Logf("CPUs (hot) changed to %d", obj.CPUs) obj.init.Logf("cpus (hot) changed to: %d", obj.CPUs)
case libvirt.DOMAIN_SHUTOFF, libvirt.DOMAIN_SHUTDOWN: case libvirt.DOMAIN_SHUTOFF, libvirt.DOMAIN_SHUTDOWN:
if !obj.Transient { if !obj.Transient {
@@ -629,7 +654,7 @@ func (obj *VirtRes) attrCheckApply(apply bool, dom *libvirt.Domain) (bool, error
return false, errwrap.Wrapf(err, "domain.SetVcpus failed") return false, errwrap.Wrapf(err, "domain.SetVcpus failed")
} }
checkOK = false checkOK = false
obj.init.Logf("CPUs (cold) changed to %d", obj.CPUs) obj.init.Logf("cpus (cold) changed to: %d", obj.CPUs)
} }
default: default:
@@ -660,7 +685,7 @@ func (obj *VirtRes) attrCheckApply(apply bool, dom *libvirt.Domain) (bool, error
return false, errwrap.Wrapf(err, "domain.SetVcpus failed") return false, errwrap.Wrapf(err, "domain.SetVcpus failed")
} }
checkOK = false checkOK = false
obj.init.Logf("CPUs (guest) changed to %d", obj.CPUs) obj.init.Logf("cpus (guest) changed to: %d", obj.CPUs)
} }
} }
@@ -684,7 +709,7 @@ func (obj *VirtRes) domainShutdownSync(apply bool, dom *libvirt.Domain) (bool, e
return false, errwrap.Wrapf(err, "domain.GetInfo failed") return false, errwrap.Wrapf(err, "domain.GetInfo failed")
} }
if domInfo.State == libvirt.DOMAIN_SHUTOFF || domInfo.State == libvirt.DOMAIN_SHUTDOWN { if domInfo.State == libvirt.DOMAIN_SHUTOFF || domInfo.State == libvirt.DOMAIN_SHUTDOWN {
obj.init.Logf("Shutdown") obj.init.Logf("shutdown")
break break
} }
@@ -696,7 +721,7 @@ func (obj *VirtRes) domainShutdownSync(apply bool, dom *libvirt.Domain) (bool, e
obj.processExitChan = make(chan struct{}) obj.processExitChan = make(chan struct{})
// if machine shuts down before we call this, we error; // if machine shuts down before we call this, we error;
// this isn't ideal, but it happened due to user error! // this isn't ideal, but it happened due to user error!
obj.init.Logf("Running shutdown") obj.init.Logf("running shutdown")
if err := dom.Shutdown(); err != nil { if err := dom.Shutdown(); err != nil {
// FIXME: if machine is already shutdown completely, return early // FIXME: if machine is already shutdown completely, return early
return false, errwrap.Wrapf(err, "domain.Shutdown failed") return false, errwrap.Wrapf(err, "domain.Shutdown failed")
@@ -717,7 +742,7 @@ func (obj *VirtRes) domainShutdownSync(apply bool, dom *libvirt.Domain) (bool, e
// https://libvirt.org/formatdomain.html#elementsEvents // https://libvirt.org/formatdomain.html#elementsEvents
continue continue
case <-timeout: case <-timeout:
return false, fmt.Errorf("%s: didn't shutdown after %d seconds", obj, MaxShutdownDelayTimeout) return false, fmt.Errorf("didn't shutdown after %d seconds", MaxShutdownDelayTimeout)
} }
} }
@@ -727,8 +752,8 @@ func (obj *VirtRes) domainShutdownSync(apply bool, dom *libvirt.Domain) (bool, e
// CheckApply checks the resource state and applies the resource if the bool // CheckApply checks the resource state and applies the resource if the bool
// input is true. It returns error info and if the state check passed or not. // input is true. It returns error info and if the state check passed or not.
func (obj *VirtRes) CheckApply(apply bool) (bool, error) { func (obj *VirtRes) CheckApply(apply bool) (bool, error) {
if obj.conn == nil { if obj.conn == nil { // programming error?
panic("virt: CheckApply is being called with nil connection") return false, fmt.Errorf("got called with nil connection")
} }
// if we do the restart, we must flip the flag back to false as evidence // if we do the restart, we must flip the flag back to false as evidence
var restart bool // do we need to do a restart? var restart bool // do we need to do a restart?
@@ -789,7 +814,7 @@ func (obj *VirtRes) CheckApply(apply bool) (bool, error) {
if err := dom.Undefine(); err != nil { if err := dom.Undefine(); err != nil {
return false, errwrap.Wrapf(err, "domain.Undefine failed") return false, errwrap.Wrapf(err, "domain.Undefine failed")
} }
obj.init.Logf("Domain undefined") obj.init.Logf("domain undefined")
} else { } else {
domXML, err := dom.GetXMLDesc(libvirt.DOMAIN_XML_INACTIVE) domXML, err := dom.GetXMLDesc(libvirt.DOMAIN_XML_INACTIVE)
if err != nil { if err != nil {
@@ -798,7 +823,7 @@ func (obj *VirtRes) CheckApply(apply bool) (bool, error) {
if _, err = obj.conn.DomainDefineXML(domXML); err != nil { if _, err = obj.conn.DomainDefineXML(domXML); err != nil {
return false, errwrap.Wrapf(err, "conn.DomainDefineXML failed") return false, errwrap.Wrapf(err, "conn.DomainDefineXML failed")
} }
obj.init.Logf("Domain defined") obj.init.Logf("domain defined")
} }
checkOK = false checkOK = false
} }
@@ -846,7 +871,7 @@ func (obj *VirtRes) CheckApply(apply bool) (bool, error) {
// we had to do a restart, we didn't, and we should error if it was needed // we had to do a restart, we didn't, and we should error if it was needed
if obj.restartScheduled && restart == true && obj.RestartOnDiverge == "error" { if obj.restartScheduled && restart == true && obj.RestartOnDiverge == "error" {
return false, fmt.Errorf("%s: needed restart but didn't! (RestartOnDiverge: %v)", obj, obj.RestartOnDiverge) return false, fmt.Errorf("needed restart but didn't! (RestartOnDiverge: %s)", obj.RestartOnDiverge)
} }
return checkOK, nil // w00t return checkOK, nil // w00t
@@ -970,44 +995,56 @@ type virtDevice interface {
GetXML(idx int) string GetXML(idx int) string
} }
type diskDevice struct { // DiskDevice represents a disk that is attached to the virt machine.
Source string `yaml:"source"` type DiskDevice struct {
Type string `yaml:"type"` Source string `lang:"source" yaml:"source"`
Type string `lang:"type" yaml:"type"`
} }
type cdRomDevice struct { // GetXML returns the XML representation of this device.
Source string `yaml:"source"` func (obj *DiskDevice) GetXML(idx int) string {
Type string `yaml:"type"` source, _ := util.ExpandHome(obj.Source) // TODO: should we handle errors?
}
type networkDevice struct {
Name string `yaml:"name"`
MAC string `yaml:"mac"`
}
type filesystemDevice struct {
Access string `yaml:"access"`
Source string `yaml:"source"`
Target string `yaml:"target"`
ReadOnly bool `yaml:"read_only"`
}
func (d *diskDevice) GetXML(idx int) string {
source, _ := util.ExpandHome(d.Source) // TODO: should we handle errors?
var b string var b string
b += "<disk type='file' device='disk'>" b += "<disk type='file' device='disk'>"
b += fmt.Sprintf("<driver name='qemu' type='%s'/>", d.Type) b += fmt.Sprintf("<driver name='qemu' type='%s'/>", obj.Type)
b += fmt.Sprintf("<source file='%s'/>", source) b += fmt.Sprintf("<source file='%s'/>", source)
b += fmt.Sprintf("<target dev='vd%s' bus='virtio'/>", util.NumToAlpha(idx)) b += fmt.Sprintf("<target dev='vd%s' bus='virtio'/>", util.NumToAlpha(idx))
b += "</disk>" b += "</disk>"
return b return b
} }
func (d *cdRomDevice) GetXML(idx int) string { // Cmp compares two DiskDevice's and returns an error if they are not
source, _ := util.ExpandHome(d.Source) // TODO: should we handle errors? // equivalent.
func (obj *DiskDevice) Cmp(dev *DiskDevice) error {
if (obj == nil) != (dev == nil) { // xor
return fmt.Errorf("the DiskDevice differs")
}
if obj == nil && dev == nil {
return nil
}
if obj.Source != dev.Source {
return fmt.Errorf("the Source differs")
}
if obj.Type != dev.Type {
return fmt.Errorf("the Type differs")
}
return nil
}
// CDRomDevice represents a CDRom device that is attached to the virt machine.
type CDRomDevice struct {
Source string `lang:"source" yaml:"source"`
Type string `lang:"type" yaml:"type"`
}
// GetXML returns the XML representation of this device.
func (obj *CDRomDevice) GetXML(idx int) string {
source, _ := util.ExpandHome(obj.Source) // TODO: should we handle errors?
var b string var b string
b += "<disk type='file' device='cdrom'>" b += "<disk type='file' device='cdrom'>"
b += fmt.Sprintf("<driver name='qemu' type='%s'/>", d.Type) b += fmt.Sprintf("<driver name='qemu' type='%s'/>", obj.Type)
b += fmt.Sprintf("<source file='%s'/>", source) b += fmt.Sprintf("<source file='%s'/>", source)
b += fmt.Sprintf("<target dev='hd%s' bus='ide'/>", util.NumToAlpha(idx)) b += fmt.Sprintf("<target dev='hd%s' bus='ide'/>", util.NumToAlpha(idx))
b += "<readonly/>" b += "<readonly/>"
@@ -1015,68 +1052,147 @@ func (d *cdRomDevice) GetXML(idx int) string {
return b return b
} }
func (d *networkDevice) GetXML(idx int) string { // Cmp compares two CDRomDevice's and returns an error if they are not
if d.MAC == "" { // equivalent.
d.MAC = randMAC() func (obj *CDRomDevice) Cmp(dev *CDRomDevice) error {
if (obj == nil) != (dev == nil) { // xor
return fmt.Errorf("the CDRomDevice differs")
}
if obj == nil && dev == nil {
return nil
}
if obj.Source != dev.Source {
return fmt.Errorf("the Source differs")
}
if obj.Type != dev.Type {
return fmt.Errorf("the Type differs")
}
return nil
}
// NetworkDevice represents a network card that is attached to the virt machine.
type NetworkDevice struct {
Name string `lang:"name" yaml:"name"`
MAC string `lang:"mac" yaml:"mac"`
}
// GetXML returns the XML representation of this device.
func (obj *NetworkDevice) GetXML(idx int) string {
if obj.MAC == "" {
obj.MAC = randMAC()
} }
var b string var b string
b += "<interface type='network'>" b += "<interface type='network'>"
b += fmt.Sprintf("<mac address='%s'/>", d.MAC) b += fmt.Sprintf("<mac address='%s'/>", obj.MAC)
b += fmt.Sprintf("<source network='%s'/>", d.Name) b += fmt.Sprintf("<source network='%s'/>", obj.Name)
b += "</interface>" b += "</interface>"
return b return b
} }
func (d *filesystemDevice) GetXML(idx int) string { // Cmp compares two NetworkDevice's and returns an error if they are not
source, _ := util.ExpandHome(d.Source) // TODO: should we handle errors? // equivalent.
func (obj *NetworkDevice) Cmp(dev *NetworkDevice) error {
if (obj == nil) != (dev == nil) { // xor
return fmt.Errorf("the NetworkDevice differs")
}
if obj == nil && dev == nil {
return nil
}
if obj.Name != dev.Name {
return fmt.Errorf("the Name differs")
}
if obj.MAC != dev.MAC {
return fmt.Errorf("the MAC differs")
}
return nil
}
// FilesystemDevice represents a filesystem that is attached to the virt
// machine.
type FilesystemDevice struct {
Access string `lang:"access" yaml:"access"`
Source string `lang:"source" yaml:"source"`
Target string `lang:"target" yaml:"target"`
ReadOnly bool `lang:"read_only" yaml:"read_only"`
}
// GetXML returns the XML representation of this device.
func (obj *FilesystemDevice) GetXML(idx int) string {
source, _ := util.ExpandHome(obj.Source) // TODO: should we handle errors?
var b string var b string
b += "<filesystem" // open b += "<filesystem" // open
if d.Access != "" { if obj.Access != "" {
b += fmt.Sprintf(" accessmode='%s'", d.Access) b += fmt.Sprintf(" accessmode='%s'", obj.Access)
} }
b += ">" // close b += ">" // close
b += fmt.Sprintf("<source dir='%s'/>", source) b += fmt.Sprintf("<source dir='%s'/>", source)
b += fmt.Sprintf("<target dir='%s'/>", d.Target) b += fmt.Sprintf("<target dir='%s'/>", obj.Target)
if d.ReadOnly { if obj.ReadOnly {
b += "<readonly/>" b += "<readonly/>"
} }
b += "</filesystem>" b += "</filesystem>"
return b return b
} }
// Cmp compares two resources and returns an error if they are not equivalent. // Cmp compares two FilesystemDevice's and returns an error if they are not
func (obj *VirtRes) Cmp(r engine.Res) error { // equivalent.
if !obj.Compare(r) { func (obj *FilesystemDevice) Cmp(dev *FilesystemDevice) error {
return fmt.Errorf("did not compare") if (obj == nil) != (dev == nil) { // xor
return fmt.Errorf("the FilesystemDevice differs")
} }
if obj == nil && dev == nil {
return nil
}
if obj.Access != dev.Access {
return fmt.Errorf("the Access differs")
}
if obj.Source != dev.Source {
return fmt.Errorf("the Source differs")
}
if obj.Target != dev.Target {
return fmt.Errorf("the Target differs")
}
if obj.ReadOnly != dev.ReadOnly {
return fmt.Errorf("the ReadOnly differs")
}
return nil return nil
} }
// Compare two resources and return if they are equivalent. // Cmp compares two resources and returns an error if they are not equivalent.
func (obj *VirtRes) Compare(r engine.Res) bool { func (obj *VirtRes) Cmp(r engine.Res) error {
// we can only compare VirtRes to others of the same resource kind // we can only compare VirtRes to others of the same resource kind
res, ok := r.(*VirtRes) res, ok := r.(*VirtRes)
if !ok { if !ok {
return false return fmt.Errorf("not a %s", obj.Kind())
} }
if obj.URI != res.URI { if obj.URI != res.URI {
return false return fmt.Errorf("the URI differs")
} }
if obj.State != res.State { if obj.State != res.State {
return false return fmt.Errorf("the State differs")
} }
if obj.Transient != res.Transient { if obj.Transient != res.Transient {
return false return fmt.Errorf("the Transient differs")
} }
if obj.CPUs != res.CPUs { if obj.CPUs != res.CPUs {
return false return fmt.Errorf("the CPUs differ")
} }
// we can't change this property while machine is running! // we can't change this property while machine is running!
// we do need to return false, so that a new struct gets built, // we do need to return false, so that a new struct gets built,
// which will cause at least one Init() & CheckApply() to run. // which will cause at least one Init() & CheckApply() to run.
if obj.MaxCPUs != res.MaxCPUs { if obj.MaxCPUs != res.MaxCPUs {
return false return fmt.Errorf("the MaxCPUs differ")
}
if obj.HotCPUs != res.HotCPUs {
return fmt.Errorf("the HotCPUs differ")
} }
// TODO: can we skip the compare of certain properties such as // TODO: can we skip the compare of certain properties such as
// Memory because this object (but with different memory) can be // Memory because this object (but with different memory) can be
@@ -1084,26 +1200,61 @@ func (obj *VirtRes) Compare(r engine.Res) bool {
// We would need to run some sort of "old struct update", to get // We would need to run some sort of "old struct update", to get
// the new values, but that's easy to add. // the new values, but that's easy to add.
if obj.Memory != res.Memory { if obj.Memory != res.Memory {
return false return fmt.Errorf("the Memory differs")
} }
// TODO:
//if obj.Boot != res.Boot {
// return false
//}
//if obj.Disk != res.Disk {
// return false
//}
//if obj.CDRom != res.CDRom {
// return false
//}
//if obj.Network != res.Network {
// return false
//}
//if obj.Filesystem != res.Filesystem {
// return false
//}
return true if obj.OSInit != res.OSInit {
return fmt.Errorf("the OSInit differs")
}
if err := engineUtil.StrListCmp(obj.Boot, res.Boot); err != nil {
return errwrap.Wrapf(err, "the Boot differs")
}
if len(obj.Disk) != len(res.Disk) {
return fmt.Errorf("the Disk length differs")
}
for i := range obj.Disk {
if err := obj.Disk[i].Cmp(res.Disk[i]); err != nil {
return errwrap.Wrapf(err, "the Disk differs")
}
}
if len(obj.CDRom) != len(res.CDRom) {
return fmt.Errorf("the CDRom length differs")
}
for i := range obj.CDRom {
if err := obj.CDRom[i].Cmp(res.CDRom[i]); err != nil {
return errwrap.Wrapf(err, "the CDRom differs")
}
}
if len(obj.Network) != len(res.Network) {
return fmt.Errorf("the Network length differs")
}
for i := range obj.Network {
if err := obj.Network[i].Cmp(res.Network[i]); err != nil {
return errwrap.Wrapf(err, "the Network differs")
}
}
if len(obj.Filesystem) != len(res.Filesystem) {
return fmt.Errorf("the Filesystem length differs")
}
for i := range obj.Filesystem {
if err := obj.Filesystem[i].Cmp(res.Filesystem[i]); err != nil {
return errwrap.Wrapf(err, "the Filesystem differs")
}
}
if err := obj.Auth.Cmp(res.Auth); err != nil {
return errwrap.Wrapf(err, "the Auth differs")
}
if obj.RestartOnDiverge != res.RestartOnDiverge {
return fmt.Errorf("the RestartOnDiverge differs")
}
if obj.RestartOnRefresh != res.RestartOnRefresh {
return fmt.Errorf("the RestartOnRefresh differs")
}
return nil
} }
// VirtUID is the UID struct for FileRes. // VirtUID is the UID struct for FileRes.

85
engine/reverse.go Normal file
View File

@@ -0,0 +1,85 @@
// Mgmt
// Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package engine
import (
"fmt"
)
// ReversibleRes is an interface that a resource can implement if it wants to
// have some resource run when it disappears. A disappearance happens when a
// resource is defined in one instance of the graph, and is gone in the
// subsequent one. This is helpful for building robust programs with the engine.
// Default implementations for most of the methods declared in this interface
// can be obtained for your resource by anonymously adding the traits.Reversible
// struct to your resource implementation.
type ReversibleRes interface {
Res
// ReversibleMeta lets you get or set meta params for the reversible
// trait.
ReversibleMeta() *ReversibleMeta
// SetReversibleMeta lets you set all of the meta params for the
// reversible trait in a single call.
SetReversibleMeta(*ReversibleMeta)
// Reversed returns the "reverse" or "reciprocal" resource. This is used
// to "clean" up after a previously defined resource has been removed.
// Interestingly, this could return the core Res interface instead of a
// ReversibleRes, because there is no requirement that the reverse of a
// Res be the same kind of Res, and the reverse might not be reversible!
// However, in practice, it's nice to use some of the Reversible meta
// params in the built value, so keep things simple and have this be a
// reversible res. The Res itself doesn't have to implement Reversed()
// in a meaningful way, it can just return nil and it will get ignored.
Reversed() (ReversibleRes, error)
}
// ReversibleMeta provides some parameters specific to reversible resources.
type ReversibleMeta struct {
// Disabled specifies that reversing should be disabled for this
// resource.
Disabled bool
// Reversal specifies that the resource was built from a reversal. This
// must be set if the resource was built by a reversal.
Reversal bool
// Overwrite specifies that we should overwrite any existing stored
// reversible resource if one that is pending already exists. If this is
// false, and a resource with the same name and kind exists, then this
// will cause an error.
Overwrite bool
// TODO: add options here, including whether to reverse edges, etc...
}
// Cmp compares two ReversibleMeta structs and determines if they're equivalent.
func (obj *ReversibleMeta) Cmp(rm *ReversibleMeta) error {
if obj.Disabled != rm.Disabled {
return fmt.Errorf("values for Disabled are different")
}
if obj.Reversal != rm.Reversal { // TODO: do we want to compare these?
return fmt.Errorf("values for Reversal are different")
}
if obj.Overwrite != rm.Overwrite {
return fmt.Errorf("values for Overwrite are different")
}
return nil
}

2
engine/sendrecv.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

14
engine/traits/autoedge.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -25,7 +25,9 @@ import (
// methods needed to support autoedges on resources. It may be used as a start // methods needed to support autoedges on resources. It may be used as a start
// point to avoid re-implementing the straightforward methods. // point to avoid re-implementing the straightforward methods.
type Edgeable struct { type Edgeable struct {
meta *engine.AutoEdgeMeta // Xmeta is the stored meta. It should be called `meta` but it must be
// public so that the `encoding/gob` package can encode it properly.
Xmeta *engine.AutoEdgeMeta
// Bug5819 works around issue https://github.com/golang/go/issues/5819 // Bug5819 works around issue https://github.com/golang/go/issues/5819
Bug5819 interface{} // XXX: workaround Bug5819 interface{} // XXX: workaround
@@ -33,16 +35,16 @@ type Edgeable struct {
// AutoEdgeMeta lets you get or set meta params for the automatic edges trait. // AutoEdgeMeta lets you get or set meta params for the automatic edges trait.
func (obj *Edgeable) AutoEdgeMeta() *engine.AutoEdgeMeta { func (obj *Edgeable) AutoEdgeMeta() *engine.AutoEdgeMeta {
if obj.meta == nil { // set the defaults if previously empty if obj.Xmeta == nil { // set the defaults if previously empty
obj.meta = &engine.AutoEdgeMeta{ obj.Xmeta = &engine.AutoEdgeMeta{
Disabled: false, Disabled: false,
} }
} }
return obj.meta return obj.Xmeta
} }
// SetAutoEdgeMeta lets you set all of the meta params for the automatic edges // SetAutoEdgeMeta lets you set all of the meta params for the automatic edges
// trait in a single call. // trait in a single call.
func (obj *Edgeable) SetAutoEdgeMeta(meta *engine.AutoEdgeMeta) { func (obj *Edgeable) SetAutoEdgeMeta(meta *engine.AutoEdgeMeta) {
obj.meta = meta obj.Xmeta = meta
} }

14
engine/traits/autogroup.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -27,7 +27,9 @@ import (
// methods needed to support autogrouping on resources. It may be used as a // methods needed to support autogrouping on resources. It may be used as a
// starting point to avoid re-implementing the straightforward methods. // starting point to avoid re-implementing the straightforward methods.
type Groupable struct { type Groupable struct {
meta *engine.AutoGroupMeta // Xmeta is the stored meta. It should be called `meta` but it must be
// public so that the `encoding/gob` package can encode it properly.
Xmeta *engine.AutoGroupMeta
isGrouped bool // am i contained within a group? isGrouped bool // am i contained within a group?
grouped []engine.GroupableRes // list of any grouped resources grouped []engine.GroupableRes // list of any grouped resources
@@ -39,18 +41,18 @@ type Groupable struct {
// AutoGroupMeta lets you get or set meta params for the automatic grouping // AutoGroupMeta lets you get or set meta params for the automatic grouping
// trait. // trait.
func (obj *Groupable) AutoGroupMeta() *engine.AutoGroupMeta { func (obj *Groupable) AutoGroupMeta() *engine.AutoGroupMeta {
if obj.meta == nil { // set the defaults if previously empty if obj.Xmeta == nil { // set the defaults if previously empty
obj.meta = &engine.AutoGroupMeta{ obj.Xmeta = &engine.AutoGroupMeta{
Disabled: false, Disabled: false,
} }
} }
return obj.meta return obj.Xmeta
} }
// SetAutoGroupMeta lets you set all of the meta params for the automatic // SetAutoGroupMeta lets you set all of the meta params for the automatic
// grouping trait in a single call. // grouping trait in a single call.
func (obj *Groupable) SetAutoGroupMeta(meta *engine.AutoGroupMeta) { func (obj *Groupable) SetAutoGroupMeta(meta *engine.AutoGroupMeta) {
obj.meta = meta obj.Xmeta = meta
} }
// GroupCmp compares two resources and decides if they're suitable for grouping. // GroupCmp compares two resources and decides if they're suitable for grouping.

2
engine/traits/base.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

18
engine/traits/kind.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -17,11 +17,21 @@
package traits package traits
import (
"encoding/gob"
)
func init() {
gob.Register(&Kinded{})
}
// Kinded contains a general implementation of the properties and methods needed // Kinded contains a general implementation of the properties and methods needed
// to support the resource kind. It should be used as a starting point to avoid // to support the resource kind. It should be used as a starting point to avoid
// re-implementing the straightforward kind methods. // re-implementing the straightforward kind methods.
type Kinded struct { type Kinded struct {
kind string // Xkind is the stored kind. It should be called `kind` but it must be
// public so that the `encoding/gob` package can encode it properly.
Xkind string
// Bug5819 works around issue https://github.com/golang/go/issues/5819 // Bug5819 works around issue https://github.com/golang/go/issues/5819
Bug5819 interface{} // XXX: workaround Bug5819 interface{} // XXX: workaround
@@ -29,11 +39,11 @@ type Kinded struct {
// Kind returns the string representation for the kind this resource is. // Kind returns the string representation for the kind this resource is.
func (obj *Kinded) Kind() string { func (obj *Kinded) Kind() string {
return obj.kind return obj.Xkind
} }
// SetKind sets the kind string for this resource. It must only be set by the // SetKind sets the kind string for this resource. It must only be set by the
// engine. // engine.
func (obj *Kinded) SetKind(kind string) { func (obj *Kinded) SetKind(kind string) {
obj.kind = kind obj.Xkind = kind
} }

14
engine/traits/meta.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -25,7 +25,9 @@ import (
// to support meta parameters. It should be used as a starting point to avoid // to support meta parameters. It should be used as a starting point to avoid
// re-implementing the straightforward meta methods. // re-implementing the straightforward meta methods.
type Meta struct { type Meta struct {
meta *engine.MetaParams // Xmeta is the stored meta. It should be called `meta` but it must be
// public so that the `encoding/gob` package can encode it properly.
Xmeta *engine.MetaParams
// Bug5819 works around issue https://github.com/golang/go/issues/5819 // Bug5819 works around issue https://github.com/golang/go/issues/5819
Bug5819 interface{} // XXX: workaround Bug5819 interface{} // XXX: workaround
@@ -33,14 +35,14 @@ type Meta struct {
// MetaParams lets you get or set meta params for this trait. // MetaParams lets you get or set meta params for this trait.
func (obj *Meta) MetaParams() *engine.MetaParams { func (obj *Meta) MetaParams() *engine.MetaParams {
if obj.meta == nil { // set the defaults if previously empty if obj.Xmeta == nil { // set the defaults if previously empty
obj.meta = engine.DefaultMetaParams.Copy() obj.Xmeta = engine.DefaultMetaParams.Copy()
} }
return obj.meta return obj.Xmeta
} }
// SetMetaParams lets you set all of the meta params for the resource in a // SetMetaParams lets you set all of the meta params for the resource in a
// single call. // single call.
func (obj *Meta) SetMetaParams(meta *engine.MetaParams) { func (obj *Meta) SetMetaParams(meta *engine.MetaParams) {
obj.meta = meta obj.Xmeta = meta
} }

10
engine/traits/named.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify
@@ -21,7 +21,9 @@ package traits
// to support named resources. It should be used as a starting point to avoid // to support named resources. It should be used as a starting point to avoid
// re-implementing the straightforward name methods. // re-implementing the straightforward name methods.
type Named struct { type Named struct {
name string // Xname is the stored name. It should be called `name` but it must be
// public so that the `encoding/gob` package can encode it properly.
Xname string
// Bug5819 works around issue https://github.com/golang/go/issues/5819 // Bug5819 works around issue https://github.com/golang/go/issues/5819
Bug5819 interface{} // XXX: workaround Bug5819 interface{} // XXX: workaround
@@ -30,11 +32,11 @@ type Named struct {
// Name returns the unique name this resource has. It is only unique within its // Name returns the unique name this resource has. It is only unique within its
// own kind. // own kind.
func (obj *Named) Name() string { func (obj *Named) Name() string {
return obj.name return obj.Xname
} }
// SetName sets the unique name for this resource. It must only be unique within // SetName sets the unique name for this resource. It must only be unique within
// its own kind. // its own kind.
func (obj *Named) SetName(name string) { func (obj *Named) SetName(name string) {
obj.name = name obj.Xname = name
} }

2
engine/traits/refresh.go
View File

@@ -1,5 +1,5 @@
// Mgmt // Mgmt
// Copyright (C) 2013-2018+ James Shubin and the project contributors // Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors // Written by James Shubin <james@shubin.ca> and the project contributors
// //
// This program is free software: you can redistribute it and/or modify // This program is free software: you can redistribute it and/or modify

50
engine/traits/reverse.go Normal file
View File

@@ -0,0 +1,50 @@
// Mgmt
// Copyright (C) 2013-2019+ James Shubin and the project contributors
// Written by James Shubin <james@shubin.ca> and the project contributors
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package traits
import (
"github.com/purpleidea/mgmt/engine"
)
// Reversible contains a general implementation with most of the properties and
// methods needed to support reversing resources. It may be used as a starting
// point to avoid re-implementing the straightforward methods.
type Reversible struct {
// Xmeta is the stored meta. It should be called `meta` but it must be
// public so that the `encoding/gob` package can encode it properly.
Xmeta *engine.ReversibleMeta
// Bug5819 works around issue https://github.com/golang/go/issues/5819
Bug5819 interface{} // XXX: workaround
}
// ReversibleMeta lets you get or set meta params for the reversing trait.
func (obj *Reversible) ReversibleMeta() *engine.ReversibleMeta {
if obj.Xmeta == nil { // set the defaults if previously empty
obj.Xmeta = &engine.ReversibleMeta{
Disabled: true, // by default we're disabled
}
}
return obj.Xmeta
}
// SetReversibleMeta lets you set all of the meta params for the reversing trait
// in a single call.
func (obj *Reversible) SetReversibleMeta(meta *engine.ReversibleMeta) {
obj.Xmeta = meta
}

Some files were not shown because too many files have changed in this diff Show More