Commit Graph

113 Commits

Author SHA1 Message Date
Donald Bakong
4b4b7dc169 engine: resources: Adding tests to file mode 2019-11-08 10:02:30 -05:00
James Shubin
1eba5833d5 engine: resources: Consistency changes and cleanup for file mode
This makes a few consistency changes and cleanups to the file mode
feature so that it's more in style with the rest of the code base.
2019-11-01 22:01:38 -04:00
Derek Buckley
83a747794e engine: resources: Adds symbolic mode to file resource
Adds a symbolic parsing function to the util package for parsing in the
file resource.
2019-11-01 21:57:10 -04:00
Julien Pivotto
3e16d1da46 engine: resources: Add new consul resource
This commit adds a new consul:kv resource which allows us to set and
watch keys inside a consul kv datastore.

This was started by roidelapluie, and was finished during pair
programming with purpleidea.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
Signed-off-by: James Shubin <james@shubin.ca>
2019-11-01 21:38:08 -04:00
James Shubin
46a71296a9 engine: resources: Add a Purge option to the file resource
This adds a "purge" parameter to the file resource. To do this, we have
to add the API hooks so the file resource can query other resources in
the graph to know if they are present, and as a result whether they
should be excluded from the purge or not.

This is useful for when we have a managed directory with some managed
contents. If a managed file is removed from the directory, then it will
be removed by the file (directory) resource if it has Purge set.
Alternatively, you can use the Reverse meta param, which is sometimes
preferable for this use case and sometimes not. This will be discussed
elsewhere.

This also adds a bunch of tests for this feature.

This also makes a few somewhat related cleanups in the file code.
2019-10-29 11:54:08 -04:00
James Shubin
1285588b62 engine: resources: Fix file absent helper
We should check this nil case too.
2019-10-29 07:15:43 -04:00
James Shubin
d96392f65e engine: resources: Improve error message in test
Nothing major, just improve testing here.
2019-10-29 07:15:43 -04:00
James Shubin
d1c5a736ae engine: resources: Allow nil helper functions in tests
This reduces some of the boilerplate when writing resource tests.
2019-10-29 07:15:43 -04:00
James Shubin
6b1e038c5c engine: resources: Add file exists helper
This will allow us to test even more things!
2019-10-29 07:15:43 -04:00
James Shubin
eaab1aae28 engine: graph, resources: Add filtered graph function
This lets a resource query the resource graph in a controlled way.
2019-10-29 07:15:43 -04:00
James Shubin
31030343a2 engine: Add graph queryable trait
This is a trait that specifies that your resource can be queried by
others. You can either enable it plainly to add wholesale access by
everyone, or you can implement your own Allow method to limit what is
permitted.
2019-10-29 07:15:43 -04:00
James Shubin
325ca03a13 engine: graph: Pass through the graph struct
We want to use it in the resources.
2019-10-29 07:15:43 -04:00
James Shubin
58421fd31a engine: resources: Add fragments support to file resource
This adds a "fragments" mode to the file resource. In addition to
"content" and "source", you can now alternatively build a file from the
file fragments of other files.
2019-10-29 07:15:43 -04:00
James Shubin
06952c224b engine: resources: nspawn: Use populate variable
We referred to the wrong variable. Not a major bug, but would produce a
useless or confusing error message otherwise.
2019-10-21 07:38:42 -04:00
Donald Bakong
fc1c631c98 engine: resources: Change Res API from Compare to Cmp
This will be done by refactoring the current method, to return an error
message instead of a boolean value. This will also update a typo on the
user res.
2019-09-27 18:10:58 -04:00
James Shubin
9dae5ef83b engine: resources: Improve the file res and add strict state
This might be slightly controversial, in that you must specify the state
if a file would need to be created to perform the action. We no longer
implicitly assume that just specifying content is enough. As it turns
out, I believe this is safer and more correct. The code to implement
this turns out to be much more logical and simplified, and does this
removes an ambiguous corner case from the reversed resource code.

Some discussion in: https://github.com/purpleidea/mgmt/issues/540

This patch also does a bit of related cleanup.
2019-09-14 16:07:53 -04:00
James Shubin
5c27a249b7 engine: resources: Add reversible API and file resource
This adds the first reversible resource (file) and the necessary engine
API hooks to make it all work. This allows a special "reversed" resource
to be added to the subsequent graph in the stream when an earlier
version "disappears". This disappearance can happen if it was previously
in an if statement that then becomes false.

It might be wise to combine the use of this meta parameter with the use
of the `realize` meta parameter to ensure that your reversed resource
actually runs at least once, if there's a chance that it might be gone
for a while.

This patch also adds a new test harness for testing resources. It
doesn't test the "live" aspect of resources, as it doesn't run Watch,
but it was designed to ensure CheckApply works as intended, and it runs
very quickly with a simplified timeline of happenings.
2019-09-11 03:40:22 -04:00
James Shubin
43ff92bbe7 engine: resources: Clean up test log message 2019-09-11 03:40:22 -04:00
James Shubin
28adc7e563 engine: resource: Refactor helper functions
Maybe we can use them in other tests too.
2019-09-11 03:40:22 -04:00
James Shubin
9788411995 engine: resources: Add another validation check
This simple check should prevent some silly mistakes and make the logic
easier for other parts of the code that won't have to worry about this
pattern.
2019-09-11 03:40:22 -04:00
James Shubin
0c9e8cc50e engine: resources: Change the default file state
The default file state should be undefined. This is important because if
a reverse scenario that doesn't specify the state gets given this
default, it will be as if it was specified explicitly, which wouldn't
necessarily be what we want. Instead, an undefined state should
implicitly cause a file to get created if there's a reason to do so,
such as if content or another attribute is specified.

Hopefully this change doesn't introduce any bugs in the CheckApply code,
if it does, then it was due to a lack of implicit file creation.
2019-09-11 03:16:57 -04:00
James Shubin
34d572c523 engine: Improve the way we make a unique res path token
This is needed in the state directory.
2019-09-11 03:16:57 -04:00
James Shubin
011b496b3f engine: resources: Ensure the Kind and Name methods work
Triple check these work after decoding, by adding a test.
2019-09-11 03:16:57 -04:00
James Shubin
12b906eac6 engine: Refactor state dir into a separate function
This lets us re-use it, and know the path is fixed.
2019-09-11 03:16:57 -04:00
James Shubin
20937d05c3 engine: resources: file: Add undefined file state and validate it
We should consider using *string instead of the empty string, but let's
keep the diff smaller for now.
2019-09-11 03:16:57 -04:00
James Shubin
4943d37ccf engine: resources: file: Use constants for state values
More robustness is yay!
2019-09-11 03:16:57 -04:00
James Shubin
3a8fd215de engine: resources: file: Add Copy method to file res
This lets us implement the CopyableRes interface.
2019-09-11 03:16:57 -04:00
James Shubin
87572e8922 test: Catch capitalized error messages in tests 2019-09-06 03:28:49 -04:00
James Shubin
1161872324 etcd: fs: Errors should start with lower case 2019-07-26 04:20:04 -04:00
Felix Frank
191a2495a5 engine: resources: mount: Fix the dbus call for reloading systemd
The Reload method cannot just be invoked on the administrative DBus
object. Just like the method for reloading specific units, it needs
to be invoked on the proper DBus service, addressing the proper object
and using the right interface.

Added an additional constant for the systemd DBus service. Even though
it shares the same value as the interface base name, this is
happenstance and it's technically incorrect to open a connection to an
interface name. The connection needs a service name.

Fixes #509
2019-06-04 23:44:57 +02:00
James Shubin
7a756cacb9 engine: graph: Add a mutex around waits map access
If you ran some extremely absurd code, it turns out you can cause a
race. This was found by roiedelapluie experimenting! In this case, it
would panic with: fatal error: concurrent map read and map write. This
patch adds the mutex to avoid this rare race.
2019-05-14 10:53:36 -04:00
Jan Martens
3c1da423fa engine: resources: nspawn: Trim possible systemd version suffix 2019-05-14 16:00:26 +02:00
Jonathan Gold
8f1f5d33fd engine: resources: mount: Restart remote-fs target 2019-04-23 16:24:49 -04:00
James Shubin
a5842a41b2 etcd: Rewrite embed etcd implementation
This is a giant cleanup of the etcd code. The earlier version was
written when I was less experienced with golang.

This is still not perfect, and does contain some races, but at least
it's a decent base to start from. The automatic elastic clustering
should be considered an experimental feature. If you need a more
battle-tested cluster, then you should manage etcd manually and point
mgmt at your existing cluster.
2019-04-11 21:43:48 -04:00
James Shubin
e4d874cc69 engine: resources: Remove named return params
Named return params aren't a favourite feature of mine, and they're
rarely used in the resource writing. They keep popping up because I once
used them and we've been copying and pasting code ever since. Get rid of
them all to help prevent the unnecessary spread.
2019-03-24 15:30:02 -04:00
James Shubin
07f542b4d7 legal: Happy 2019 everyone...
Done with:

ack '2018+' -l | xargs sed -i -e 's/2018+/2019+/g'

Checked manually with:

git add -p

Hello to future James from 2020, and Happy Hacking!
2019-03-24 15:08:50 -04:00
James Shubin
ae56261961 engine: util, resources: virt: Clean up virt resource
Do some cleanups which were long overdue.
2019-03-15 15:24:40 -04:00
James Shubin
753d1104ef util: Port all multierr code to new errwrap package
This cleans things up and simplifies a lot of the code. Also it's easier
to just import one error package when needed.
2019-03-12 16:51:37 -04:00
James Shubin
880652f5d4 util: Port all code to new errwrap package
This should keep things more uniform.
2019-03-12 16:49:01 -04:00
James Shubin
54c81d6bb2 engine, pgp: Fixup incorrect error usage
Small fixups found by the next commit.
2019-03-12 16:49:01 -04:00
James Shubin
5bbb474db6 engine: resources: Clean up KV resource 2019-03-09 18:48:26 -05:00
James Shubin
de43569fa2 engine, lang: Improve send/recv significantly
Part of this was rotten, and not fully functional. This fixes the rot,
adds some tests, and improves the type checking that occurs when sending
and receiving values. In addition, a significant portion of this happens
at compile time.

There is still more work to be done here, but this should get us a good
chunk of the way for now.
2019-03-09 17:37:58 -05:00
James Shubin
4c104d55cb engine: util: Add a new utility function for send/recv
This new utility function makes verifying send/recv struct comparisons
consistent. Unfortunately it doesn't yet support coercing from *string
to string or from string to *string.
2019-03-09 16:03:30 -05:00
James Shubin
8a8215fabe engine: util: Improve StructTagToFieldName and add tests
This improves this function to make it more generic.
2019-03-09 16:02:33 -05:00
James Shubin
4badeafb98 engine: resources: Add missing struct tags
These are needed for send/recv.
2019-03-09 16:01:25 -05:00
James Shubin
7cb79bec49 engine: resources: Replace the cached values with a live calculation
This replaces the static obj.path and obj.isDir with live variants. I
don't know why I ever cared about caching these before, and if we ever
care we can memoize properly in the future.

This caused a small bug to actually be masked in the gob code. It is now
fixed in the previous commit.
2019-03-06 10:08:14 -05:00
James Shubin
8da0da02d9 engine: traits: Make encoded fields public
The fields that we might want to store with encoding/gob or any other
encoding package, need to be public. We currently don't use any of these
at the moment, but we might in the future.
2019-03-06 10:08:14 -05:00
James Shubin
efef260764 engine: resources: Improve file Cmp function 2019-03-06 07:09:39 -05:00
James Shubin
a56991d081 engine: resources: Remove possible panic from within file res
Not sure how I let this in, but we should never do this. Hopefully the
Validate should catch this issue in advance, and if not, at least we'll
only error.
2019-03-06 07:09:39 -05:00
James Shubin
f0196540ab resources: file: Make some small cleanups to file res
This does some small cleaning for consistency, since I haven't reviewed
this code in a long while.
2019-03-06 07:09:39 -05:00