302 Commits

Author SHA1 Message Date
James Shubin
90fd8023dd lang, engine: Add a facility for resources to export constants
Since we focus on safety, it would be nice to reduce the chance of any
runtime errors if we made a typo for a resource parameter. With this
patch, each resource can export constants into the global namespace so
that typos would cause a compile error.

Of course in the future if we had a more advanced type system, then we
could support precise types for each individual resource param, but in
an attempt to keep things simple, we'll leave that for another day. It
would add complexity too if we ever wanted to store a parameter
externally.

Lastly, we might consider adding "special case" parsing so that directly
specified fields would parse intelligently. For example, we could allow:

	file "/tmp/hello" {
		state => exists,	# magic sugar!
	}

This isn't supported for now, but if it works after all the other parser
changes have been made, it might be something to consider.
2020-01-29 11:16:04 -05:00
James Shubin
f67ad9c061 test: Add a check for too long or badly reflowed docstrings
This ensures that docstring comments are wrapped to 80 chars. ffrank
seemed to be making this mistake far too often, and it's a silly thing
to look for manually. As it turns out, I've made it too, as have many
others. Now we have a test that checks for most cases. There are still a
few stray cases that aren't checked automatically, but this can be
improved upon if someone is motivated to do so.

Before anyone complains about the 80 character limit: this only checks
docstring comments, not source code length or inline source code
comments. There's no excuse for having docstrings that are badly
reflowed or over 80 chars, particularly if you have an automated test.
2020-01-25 04:43:33 -05:00
James Shubin
fec94aa53a engine, lang: Fix simple test failures
Two bugs sneaked in while pushing old stuff.
2020-01-12 19:35:11 -05:00
James Shubin
f07387225b engine: resources: Log more info about tftp errors
This helps for debugging this kind of issue:
https://github.com/pin/tftp/issues/41#issuecomment-570744056
2020-01-03 20:42:34 -05:00
James Shubin
2648fb1bb1 legal: Happy 2020 everyone...
Done with:

ack '2019+' -l | xargs sed -i -e 's/2019+/2020+/g'

Checked manually with:

git add -p

Hello to future James from 2021, and Happy Hacking!
2020-01-03 20:08:37 -05:00
James Shubin
d34715b4ba engine: resources: pippet: Cleanup and proper wrapping
Felix, please configure your editor to wrap at 80 chars and/or help us
write a test for this please =D
2020-01-03 01:20:00 -05:00
Felix Frank
63af50bf98 engine: resources: pippet: Initial implementation for new resource type
The pippet resource implements faster integration of Puppet resources
in mgmt at runtime, by piping synchronization commands to a Puppet
process that keeps running alongside mgmt. This avoids huge overhead
through launching a Puppet process for each operation on a resource
that is delegated to Puppet.
2020-01-03 01:19:37 -05:00
James Shubin
456550c1d4 engine: resources: docker: Make a few fixups
Here are a few fixups to the docker resources. All miscellaneous stuff,
nothing major.
2020-01-03 00:53:20 -05:00
Jonathan Gold
8174b88ec3 engine: resources: docker: Add sensible defaults 2020-01-03 00:30:01 -05:00
Jonathan Gold
3233973748 engine: resources: docker: Add AutoEdges between container and image 2020-01-03 00:30:01 -05:00
Jonathan Gold
bdfb1cf33e engine: resources: docker: Ensure image is specified for containers 2020-01-03 00:30:01 -05:00
Jonathan Gold
1c5fcd59e7 engine: resources: docker: Add a docker image resource 2020-01-03 00:30:01 -05:00
James Shubin
5526bbba64 engine: resources: Add a tftp server and tftp file resource
This adds a tftp server and tftp file resource to help you run a small
pure golang tftp server embedded inside the mgmt resource model.
2019-12-17 03:41:45 -05:00
James Shubin
1eba5833d5 engine: resources: Consistency changes and cleanup for file mode
This makes a few consistency changes and cleanups to the file mode
feature so that it's more in style with the rest of the code base.
2019-11-01 22:01:38 -04:00
Derek Buckley
83a747794e engine: resources: Adds symbolic mode to file resource
Adds a symbolic parsing function to the util package for parsing in the
file resource.
2019-11-01 21:57:10 -04:00
Julien Pivotto
3e16d1da46 engine: resources: Add new consul resource
This commit adds a new consul:kv resource which allows us to set and
watch keys inside a consul kv datastore.

This was started by roidelapluie, and was finished during pair
programming with purpleidea.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
Signed-off-by: James Shubin <james@shubin.ca>
2019-11-01 21:38:08 -04:00
James Shubin
46a71296a9 engine: resources: Add a Purge option to the file resource
This adds a "purge" parameter to the file resource. To do this, we have
to add the API hooks so the file resource can query other resources in
the graph to know if they are present, and as a result whether they
should be excluded from the purge or not.

This is useful for when we have a managed directory with some managed
contents. If a managed file is removed from the directory, then it will
be removed by the file (directory) resource if it has Purge set.
Alternatively, you can use the Reverse meta param, which is sometimes
preferable for this use case and sometimes not. This will be discussed
elsewhere.

This also adds a bunch of tests for this feature.

This also makes a few somewhat related cleanups in the file code.
2019-10-29 11:54:08 -04:00
James Shubin
1285588b62 engine: resources: Fix file absent helper
We should check this nil case too.
2019-10-29 07:15:43 -04:00
James Shubin
d96392f65e engine: resources: Improve error message in test
Nothing major, just improve testing here.
2019-10-29 07:15:43 -04:00
James Shubin
d1c5a736ae engine: resources: Allow nil helper functions in tests
This reduces some of the boilerplate when writing resource tests.
2019-10-29 07:15:43 -04:00
James Shubin
6b1e038c5c engine: resources: Add file exists helper
This will allow us to test even more things!
2019-10-29 07:15:43 -04:00
James Shubin
eaab1aae28 engine: graph, resources: Add filtered graph function
This lets a resource query the resource graph in a controlled way.
2019-10-29 07:15:43 -04:00
James Shubin
58421fd31a engine: resources: Add fragments support to file resource
This adds a "fragments" mode to the file resource. In addition to
"content" and "source", you can now alternatively build a file from the
file fragments of other files.
2019-10-29 07:15:43 -04:00
James Shubin
06952c224b engine: resources: nspawn: Use populate variable
We referred to the wrong variable. Not a major bug, but would produce a
useless or confusing error message otherwise.
2019-10-21 07:38:42 -04:00
Donald Bakong
fc1c631c98 engine: resources: Change Res API from Compare to Cmp
This will be done by refactoring the current method, to return an error
message instead of a boolean value. This will also update a typo on the
user res.
2019-09-27 18:10:58 -04:00
James Shubin
9dae5ef83b engine: resources: Improve the file res and add strict state
This might be slightly controversial, in that you must specify the state
if a file would need to be created to perform the action. We no longer
implicitly assume that just specifying content is enough. As it turns
out, I believe this is safer and more correct. The code to implement
this turns out to be much more logical and simplified, and does this
removes an ambiguous corner case from the reversed resource code.

Some discussion in: https://github.com/purpleidea/mgmt/issues/540

This patch also does a bit of related cleanup.
2019-09-14 16:07:53 -04:00
James Shubin
5c27a249b7 engine: resources: Add reversible API and file resource
This adds the first reversible resource (file) and the necessary engine
API hooks to make it all work. This allows a special "reversed" resource
to be added to the subsequent graph in the stream when an earlier
version "disappears". This disappearance can happen if it was previously
in an if statement that then becomes false.

It might be wise to combine the use of this meta parameter with the use
of the `realize` meta parameter to ensure that your reversed resource
actually runs at least once, if there's a chance that it might be gone
for a while.

This patch also adds a new test harness for testing resources. It
doesn't test the "live" aspect of resources, as it doesn't run Watch,
but it was designed to ensure CheckApply works as intended, and it runs
very quickly with a simplified timeline of happenings.
2019-09-11 03:40:22 -04:00
James Shubin
43ff92bbe7 engine: resources: Clean up test log message 2019-09-11 03:40:22 -04:00
James Shubin
28adc7e563 engine: resource: Refactor helper functions
Maybe we can use them in other tests too.
2019-09-11 03:40:22 -04:00
James Shubin
9788411995 engine: resources: Add another validation check
This simple check should prevent some silly mistakes and make the logic
easier for other parts of the code that won't have to worry about this
pattern.
2019-09-11 03:40:22 -04:00
James Shubin
0c9e8cc50e engine: resources: Change the default file state
The default file state should be undefined. This is important because if
a reverse scenario that doesn't specify the state gets given this
default, it will be as if it was specified explicitly, which wouldn't
necessarily be what we want. Instead, an undefined state should
implicitly cause a file to get created if there's a reason to do so,
such as if content or another attribute is specified.

Hopefully this change doesn't introduce any bugs in the CheckApply code,
if it does, then it was due to a lack of implicit file creation.
2019-09-11 03:16:57 -04:00
James Shubin
011b496b3f engine: resources: Ensure the Kind and Name methods work
Triple check these work after decoding, by adding a test.
2019-09-11 03:16:57 -04:00
James Shubin
20937d05c3 engine: resources: file: Add undefined file state and validate it
We should consider using *string instead of the empty string, but let's
keep the diff smaller for now.
2019-09-11 03:16:57 -04:00
James Shubin
4943d37ccf engine: resources: file: Use constants for state values
More robustness is yay!
2019-09-11 03:16:57 -04:00
James Shubin
3a8fd215de engine: resources: file: Add Copy method to file res
This lets us implement the CopyableRes interface.
2019-09-11 03:16:57 -04:00
James Shubin
87572e8922 test: Catch capitalized error messages in tests 2019-09-06 03:28:49 -04:00
James Shubin
1161872324 etcd: fs: Errors should start with lower case 2019-07-26 04:20:04 -04:00
Felix Frank
191a2495a5 engine: resources: mount: Fix the dbus call for reloading systemd
The Reload method cannot just be invoked on the administrative DBus
object. Just like the method for reloading specific units, it needs
to be invoked on the proper DBus service, addressing the proper object
and using the right interface.

Added an additional constant for the systemd DBus service. Even though
it shares the same value as the interface base name, this is
happenstance and it's technically incorrect to open a connection to an
interface name. The connection needs a service name.

Fixes #509
2019-06-04 23:44:57 +02:00
Jan Martens
3c1da423fa engine: resources: nspawn: Trim possible systemd version suffix 2019-05-14 16:00:26 +02:00
Jonathan Gold
8f1f5d33fd engine: resources: mount: Restart remote-fs target 2019-04-23 16:24:49 -04:00
James Shubin
a5842a41b2 etcd: Rewrite embed etcd implementation
This is a giant cleanup of the etcd code. The earlier version was
written when I was less experienced with golang.

This is still not perfect, and does contain some races, but at least
it's a decent base to start from. The automatic elastic clustering
should be considered an experimental feature. If you need a more
battle-tested cluster, then you should manage etcd manually and point
mgmt at your existing cluster.
2019-04-11 21:43:48 -04:00
James Shubin
e4d874cc69 engine: resources: Remove named return params
Named return params aren't a favourite feature of mine, and they're
rarely used in the resource writing. They keep popping up because I once
used them and we've been copying and pasting code ever since. Get rid of
them all to help prevent the unnecessary spread.
2019-03-24 15:30:02 -04:00
James Shubin
07f542b4d7 legal: Happy 2019 everyone...
Done with:

ack '2018+' -l | xargs sed -i -e 's/2018+/2019+/g'

Checked manually with:

git add -p

Hello to future James from 2020, and Happy Hacking!
2019-03-24 15:08:50 -04:00
James Shubin
ae56261961 engine: util, resources: virt: Clean up virt resource
Do some cleanups which were long overdue.
2019-03-15 15:24:40 -04:00
James Shubin
753d1104ef util: Port all multierr code to new errwrap package
This cleans things up and simplifies a lot of the code. Also it's easier
to just import one error package when needed.
2019-03-12 16:51:37 -04:00
James Shubin
880652f5d4 util: Port all code to new errwrap package
This should keep things more uniform.
2019-03-12 16:49:01 -04:00
James Shubin
54c81d6bb2 engine, pgp: Fixup incorrect error usage
Small fixups found by the next commit.
2019-03-12 16:49:01 -04:00
James Shubin
5bbb474db6 engine: resources: Clean up KV resource 2019-03-09 18:48:26 -05:00
James Shubin
4badeafb98 engine: resources: Add missing struct tags
These are needed for send/recv.
2019-03-09 16:01:25 -05:00
James Shubin
7cb79bec49 engine: resources: Replace the cached values with a live calculation
This replaces the static obj.path and obj.isDir with live variants. I
don't know why I ever cared about caching these before, and if we ever
care we can memoize properly in the future.

This caused a small bug to actually be masked in the gob code. It is now
fixed in the previous commit.
2019-03-06 10:08:14 -05:00