Commit Graph

147 Commits

Author SHA1 Message Date
James Shubin
c555478b54 engine, lang: Misc fixes for golang lint 2021-08-09 16:55:31 -04:00
Joe Groocock
fe2b8c9fee engine: resources: exec: AutoEdge to User/Group/File
Fixes https://github.com/purpleidea/mgmt/issues/221

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-11 16:51:02 -04:00
James Shubin
4d187419ac engine: Small typo/cleanups in autogrouping code 2021-05-04 05:30:27 -04:00
James Shubin
58998f9cab engine: Transform the send/recv init functions into helpers
Since we'll want to use them elsewhere, we should make these helper
functions. It also makes the code look a lot neater. Unfortunately, it
adds a bit more indirection, but this isn't a critical flaw here.
2021-05-04 05:30:27 -04:00
Joe Groocock
7f1baea3b0 engine: resources: docker: Replace deprecated NewClient() with NewClientWithOpts()
docker/client.NewClient() is deprecated in favour of NewClientWithOpts()
which takes a series of client.Opt functions to configure the API
client. As mgmt only passes the API version through, this simplifies the
NewClient() calls.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-02 10:39:04 +01:00
Joe Groocock
6a61774fb7 docker: Bump docker dependencies, add containerd
These dependencies are maintained because the upstream repos bundle
vendor directories into the repos, which cause namespacing issues during
build. Git submodules don't strip the vendor directory whereas most
vendoring tools would.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-09 21:14:34 +00:00
Joe Groocock
c950568f1b lang: Move StructTag const into lang/types
This constant value is strongly tied to the language, and little to do
with the engine. Move the definition into the lang/types package to
prevent circular imports between lang/types and engine/util.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-06 16:56:57 +00:00
James Shubin
336a38081a legal: Happy 2021 everyone...
Done with:

ack '2020+' -l | xargs sed -i -e 's/2020+/2021+/g'

Checked manually with:

git add -p

Hello to future James from 2022, and Happy Hacking!
2021-01-31 16:52:46 -05:00
James Shubin
d75f763c99 misc, docs: Move to golang 1.12 2020-09-23 12:34:54 -04:00
Donald Bakong
e99dd749a0 engine: resources: Fixing return bug 2020-09-23 11:10:38 -04:00
Adam Sigal
a5152b82e9 engine: resources: exec: Add Env
Add functionality to specify environment variables in exec.
2020-04-19 20:30:43 -06:00
James Shubin
e9af8a2595 engine: resources: exec: Clean up error handling
Some quick fixes, this whole resource should be looked at for
discrepancies, since it was written very early.
2020-04-14 22:59:33 -04:00
Adam Sigal
84b5b60d49 engine: resources: exec: Fix typo
Typo in description of cwd field fixed.
2020-04-14 22:36:25 -04:00
James Shubin
8f60f42be3 engine: resources: Add http:server and http:file resources
This adds a new http server resource, as well as a http file resource
that is used to specify files to serve in that server. This allows you
to have an http server that is entirely server from memory without
needing files on disk.

It does this by using the autogrouping magic that is already available
in the engine.

The http resource is not meant to be a full-featured http server
replacement, and it might still be useful to use the venerable webserver
of your choice, however for integrated, pure-golang bootstrapping
environments, this might prove to be very useful.

It can be combined with the tftp and dhcp resources to build PXE setups
with mgmt!

This resource can be extended further to support an http:flag endpoint,
an http:ui endpoint, automatic edges, and more!
2020-04-11 03:23:04 -04:00
James Shubin
583344138a engine: resources: Add dhcp:server and dhcp:host resources
This adds a new dhcp server resource, as well as a dhcp host resource
used to specify the static mapping between mac address and ip address.
It also adds a simple, pure-golang example dhcp client which might make
testing easier.

The dhcp resource is not meant to be a full-featured dhcp server
replacement, and it might still be useful to use the venerable dhcpd,
however for integrated, pure-golang bootstrapping environments, this
might prove to be very useful.

It can be combined with the tftp resource to build PXE setups with mgmt.

This resource can be extended further to support a dhcp:range directive,
automatic edges, and more!
2020-04-11 02:45:18 -04:00
James Shubin
016d021d5a engine: resources: tftp: Improve validation and error messages
Just some small cleanups for our tftp resource. We also rename the
struct to make it consistent, since golint complains about similar
protocols when it is not all capitalized.
2020-04-11 02:45:18 -04:00
James Shubin
115dc4bfa4 engine: resources: net: Add an IP forward field
This adds an IP forward field (boolean) and improves the documentation.
2020-04-11 02:02:35 -04:00
Francois Rompre-Lanctot
f3b4a8d055 engine: resources: Add a test case for resource owner check
This adds FileOwnerExpect as a new Step which allows validating if the
owner was set properly on a resource.
2020-02-10 22:17:42 -05:00
Francois Rompre-Lanctot
104af7e86f engine: resources: Fix typo 2020-02-03 22:57:13 -05:00
James Shubin
90fd8023dd lang, engine: Add a facility for resources to export constants
Since we focus on safety, it would be nice to reduce the chance of any
runtime errors if we made a typo for a resource parameter. With this
patch, each resource can export constants into the global namespace so
that typos would cause a compile error.

Of course in the future if we had a more advanced type system, then we
could support precise types for each individual resource param, but in
an attempt to keep things simple, we'll leave that for another day. It
would add complexity too if we ever wanted to store a parameter
externally.

Lastly, we might consider adding "special case" parsing so that directly
specified fields would parse intelligently. For example, we could allow:

	file "/tmp/hello" {
		state => exists,	# magic sugar!
	}

This isn't supported for now, but if it works after all the other parser
changes have been made, it might be something to consider.
2020-01-29 11:16:04 -05:00
James Shubin
f67ad9c061 test: Add a check for too long or badly reflowed docstrings
This ensures that docstring comments are wrapped to 80 chars. ffrank
seemed to be making this mistake far too often, and it's a silly thing
to look for manually. As it turns out, I've made it too, as have many
others. Now we have a test that checks for most cases. There are still a
few stray cases that aren't checked automatically, but this can be
improved upon if someone is motivated to do so.

Before anyone complains about the 80 character limit: this only checks
docstring comments, not source code length or inline source code
comments. There's no excuse for having docstrings that are badly
reflowed or over 80 chars, particularly if you have an automated test.
2020-01-25 04:43:33 -05:00
James Shubin
fec94aa53a engine, lang: Fix simple test failures
Two bugs sneaked in while pushing old stuff.
2020-01-12 19:35:11 -05:00
James Shubin
579975f08d engine: graph: Don't error when state file is missing
For some reason we get errors when we try to remove a non-existent state
file. There's a slight possibility that it could be a bug we're working
around, but it's not clear that this is the case, and I think it's
possible that a state file could have gotten nuked by the user somehow,
although this was occurring "naturally" when running reverse1.mcl so
let's keep that working for now.
2020-01-12 16:41:09 -05:00
James Shubin
3707b39fef engine: graph: Improve comments
Clarify that we're referring to cycles in the graph, since it needs to
be a DAG.
2020-01-12 16:39:32 -05:00
James Shubin
f07387225b engine: resources: Log more info about tftp errors
This helps for debugging this kind of issue:
https://github.com/pin/tftp/issues/41#issuecomment-570744056
2020-01-03 20:42:34 -05:00
James Shubin
2648fb1bb1 legal: Happy 2020 everyone...
Done with:

ack '2019+' -l | xargs sed -i -e 's/2019+/2020+/g'

Checked manually with:

git add -p

Hello to future James from 2021, and Happy Hacking!
2020-01-03 20:08:37 -05:00
James Shubin
d34715b4ba engine: resources: pippet: Cleanup and proper wrapping
Felix, please configure your editor to wrap at 80 chars and/or help us
write a test for this please =D
2020-01-03 01:20:00 -05:00
Felix Frank
63af50bf98 engine: resources: pippet: Initial implementation for new resource type
The pippet resource implements faster integration of Puppet resources
in mgmt at runtime, by piping synchronization commands to a Puppet
process that keeps running alongside mgmt. This avoids huge overhead
through launching a Puppet process for each operation on a resource
that is delegated to Puppet.
2020-01-03 01:19:37 -05:00
James Shubin
456550c1d4 engine: resources: docker: Make a few fixups
Here are a few fixups to the docker resources. All miscellaneous stuff,
nothing major.
2020-01-03 00:53:20 -05:00
Jonathan Gold
8174b88ec3 engine: resources: docker: Add sensible defaults 2020-01-03 00:30:01 -05:00
Jonathan Gold
3233973748 engine: resources: docker: Add AutoEdges between container and image 2020-01-03 00:30:01 -05:00
Jonathan Gold
bdfb1cf33e engine: resources: docker: Ensure image is specified for containers 2020-01-03 00:30:01 -05:00
Jonathan Gold
1c5fcd59e7 engine: resources: docker: Add a docker image resource 2020-01-03 00:30:01 -05:00
James Shubin
5526bbba64 engine: resources: Add a tftp server and tftp file resource
This adds a tftp server and tftp file resource to help you run a small
pure golang tftp server embedded inside the mgmt resource model.
2019-12-17 03:41:45 -05:00
Donald Bakong
4b4b7dc169 engine: resources: Adding tests to file mode 2019-11-08 10:02:30 -05:00
James Shubin
1eba5833d5 engine: resources: Consistency changes and cleanup for file mode
This makes a few consistency changes and cleanups to the file mode
feature so that it's more in style with the rest of the code base.
2019-11-01 22:01:38 -04:00
Derek Buckley
83a747794e engine: resources: Adds symbolic mode to file resource
Adds a symbolic parsing function to the util package for parsing in the
file resource.
2019-11-01 21:57:10 -04:00
Julien Pivotto
3e16d1da46 engine: resources: Add new consul resource
This commit adds a new consul:kv resource which allows us to set and
watch keys inside a consul kv datastore.

This was started by roidelapluie, and was finished during pair
programming with purpleidea.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
Signed-off-by: James Shubin <james@shubin.ca>
2019-11-01 21:38:08 -04:00
James Shubin
46a71296a9 engine: resources: Add a Purge option to the file resource
This adds a "purge" parameter to the file resource. To do this, we have
to add the API hooks so the file resource can query other resources in
the graph to know if they are present, and as a result whether they
should be excluded from the purge or not.

This is useful for when we have a managed directory with some managed
contents. If a managed file is removed from the directory, then it will
be removed by the file (directory) resource if it has Purge set.
Alternatively, you can use the Reverse meta param, which is sometimes
preferable for this use case and sometimes not. This will be discussed
elsewhere.

This also adds a bunch of tests for this feature.

This also makes a few somewhat related cleanups in the file code.
2019-10-29 11:54:08 -04:00
James Shubin
1285588b62 engine: resources: Fix file absent helper
We should check this nil case too.
2019-10-29 07:15:43 -04:00
James Shubin
d96392f65e engine: resources: Improve error message in test
Nothing major, just improve testing here.
2019-10-29 07:15:43 -04:00
James Shubin
d1c5a736ae engine: resources: Allow nil helper functions in tests
This reduces some of the boilerplate when writing resource tests.
2019-10-29 07:15:43 -04:00
James Shubin
6b1e038c5c engine: resources: Add file exists helper
This will allow us to test even more things!
2019-10-29 07:15:43 -04:00
James Shubin
eaab1aae28 engine: graph, resources: Add filtered graph function
This lets a resource query the resource graph in a controlled way.
2019-10-29 07:15:43 -04:00
James Shubin
31030343a2 engine: Add graph queryable trait
This is a trait that specifies that your resource can be queried by
others. You can either enable it plainly to add wholesale access by
everyone, or you can implement your own Allow method to limit what is
permitted.
2019-10-29 07:15:43 -04:00
James Shubin
325ca03a13 engine: graph: Pass through the graph struct
We want to use it in the resources.
2019-10-29 07:15:43 -04:00
James Shubin
58421fd31a engine: resources: Add fragments support to file resource
This adds a "fragments" mode to the file resource. In addition to
"content" and "source", you can now alternatively build a file from the
file fragments of other files.
2019-10-29 07:15:43 -04:00
James Shubin
06952c224b engine: resources: nspawn: Use populate variable
We referred to the wrong variable. Not a major bug, but would produce a
useless or confusing error message otherwise.
2019-10-21 07:38:42 -04:00
Donald Bakong
fc1c631c98 engine: resources: Change Res API from Compare to Cmp
This will be done by refactoring the current method, to return an error
message instead of a boolean value. This will also update a typo on the
user res.
2019-09-27 18:10:58 -04:00
James Shubin
9dae5ef83b engine: resources: Improve the file res and add strict state
This might be slightly controversial, in that you must specify the state
if a file would need to be created to perform the action. We no longer
implicitly assume that just specifying content is enough. As it turns
out, I believe this is safer and more correct. The code to implement
this turns out to be much more logical and simplified, and does this
removes an ambiguous corner case from the reversed resource code.

Some discussion in: https://github.com/purpleidea/mgmt/issues/540

This patch also does a bit of related cleanup.
2019-09-14 16:07:53 -04:00