Commit Graph

1298 Commits

Author SHA1 Message Date
Jean-Philippe Evrard
01c2131436 misc: Make-deps should assume go vet present
While the code comment says to check if go vet command is present,
it actually tests if go vet command returns properly.

This is a problem if go vet is _not_ returning 0 due to a
failure while go vet is present: it will try to install the
legacy go vet.

This fixes it by removing this block of code completely,
as we assume a golang version which contains it anyway.
2020-12-09 19:25:19 -05:00
Joe Groocock
c274231544 test: Fix implicit test fail in test-markdownlint
Fix the silent test failure by catching the uncaught error from
`command`, handling the failure gracefully.

    $ bash -x test/test-markdownlint.sh
    ...
    ++ command -v mdl
    + MDL=
    $ echo $?
    1

    $ bash -x test/test-markdownlint.sh
    ...
    ++ command -v mdl
    + MDL=
    + true
    + '[' -z '' ']'
    + fail_test 'The '\''mdl'\'' utility can'\''t be found.'
    + echo -e 'FAIL: The '\''mdl'\'' utility can'\''t be found.'
    FAIL: The 'mdl' utility can't be found.

Fix a couple of glaring shellcheck warnings and errors mostly
surrounding variable quoting.

Signed-off-by: Joe Groocock <me@frebib.net>
2020-12-08 15:51:52 +00:00
Joe Groocock
4a2864701c lang: Assert that 'metadata.yaml' is not parsed as raw mcl
Contradictory to expectations, `mgmt run lang metadata.yaml` would
produce an error similar to the following, which is likely unexpected
from the user perspective:

    2020-11-24 12:24:08.330968 I | cli: lang: lexing/parsing...
    2020-11-24 12:24:08.331153 I | run: error: cli parse error: could not generate AST: parser: `syntax error: unexpected DOT` @1:1

Produce a user-friendly warning instead, hinting with the supported
usage:

    2020-11-24 13:15:01.686659 I | run: error: cli parse error: could not activate an input parser: unexpected raw code 'metadata.yaml'. Did you mean './metadata.yaml'?

Signed-off-by: Joe Groocock <me@frebib.net>
2020-11-24 13:56:08 +00:00
James Shubin
76ede10e0a misc: Update path to go-fuzz
They moved it :/
2020-09-23 13:42:51 -04:00
Ahmed Al-Hulaibi
274e01bb75 misc, docs: Update minimum required golang version to 1.13 2020-09-23 12:34:54 -04:00
James Shubin
d75f763c99 misc, docs: Move to golang 1.12 2020-09-23 12:34:54 -04:00
Donald Bakong
5bc985663c docs: Add underscore issue to FAQ 2020-09-23 11:25:46 -04:00
James Shubin
df9e2e853f docs: Update state of remote execution and resource 2020-09-23 11:21:03 -04:00
Ivan Pejić
b4828a6f0a docs: Update FAQ to mention temp absence of remote 2020-09-23 11:15:44 -04:00
Donald Bakong
e99dd749a0 engine: resources: Fixing return bug 2020-09-23 11:10:38 -04:00
David Randall
10ce7178c0 misc: Incorrect dependency path for dvyukov/go-fuzz
Fix the dependency path for dvyukov/go-fuzz to github.com from golang.org.

Fixes #601
2020-06-07 15:05:46 -04:00
Donald Bakong
5c6a66eaf5 lang: funcs: Add cidr_to_ip function 2020-06-03 22:01:01 -04:00
Francois Rompre-Lanctot
36d30bc985 lang: funcs: Add macfmt function 2020-06-03 17:46:39 -04:00
Adam Sigal
a5152b82e9 engine: resources: exec: Add Env
Add functionality to specify environment variables in exec.
2020-04-19 20:30:43 -06:00
James Shubin
e9af8a2595 engine: resources: exec: Clean up error handling
Some quick fixes, this whole resource should be looked at for
discrepancies, since it was written very early.
2020-04-14 22:59:33 -04:00
Adam Sigal
84b5b60d49 engine: resources: exec: Fix typo
Typo in description of cwd field fixed.
2020-04-14 22:36:25 -04:00
James Shubin
8f60f42be3 engine: resources: Add http:server and http:file resources
This adds a new http server resource, as well as a http file resource
that is used to specify files to serve in that server. This allows you
to have an http server that is entirely server from memory without
needing files on disk.

It does this by using the autogrouping magic that is already available
in the engine.

The http resource is not meant to be a full-featured http server
replacement, and it might still be useful to use the venerable webserver
of your choice, however for integrated, pure-golang bootstrapping
environments, this might prove to be very useful.

It can be combined with the tftp and dhcp resources to build PXE setups
with mgmt!

This resource can be extended further to support an http:flag endpoint,
an http:ui endpoint, automatic edges, and more!
2020-04-11 03:23:04 -04:00
James Shubin
583344138a engine: resources: Add dhcp:server and dhcp:host resources
This adds a new dhcp server resource, as well as a dhcp host resource
used to specify the static mapping between mac address and ip address.
It also adds a simple, pure-golang example dhcp client which might make
testing easier.

The dhcp resource is not meant to be a full-featured dhcp server
replacement, and it might still be useful to use the venerable dhcpd,
however for integrated, pure-golang bootstrapping environments, this
might prove to be very useful.

It can be combined with the tftp resource to build PXE setups with mgmt.

This resource can be extended further to support a dhcp:range directive,
automatic edges, and more!
2020-04-11 02:45:18 -04:00
James Shubin
016d021d5a engine: resources: tftp: Improve validation and error messages
Just some small cleanups for our tftp resource. We also rename the
struct to make it consistent, since golint complains about similar
protocols when it is not all capitalized.
2020-04-11 02:45:18 -04:00
James Shubin
115dc4bfa4 engine: resources: net: Add an IP forward field
This adds an IP forward field (boolean) and improves the documentation.
2020-04-11 02:02:35 -04:00
James Shubin
5b83febb23 etcd: Log an error that wasn't getting seen
We would error when the address could not bind, but it was for
non-standard reasons, and we didn't see the specific reason why.
2020-04-11 02:02:35 -04:00
James Shubin
c9d5c50402 test: Improve our test for long lines
This now allows long URL's to start part way through a sentence instead
of requiring them to start on the beginning of a new line.
2020-04-03 01:15:57 -04:00
James Shubin
fc839d2983 vendor: Fix broken upstream hashicorp module
This module broke compat with old versions of golang. Vendor it until
we're at a minimum of golang 1.13.x everywhere.
2020-04-02 22:27:06 -04:00
James Shubin
3bce96bbd5 docs: Add new talk from cfgmgmtcamp 2020
Those first ten seconds of the video are awesome!
2020-02-29 19:40:26 -05:00
Patrick Meyer
6279be073b lang: Prevent struct types with duplicate field names
The previous fix for #591 in 70eecd5 didn't address all issues
concerning duplicate struct field names. It still crashed for inputs
like `$d []struct{x int; x float}`. Note the different types but
duplicate names.
2020-02-29 19:11:51 -05:00
Patrick Meyer
ea37132ce4 lang: Fix wrong go-fuzz bin name
I missed renaming this after moving the fuzz.go from the lang package
to its own fuzz package.
2020-02-29 05:28:38 +01:00
James Shubin
70eecd5289 lang: Prevent struct types with duplicate fields
Struct types with duplicate fields are invalid types and weren't caught
by the parser. This fixes the issue and adds some associated tests. It
also checks and tests for duplicate struct value field names.

As a technical side-note, this doesn't change the lang/types/ functions
to remove panics-- the signatures are simplified to make their use
simple, and we intentionally panic if they're used incorrectly. In this
case, one was being used without having previously validated the input.

Thanks to Patrick Meyer for finding this issue via fuzzing!
2020-02-27 18:52:02 -05:00
James Shubin
380d03257f misc, lang: Small fixups
Change some minor style issues.
2020-02-27 17:34:03 -05:00
Patrick Meyer
006de6da14 lang: Add fuzz target to lang Makefile 2020-02-26 12:07:00 +01:00
Kenneth Hoste
10aa80e8f5 docs: Add link to recording of James' FOSDEM 2020 talks 2020-02-17 14:23:20 -05:00
Felix Frank
013439af6d test: Make prometheus tests safer and more verbose 2020-02-16 18:38:58 -05:00
Felix Frank
3408961155 test: Fix syntax in the loadavg test 2020-02-16 18:38:43 -05:00
Francois Rompre-Lanctot
f3b4a8d055 engine: resources: Add a test case for resource owner check
This adds FileOwnerExpect as a new Step which allows validating if the
owner was set properly on a resource.
2020-02-10 22:17:42 -05:00
Francois Rompre-Lanctot
104af7e86f engine: resources: Fix typo 2020-02-03 22:57:13 -05:00
James Shubin
be39fbeff6 examples: lang: Update examples 2020-02-01 16:48:23 -05:00
James Shubin
4109045fa4 github: Add new needinfo tag 2020-01-29 11:38:53 -05:00
James Shubin
90fd8023dd lang, engine: Add a facility for resources to export constants
Since we focus on safety, it would be nice to reduce the chance of any
runtime errors if we made a typo for a resource parameter. With this
patch, each resource can export constants into the global namespace so
that typos would cause a compile error.

Of course in the future if we had a more advanced type system, then we
could support precise types for each individual resource param, but in
an attempt to keep things simple, we'll leave that for another day. It
would add complexity too if we ever wanted to store a parameter
externally.

Lastly, we might consider adding "special case" parsing so that directly
specified fields would parse intelligently. For example, we could allow:

	file "/tmp/hello" {
		state => exists,	# magic sugar!
	}

This isn't supported for now, but if it works after all the other parser
changes have been made, it might be something to consider.
2020-01-29 11:16:04 -05:00
James Shubin
f67ad9c061 test: Add a check for too long or badly reflowed docstrings
This ensures that docstring comments are wrapped to 80 chars. ffrank
seemed to be making this mistake far too often, and it's a silly thing
to look for manually. As it turns out, I've made it too, as have many
others. Now we have a test that checks for most cases. There are still a
few stray cases that aren't checked automatically, but this can be
improved upon if someone is motivated to do so.

Before anyone complains about the 80 character limit: this only checks
docstring comments, not source code length or inline source code
comments. There's no excuse for having docstrings that are badly
reflowed or over 80 chars, particularly if you have an automated test.
2020-01-25 04:43:33 -05:00
James Shubin
525e2bafee misc: Let sigtee work on older golang versions 2020-01-25 04:43:33 -05:00
James Shubin
b65a9abf8e misc: Add two scripts to help debug things
This adds two new helper scripts that are good for debugging mgmt. Just
build and add `sigtee` to your ~/bin/ along with filter-golang-stack.py
and mgmt_debug.sh and use the later to call mgmt as you would normally.

For example, I might do:

$ mgmt_debug.sh ./mgmt run --tmp-prefix lang examples/lang/hello0.mcl

And if I kill it with ^\ then I'll get a filtered trace at the end in my
$PAGER (which is assumed to be `less`) and this should make my life
easier.

As a cool bonus, this means we use bash, python, and golang all
together!
2020-01-13 01:07:00 -05:00
James Shubin
fec94aa53a engine, lang: Fix simple test failures
Two bugs sneaked in while pushing old stuff.
2020-01-12 19:35:11 -05:00
James Shubin
3d4b345728 examples: lang: Improve reverse example
It's cool to show just the mode changes.
2020-01-12 17:42:20 -05:00
James Shubin
579975f08d engine: graph: Don't error when state file is missing
For some reason we get errors when we try to remove a non-existent state
file. There's a slight possibility that it could be a bug we're working
around, but it's not clear that this is the case, and I think it's
possible that a state file could have gotten nuked by the user somehow,
although this was occurring "naturally" when running reverse1.mcl so
let's keep that working for now.
2020-01-12 16:41:09 -05:00
James Shubin
3707b39fef engine: graph: Improve comments
Clarify that we're referring to cycles in the graph, since it needs to
be a DAG.
2020-01-12 16:39:32 -05:00
James Shubin
f07387225b engine: resources: Log more info about tftp errors
This helps for debugging this kind of issue:
https://github.com/pin/tftp/issues/41#issuecomment-570744056
2020-01-03 20:42:34 -05:00
James Shubin
2648fb1bb1 legal: Happy 2020 everyone...
Done with:

ack '2019+' -l | xargs sed -i -e 's/2019+/2020+/g'

Checked manually with:

git add -p

Hello to future James from 2021, and Happy Hacking!
2020-01-03 20:08:37 -05:00
James Shubin
d34715b4ba engine: resources: pippet: Cleanup and proper wrapping
Felix, please configure your editor to wrap at 80 chars and/or help us
write a test for this please =D
2020-01-03 01:20:00 -05:00
Felix Frank
63af50bf98 engine: resources: pippet: Initial implementation for new resource type
The pippet resource implements faster integration of Puppet resources
in mgmt at runtime, by piping synchronization commands to a Puppet
process that keeps running alongside mgmt. This avoids huge overhead
through launching a Puppet process for each operation on a resource
that is delegated to Puppet.
2020-01-03 01:19:37 -05:00
James Shubin
456550c1d4 engine: resources: docker: Make a few fixups
Here are a few fixups to the docker resources. All miscellaneous stuff,
nothing major.
2020-01-03 00:53:20 -05:00
Jonathan Gold
8174b88ec3 engine: resources: docker: Add sensible defaults 2020-01-03 00:30:01 -05:00