Commit Graph

1514 Commits

Author SHA1 Message Date
Joe Groocock
7cace52ab5 test: prevent LinuxBrew in GitHub Actions CI
Ubuntu-latest in GitHub Actions provides linuxbrew, so the tests install
both the native Debian dependency packages, and also the linuxbrew
variants which is slower and entirely redundant.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-12 16:44:12 +00:00
Joe Groocock
95b93c60d9 test: Invert negative bash assertions
In bash `-n` is `non zero length` which is the opposite of `-z` meaning
`zero length`. `-n` is semantically identical to `! -z` but `-n`.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-12 16:44:12 +00:00
Joe Groocock
5af1dcb8b1 test: Add in_ci utility test function
in_ci checks for environment variables set by a selection of CI systems
and returns true if the test appears to be running in CI. Additionally
it can test for specific CI systems, and returns true if the CI system
is listed.

Deduplicate existing environment checks for Travis and Jenkins.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-12 16:44:11 +00:00
Joe Groocock
6a61774fb7 docker: Bump docker dependencies, add containerd
These dependencies are maintained because the upstream repos bundle
vendor directories into the repos, which cause namespacing issues during
build. Git submodules don't strip the vendor directory whereas most
vendoring tools would.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-09 21:14:34 +00:00
James Shubin
ccbaca24f1 gopath: Remove this unused directory
I had this symlink hack a long time ago. I don't think it's being used
anymore.
2021-02-07 21:23:49 -05:00
Joe Groocock
07b6048dc5 etcd: Bump etcd + friends to the latest upstream version
This allows dropping the pinned grpc-prometheus and grpc-gateway
libraries as git master works fine for now.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-07 12:55:02 +00:00
Joe Groocock
60dd34d066 make: Drop support for Go 1.9 in make build
docs/development.md says the minimum required Golang version is 1.13 at
the time of writing.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-07 00:08:51 -05:00
James Shubin
28451d1e14 lang: funcs: Fixup race in vumeter example
The vumeter example was written quickly and without much care. This
fixes a possible race (panic) and also removes the busy loop that wastes
CPU while we're waiting for the first value to come in.
2021-02-06 23:59:06 -05:00
Joe Groocock
db95b6381f examples: lang: Reinstate mcl as unification bug is fixed
Struct field names now correctly map based on their `lang` tags in Go
structs, so this example now works as originally intended.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-06 16:57:01 +00:00
Joe Groocock
6b14c9bea4 lang: Map Go struct fields using lang struct tag
Converting a reflect.Type of KindStruct did not respect the `lang` tag
on struct fields incidating how fields from mcl structs should be mapped
even though resource field names did. This patch should allow structs
with mapped fields to be respected.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-06 16:57:01 +00:00
Joe Groocock
742adc00fe lang: Convert StmtRes to engine.Res with types.Into()
Replace existing field-mapping code with calls to types.Into() to
reflect the mcl data into the Go resource struct with finer granularity
and accuracy, and less reliance on the magic reflect.Set() function.

One major advantage over reflect.Value.Set() is Into() allows tailoring
the data that is set, specifically when coercing mcl struct values into
Golang struct values, the fields can be appropriately mapped based on
the lang tag on the struct field. With reflect.Value.Set() this was not
at all possible as there was a contradiction of logic given the
following rules:

- mcl struct fields must have lowercase names
- Golang struct fields with lowercase names are unexported
- Golang reflection does not allow modifying unexported fields

Prior to this change, it was impossible to map an mcl inline struct in a
resource to the matched Golang counterpart, even if the lang tag was
present on the struct field. This can be demonstrated with the following
trivial example:

    test "name" {
        key => struct{
            name => "hello",
        },
    }

and the accompanying Golang resource definition:

    type TestRes struct {
        traits.Base
        traits.Edgeable

        Key struct {
            Name string `lang:"name"`
        } `lang:"key"`
    }

Due to the mismatch in field names in the embedded struct, the type
unifier failed and refused to match mcl 'name' to Go 'Name' due to the
missing mapping logic.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-06 16:57:00 +00:00
Joe Groocock
52897cc16c lang: Implement Into() to set types.Value from reflect.Value
Into() mutates a given reflect.Value and sets the data represented by
the types.Value into the storage represented by the reflect.Value.

Into() is the opposite to ValueOf() which converts reflect.Value into
types.Value, and in theory they should be (almost) bijective with some
edge case exceptions where the conversion is lossy.

Simply, it replaces reflect.Value.Set() in the broad case, giving finer
control of how the reflect.Value is modified and how the data is set.
types.Value.Value() is now also a redundant function that achieves the
same outcome as Into(), but with less type specificity.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-06 16:57:00 +00:00
Joe Groocock
c950568f1b lang: Move StructTag const into lang/types
This constant value is strongly tied to the language, and little to do
with the engine. Move the definition into the lang/types package to
prevent circular imports between lang/types and engine/util.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-06 16:56:57 +00:00
Joe Groocock
845d7ff188 lang: Fix panic in lang/types/ValueOf() for Struct
Replace use of reflect.Value.Len() with NumField() which is intended to
return the number of fields in reflected Struct value.

Len should only be used for Array, Chan, Map, Slice and String types.

Add some trivial sanity check tests for ValueOf() for the simple and
complex container types.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-01-31 23:10:16 +00:00
James Shubin
3bd8658da6 art: Use a transparent logo for dark themes
As helpful user frebib pointed out, our logo doesn't render nicely when
github users use the dark theme. This has already been complained about
by others in:

https://github.community/t/support-theme-context-for-images-in-light-vs-dark-mode/147981/2

For now, we'll switch to a transparent background.
2021-01-31 18:00:47 -05:00
James Shubin
336a38081a legal: Happy 2021 everyone...
Done with:

ack '2020+' -l | xargs sed -i -e 's/2020+/2021+/g'

Checked manually with:

git add -p

Hello to future James from 2022, and Happy Hacking!
2021-01-31 16:52:46 -05:00
Jean-Philippe Evrard
01c2131436 misc: Make-deps should assume go vet present
While the code comment says to check if go vet command is present,
it actually tests if go vet command returns properly.

This is a problem if go vet is _not_ returning 0 due to a
failure while go vet is present: it will try to install the
legacy go vet.

This fixes it by removing this block of code completely,
as we assume a golang version which contains it anyway.
2020-12-09 19:25:19 -05:00
Joe Groocock
c274231544 test: Fix implicit test fail in test-markdownlint
Fix the silent test failure by catching the uncaught error from
`command`, handling the failure gracefully.

    $ bash -x test/test-markdownlint.sh
    ...
    ++ command -v mdl
    + MDL=
    $ echo $?
    1

    $ bash -x test/test-markdownlint.sh
    ...
    ++ command -v mdl
    + MDL=
    + true
    + '[' -z '' ']'
    + fail_test 'The '\''mdl'\'' utility can'\''t be found.'
    + echo -e 'FAIL: The '\''mdl'\'' utility can'\''t be found.'
    FAIL: The 'mdl' utility can't be found.

Fix a couple of glaring shellcheck warnings and errors mostly
surrounding variable quoting.

Signed-off-by: Joe Groocock <me@frebib.net>
2020-12-08 15:51:52 +00:00
Joe Groocock
4a2864701c lang: Assert that 'metadata.yaml' is not parsed as raw mcl
Contradictory to expectations, `mgmt run lang metadata.yaml` would
produce an error similar to the following, which is likely unexpected
from the user perspective:

    2020-11-24 12:24:08.330968 I | cli: lang: lexing/parsing...
    2020-11-24 12:24:08.331153 I | run: error: cli parse error: could not generate AST: parser: `syntax error: unexpected DOT` @1:1

Produce a user-friendly warning instead, hinting with the supported
usage:

    2020-11-24 13:15:01.686659 I | run: error: cli parse error: could not activate an input parser: unexpected raw code 'metadata.yaml'. Did you mean './metadata.yaml'?

Signed-off-by: Joe Groocock <me@frebib.net>
2020-11-24 13:56:08 +00:00
James Shubin
76ede10e0a misc: Update path to go-fuzz
They moved it :/
2020-09-23 13:42:51 -04:00
Ahmed Al-Hulaibi
274e01bb75 misc, docs: Update minimum required golang version to 1.13 2020-09-23 12:34:54 -04:00
James Shubin
d75f763c99 misc, docs: Move to golang 1.12 2020-09-23 12:34:54 -04:00
Donald Bakong
5bc985663c docs: Add underscore issue to FAQ 2020-09-23 11:25:46 -04:00
James Shubin
df9e2e853f docs: Update state of remote execution and resource 2020-09-23 11:21:03 -04:00
Ivan Pejić
b4828a6f0a docs: Update FAQ to mention temp absence of remote 2020-09-23 11:15:44 -04:00
Donald Bakong
e99dd749a0 engine: resources: Fixing return bug 2020-09-23 11:10:38 -04:00
David Randall
10ce7178c0 misc: Incorrect dependency path for dvyukov/go-fuzz
Fix the dependency path for dvyukov/go-fuzz to github.com from golang.org.

Fixes #601
2020-06-07 15:05:46 -04:00
Donald Bakong
5c6a66eaf5 lang: funcs: Add cidr_to_ip function 2020-06-03 22:01:01 -04:00
Francois Rompre-Lanctot
36d30bc985 lang: funcs: Add macfmt function 2020-06-03 17:46:39 -04:00
Adam Sigal
a5152b82e9 engine: resources: exec: Add Env
Add functionality to specify environment variables in exec.
2020-04-19 20:30:43 -06:00
James Shubin
e9af8a2595 engine: resources: exec: Clean up error handling
Some quick fixes, this whole resource should be looked at for
discrepancies, since it was written very early.
2020-04-14 22:59:33 -04:00
Adam Sigal
84b5b60d49 engine: resources: exec: Fix typo
Typo in description of cwd field fixed.
2020-04-14 22:36:25 -04:00
James Shubin
8f60f42be3 engine: resources: Add http:server and http:file resources
This adds a new http server resource, as well as a http file resource
that is used to specify files to serve in that server. This allows you
to have an http server that is entirely server from memory without
needing files on disk.

It does this by using the autogrouping magic that is already available
in the engine.

The http resource is not meant to be a full-featured http server
replacement, and it might still be useful to use the venerable webserver
of your choice, however for integrated, pure-golang bootstrapping
environments, this might prove to be very useful.

It can be combined with the tftp and dhcp resources to build PXE setups
with mgmt!

This resource can be extended further to support an http:flag endpoint,
an http:ui endpoint, automatic edges, and more!
2020-04-11 03:23:04 -04:00
James Shubin
583344138a engine: resources: Add dhcp:server and dhcp:host resources
This adds a new dhcp server resource, as well as a dhcp host resource
used to specify the static mapping between mac address and ip address.
It also adds a simple, pure-golang example dhcp client which might make
testing easier.

The dhcp resource is not meant to be a full-featured dhcp server
replacement, and it might still be useful to use the venerable dhcpd,
however for integrated, pure-golang bootstrapping environments, this
might prove to be very useful.

It can be combined with the tftp resource to build PXE setups with mgmt.

This resource can be extended further to support a dhcp:range directive,
automatic edges, and more!
2020-04-11 02:45:18 -04:00
James Shubin
016d021d5a engine: resources: tftp: Improve validation and error messages
Just some small cleanups for our tftp resource. We also rename the
struct to make it consistent, since golint complains about similar
protocols when it is not all capitalized.
2020-04-11 02:45:18 -04:00
James Shubin
115dc4bfa4 engine: resources: net: Add an IP forward field
This adds an IP forward field (boolean) and improves the documentation.
2020-04-11 02:02:35 -04:00
James Shubin
5b83febb23 etcd: Log an error that wasn't getting seen
We would error when the address could not bind, but it was for
non-standard reasons, and we didn't see the specific reason why.
2020-04-11 02:02:35 -04:00
James Shubin
c9d5c50402 test: Improve our test for long lines
This now allows long URL's to start part way through a sentence instead
of requiring them to start on the beginning of a new line.
2020-04-03 01:15:57 -04:00
James Shubin
fc839d2983 vendor: Fix broken upstream hashicorp module
This module broke compat with old versions of golang. Vendor it until
we're at a minimum of golang 1.13.x everywhere.
2020-04-02 22:27:06 -04:00
James Shubin
3bce96bbd5 docs: Add new talk from cfgmgmtcamp 2020
Those first ten seconds of the video are awesome!
2020-02-29 19:40:26 -05:00
Patrick Meyer
6279be073b lang: Prevent struct types with duplicate field names
The previous fix for #591 in 70eecd5 didn't address all issues
concerning duplicate struct field names. It still crashed for inputs
like `$d []struct{x int; x float}`. Note the different types but
duplicate names.
2020-02-29 19:11:51 -05:00
Patrick Meyer
ea37132ce4 lang: Fix wrong go-fuzz bin name
I missed renaming this after moving the fuzz.go from the lang package
to its own fuzz package.
2020-02-29 05:28:38 +01:00
James Shubin
70eecd5289 lang: Prevent struct types with duplicate fields
Struct types with duplicate fields are invalid types and weren't caught
by the parser. This fixes the issue and adds some associated tests. It
also checks and tests for duplicate struct value field names.

As a technical side-note, this doesn't change the lang/types/ functions
to remove panics-- the signatures are simplified to make their use
simple, and we intentionally panic if they're used incorrectly. In this
case, one was being used without having previously validated the input.

Thanks to Patrick Meyer for finding this issue via fuzzing!
2020-02-27 18:52:02 -05:00
James Shubin
380d03257f misc, lang: Small fixups
Change some minor style issues.
2020-02-27 17:34:03 -05:00
Patrick Meyer
006de6da14 lang: Add fuzz target to lang Makefile 2020-02-26 12:07:00 +01:00
Kenneth Hoste
10aa80e8f5 docs: Add link to recording of James' FOSDEM 2020 talks 2020-02-17 14:23:20 -05:00
Felix Frank
013439af6d test: Make prometheus tests safer and more verbose 2020-02-16 18:38:58 -05:00
Felix Frank
3408961155 test: Fix syntax in the loadavg test 2020-02-16 18:38:43 -05:00
Francois Rompre-Lanctot
f3b4a8d055 engine: resources: Add a test case for resource owner check
This adds FileOwnerExpect as a new Step which allows validating if the
owner was set properly on a resource.
2020-02-10 22:17:42 -05:00
Francois Rompre-Lanctot
104af7e86f engine: resources: Fix typo 2020-02-03 22:57:13 -05:00