Commit Graph

1334 Commits

Author SHA1 Message Date
James Shubin
525b4e6a53 lang: Move core unification structs into shared interfaces package
We should probably move these into the central interfaces package so
that these can be used from multiple places. They don't have any
dependencies, and it doesn't make sense to have the solver code mixed in
to the same package. Overall the interface being implemented here could
probably be improved, but that's a project for another day.
2021-05-02 00:52:57 -04:00
James Shubin
054eaf65b8 util: safepath: Add a new safe path helper library
This is a new path manipulation library that is designed to be safer
than using simple strings for everything. It is more work to use, but it
can help you keep track of the different path types.

It has been sitting unused in a git branch for too long, and I figured
it should see the light of day in case someone wants to start using it.
2021-05-02 00:52:57 -04:00
James Shubin
48fa796ab1 test: Disable failing test
Hit another intermittent failed test in GH CI.
2021-03-10 03:36:27 -05:00
Jean-Philippe Evrard
1873e022cc test: Add guard when no commit needs testing
Without this patch, github actions fail.

It's a temporary workaround until [1] is done.

[1]: https://github.com/purpleidea/mgmt/issues/643
2021-03-03 10:49:22 +01:00
James Shubin
35a8062b58 test: Disable travis IRC notifications for now
One of our contributors is unusually annoyed by them, and it's important
to keep your contributors happy!
2021-03-03 04:28:53 -05:00
Jean-Philippe Evrard
636248ad67 test: Ensure branches are also testable
test-commit-message runs on PR, but also on push in other branches
which aren't PRs. We need to test those too.

This is fixed by ensuring the same kind of behaviour than travis CI:
When a patch is put on a branch, it's using the branchname for
testing [1].

[1]:
https://docs-staging.travis-ci.com/user/environment-variables/#default-environment-variables
2021-03-03 09:52:49 +01:00
Jean-Philippe Evrard
4511c54fad test: Ensure github CI tests commit messages
Without this patch, the travis var is empty, and we just pass.
This is a problem, as we are using github CI nowadays.

This should fix it.
2021-03-03 02:47:30 -05:00
James Shubin
7f3970541b test: Skip more tests
I think some of these fail due to shared environments and noisy
neighbours in github. We'll have to fix that eventually or test
elsewhere.
2021-03-02 13:41:00 -05:00
James Shubin
4040f4d151 test: Skip yet another intermittent test
We shall not have intermittent tests!
2021-03-02 12:48:44 -05:00
James Shubin
887d374c53 lang: funcs: Catch simple function api usage without types
In case a programmer makes a mistake and passes in a function using the
simple function API without a type or even without the entire value,
we'll now return a sensible error message and panic in init() instead of
requiring a test to catch this alone.
2021-02-28 22:50:51 -05:00
James Shubin
be4b87155d test: Skip another intermittent test
I think this might be related to multiple jobs running at the same time
on the same host. Not sure though.
2021-02-24 04:00:34 -05:00
James Shubin
b987a7da4c examples: tftp: Fix missing error checking in example 2021-02-20 13:31:45 -05:00
James Shubin
7153fe5ad2 test: Skip intermittent tests
It would be great to fix some rare races or debug what's wrong in CI,
but for now let's get rid of these fails so that we can get better data
for when we break something more serious. We'll need to revisit all of
this for sure.
2021-02-19 21:17:57 -05:00
James Shubin
ccd8ba44d9 test: Exclude generated ragel parser from golint 2021-02-17 22:01:34 -05:00
James Shubin
e7ef0f7a6c test: Set default column size if $TERM env var isn't set
Seems Github actions breaks or unsets this, leading to the errors:

tput: No value for $TERM and no -T specified
seq: missing operand
Try 'seq --help' for more information.

Hopefully this makes things a bit more robust.
2021-02-17 04:07:30 -05:00
James Shubin
400b58c0e9 lang: Improve string interpolation
The original string interpolation was based on hil which didn't allow
proper escaping, since they used a different escape pattern. Secondly,
the golang Unquote function didn't deal with the variable substitution,
which meant it had to be performed in a second step.

Most importantly, because we did this partial job in Unquote (the fact
that is strips the leading and trailing quotes tricked me into thinking
I was done with interpolation!) it was impossible to remedy the
remaining parts in a second pass with hil. Both operations needs to be
done in a single step. This is logical when you aren't tunnel visioned.

This patch replaces both of these so that string interpolation works
properly. This removes the ability to allow inline function calls in a
string, however this was an incidental feature, and it's not clear that
having it is a good idea. It also requires you wrap the var name with
curly braces. (They are not optional.)

This comes with a load of tests, but I think I got some of it wrong,
since I'm quite new at ragel. If you find something, please say so =D In
any case, this is much better than the original hil implementation, and
easy for a new contributor to patch to make the necessary fixes.
2021-02-17 03:35:12 -05:00
James Shubin
5257496214 test: Make a few cosmetic changes and enable race testing 2021-02-17 02:45:47 -05:00
Jean-Philippe Evrard
e1bfe4a3ce test: Add GitHub Actions test support
Authored-By: Jean-Philippe Evrard <open-source@a.spamming.party>
Co-Authored-By: James Shubin <james@shubin.ca>
Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-14 21:41:02 -05:00
James Shubin
f31cce8ec2 test: misc: Clarify golang wording 2021-02-14 21:39:01 -05:00
Joe Groocock
169ebfa72c test: make-deps: Add folds around tests and dep blocks
Improves readability of CI test output and hides away the complexity
when in most cases it is not required. Retain fold behaviour for both
Travis and GitHub Actions in case both are used in any capacity.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-12 16:44:12 +00:00
Joe Groocock
7cace52ab5 test: prevent LinuxBrew in GitHub Actions CI
Ubuntu-latest in GitHub Actions provides linuxbrew, so the tests install
both the native Debian dependency packages, and also the linuxbrew
variants which is slower and entirely redundant.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-12 16:44:12 +00:00
Joe Groocock
95b93c60d9 test: Invert negative bash assertions
In bash `-n` is `non zero length` which is the opposite of `-z` meaning
`zero length`. `-n` is semantically identical to `! -z` but `-n`.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-12 16:44:12 +00:00
Joe Groocock
5af1dcb8b1 test: Add in_ci utility test function
in_ci checks for environment variables set by a selection of CI systems
and returns true if the test appears to be running in CI. Additionally
it can test for specific CI systems, and returns true if the CI system
is listed.

Deduplicate existing environment checks for Travis and Jenkins.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-12 16:44:11 +00:00
Joe Groocock
6a61774fb7 docker: Bump docker dependencies, add containerd
These dependencies are maintained because the upstream repos bundle
vendor directories into the repos, which cause namespacing issues during
build. Git submodules don't strip the vendor directory whereas most
vendoring tools would.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-09 21:14:34 +00:00
James Shubin
ccbaca24f1 gopath: Remove this unused directory
I had this symlink hack a long time ago. I don't think it's being used
anymore.
2021-02-07 21:23:49 -05:00
Joe Groocock
07b6048dc5 etcd: Bump etcd + friends to the latest upstream version
This allows dropping the pinned grpc-prometheus and grpc-gateway
libraries as git master works fine for now.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-07 12:55:02 +00:00
Joe Groocock
60dd34d066 make: Drop support for Go 1.9 in make build
docs/development.md says the minimum required Golang version is 1.13 at
the time of writing.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-07 00:08:51 -05:00
James Shubin
28451d1e14 lang: funcs: Fixup race in vumeter example
The vumeter example was written quickly and without much care. This
fixes a possible race (panic) and also removes the busy loop that wastes
CPU while we're waiting for the first value to come in.
2021-02-06 23:59:06 -05:00
Joe Groocock
db95b6381f examples: lang: Reinstate mcl as unification bug is fixed
Struct field names now correctly map based on their `lang` tags in Go
structs, so this example now works as originally intended.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-06 16:57:01 +00:00
Joe Groocock
6b14c9bea4 lang: Map Go struct fields using lang struct tag
Converting a reflect.Type of KindStruct did not respect the `lang` tag
on struct fields incidating how fields from mcl structs should be mapped
even though resource field names did. This patch should allow structs
with mapped fields to be respected.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-06 16:57:01 +00:00
Joe Groocock
742adc00fe lang: Convert StmtRes to engine.Res with types.Into()
Replace existing field-mapping code with calls to types.Into() to
reflect the mcl data into the Go resource struct with finer granularity
and accuracy, and less reliance on the magic reflect.Set() function.

One major advantage over reflect.Value.Set() is Into() allows tailoring
the data that is set, specifically when coercing mcl struct values into
Golang struct values, the fields can be appropriately mapped based on
the lang tag on the struct field. With reflect.Value.Set() this was not
at all possible as there was a contradiction of logic given the
following rules:

- mcl struct fields must have lowercase names
- Golang struct fields with lowercase names are unexported
- Golang reflection does not allow modifying unexported fields

Prior to this change, it was impossible to map an mcl inline struct in a
resource to the matched Golang counterpart, even if the lang tag was
present on the struct field. This can be demonstrated with the following
trivial example:

    test "name" {
        key => struct{
            name => "hello",
        },
    }

and the accompanying Golang resource definition:

    type TestRes struct {
        traits.Base
        traits.Edgeable

        Key struct {
            Name string `lang:"name"`
        } `lang:"key"`
    }

Due to the mismatch in field names in the embedded struct, the type
unifier failed and refused to match mcl 'name' to Go 'Name' due to the
missing mapping logic.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-06 16:57:00 +00:00
Joe Groocock
52897cc16c lang: Implement Into() to set types.Value from reflect.Value
Into() mutates a given reflect.Value and sets the data represented by
the types.Value into the storage represented by the reflect.Value.

Into() is the opposite to ValueOf() which converts reflect.Value into
types.Value, and in theory they should be (almost) bijective with some
edge case exceptions where the conversion is lossy.

Simply, it replaces reflect.Value.Set() in the broad case, giving finer
control of how the reflect.Value is modified and how the data is set.
types.Value.Value() is now also a redundant function that achieves the
same outcome as Into(), but with less type specificity.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-06 16:57:00 +00:00
Joe Groocock
c950568f1b lang: Move StructTag const into lang/types
This constant value is strongly tied to the language, and little to do
with the engine. Move the definition into the lang/types package to
prevent circular imports between lang/types and engine/util.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-06 16:56:57 +00:00
Joe Groocock
845d7ff188 lang: Fix panic in lang/types/ValueOf() for Struct
Replace use of reflect.Value.Len() with NumField() which is intended to
return the number of fields in reflected Struct value.

Len should only be used for Array, Chan, Map, Slice and String types.

Add some trivial sanity check tests for ValueOf() for the simple and
complex container types.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-01-31 23:10:16 +00:00
James Shubin
3bd8658da6 art: Use a transparent logo for dark themes
As helpful user frebib pointed out, our logo doesn't render nicely when
github users use the dark theme. This has already been complained about
by others in:

https://github.community/t/support-theme-context-for-images-in-light-vs-dark-mode/147981/2

For now, we'll switch to a transparent background.
2021-01-31 18:00:47 -05:00
James Shubin
336a38081a legal: Happy 2021 everyone...
Done with:

ack '2020+' -l | xargs sed -i -e 's/2020+/2021+/g'

Checked manually with:

git add -p

Hello to future James from 2022, and Happy Hacking!
2021-01-31 16:52:46 -05:00
Jean-Philippe Evrard
01c2131436 misc: Make-deps should assume go vet present
While the code comment says to check if go vet command is present,
it actually tests if go vet command returns properly.

This is a problem if go vet is _not_ returning 0 due to a
failure while go vet is present: it will try to install the
legacy go vet.

This fixes it by removing this block of code completely,
as we assume a golang version which contains it anyway.
2020-12-09 19:25:19 -05:00
Joe Groocock
c274231544 test: Fix implicit test fail in test-markdownlint
Fix the silent test failure by catching the uncaught error from
`command`, handling the failure gracefully.

    $ bash -x test/test-markdownlint.sh
    ...
    ++ command -v mdl
    + MDL=
    $ echo $?
    1

    $ bash -x test/test-markdownlint.sh
    ...
    ++ command -v mdl
    + MDL=
    + true
    + '[' -z '' ']'
    + fail_test 'The '\''mdl'\'' utility can'\''t be found.'
    + echo -e 'FAIL: The '\''mdl'\'' utility can'\''t be found.'
    FAIL: The 'mdl' utility can't be found.

Fix a couple of glaring shellcheck warnings and errors mostly
surrounding variable quoting.

Signed-off-by: Joe Groocock <me@frebib.net>
2020-12-08 15:51:52 +00:00
Joe Groocock
4a2864701c lang: Assert that 'metadata.yaml' is not parsed as raw mcl
Contradictory to expectations, `mgmt run lang metadata.yaml` would
produce an error similar to the following, which is likely unexpected
from the user perspective:

    2020-11-24 12:24:08.330968 I | cli: lang: lexing/parsing...
    2020-11-24 12:24:08.331153 I | run: error: cli parse error: could not generate AST: parser: `syntax error: unexpected DOT` @1:1

Produce a user-friendly warning instead, hinting with the supported
usage:

    2020-11-24 13:15:01.686659 I | run: error: cli parse error: could not activate an input parser: unexpected raw code 'metadata.yaml'. Did you mean './metadata.yaml'?

Signed-off-by: Joe Groocock <me@frebib.net>
2020-11-24 13:56:08 +00:00
James Shubin
76ede10e0a misc: Update path to go-fuzz
They moved it :/
2020-09-23 13:42:51 -04:00
Ahmed Al-Hulaibi
274e01bb75 misc, docs: Update minimum required golang version to 1.13 2020-09-23 12:34:54 -04:00
James Shubin
d75f763c99 misc, docs: Move to golang 1.12 2020-09-23 12:34:54 -04:00
Donald Bakong
5bc985663c docs: Add underscore issue to FAQ 2020-09-23 11:25:46 -04:00
James Shubin
df9e2e853f docs: Update state of remote execution and resource 2020-09-23 11:21:03 -04:00
Ivan Pejić
b4828a6f0a docs: Update FAQ to mention temp absence of remote 2020-09-23 11:15:44 -04:00
Donald Bakong
e99dd749a0 engine: resources: Fixing return bug 2020-09-23 11:10:38 -04:00
David Randall
10ce7178c0 misc: Incorrect dependency path for dvyukov/go-fuzz
Fix the dependency path for dvyukov/go-fuzz to github.com from golang.org.

Fixes #601
2020-06-07 15:05:46 -04:00
Donald Bakong
5c6a66eaf5 lang: funcs: Add cidr_to_ip function 2020-06-03 22:01:01 -04:00
Francois Rompre-Lanctot
36d30bc985 lang: funcs: Add macfmt function 2020-06-03 17:46:39 -04:00
Adam Sigal
a5152b82e9 engine: resources: exec: Add Env
Add functionality to specify environment variables in exec.
2020-04-19 20:30:43 -06:00