Commit Graph

1356 Commits

Author SHA1 Message Date
James Shubin
37be9fda9f lang: Catch duplicate resource fields or meta entries statically
This teaches the compiler to catch entries with duplicate fields, and
duplicate meta entries, because it could be ambiguous to determine which
should take precedence. For example, if you specified `content` to a
file resource twice, this should error. This is known statically, so we
can catch it. If you specified two `Meta:noop` entries, this can also be
caught.

The interesting part happens when you specify one `Meta:noop` entry, and
one `Meta` entry which happens to contain a noop field in the struct.
For this, we actually have to wait until type unification is finished,
and catch the error there. This is because after type unification we
will know the precise type of the struct being passed to `Meta`, and so
we can look at its field names, even if their values aren't yet known
because the graph hasn't run yet.
2021-05-07 23:41:00 -04:00
James Shubin
0756133a7e lang: Add a named error for catching errors in test on Init
This makes it so that we can catch errors that happen in Init. We also
name the errors so that number sequence doesn't matter.
2021-05-07 23:14:49 -04:00
Joe Groocock
83c5ab318b lang: types: Clear map/list types during Into()
Map and list types are now unconditionally initialised during an Into()
call to ensure that the only data within them after the operation is
that added by the Into() function.

Prior to this change, map/list types would likely not be cleared prior
to the data being inserted into them with a few exceptions. Nil
pointers or maps/lists that were not sufficient in capacity would be
reinitialised and used to replace the existing backing data store. In
some cases this wouldn't occur meaning any residual data existing in the
container before the Into() call could persist after the data copy
completes. This behaviour is wildly inconsistent and not ideal in the
vast majority of cases. It should be assumed that the Into() call will
preserve nothing and always produce a consistent and deterministic
output.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-05 10:41:48 +01:00
Joe Groocock
0c28957016 lang: funcs: Funcs that never load are fatal
If there is a programming error in any func Stream() implementation then
the node could never output anything, causing the engine to hang
indefinitely waiting for an initial value that will never come,

Nodes keep track of whether they are loaded, so testing for this
occurence is pretty simple. Any nodes that do not return output at least
once before they close their output channel can be considered a fatal
error on which the engine will exit.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-04 11:39:34 -04:00
James Shubin
959084040d lang: Don't block the engine for empty values
If the user passed an empty list or map, we should send that and not
block. This also includes a simple test to ensure this keeps working.
2021-05-04 11:27:58 -04:00
James Shubin
8a428c6936 lang: Add a test timeout to catch blocked cases
This should catch any blocked tests and report an error. The timeout is
arbitrary.
2021-05-04 11:26:17 -04:00
Joe Groocock
48da23226c project: Add frebib to AUTHORS
Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-04 14:16:54 +01:00
James Shubin
5f0c6e5102 lang: types: Add extra ValueOf test
Just to double check weird behaviours of the golang reflect lib.
2021-05-04 06:26:33 -04:00
Joe Groocock
29f1c6f50e lang: types: Fix ValueOf() panic with nil pointer values
Some forms of reflect.Value can cause ValueOf() to panic when there is a
nil pointer somewhere within the reflect.Value, whether that be a
container type like a struct, list or map, or just a raw nil pointer.

In these cases, ValueOf() attempted to dereference the pointer without
ever checking if it was nil. mgmt lang doesn't have pointers of any
kind, so these Golang values cannot be represented in mcl types in the
current form so return a helpful error to the user.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-04 06:26:23 -04:00
James Shubin
4d187419ac engine: Small typo/cleanups in autogrouping code 2021-05-04 05:30:27 -04:00
James Shubin
58998f9cab engine: Transform the send/recv init functions into helpers
Since we'll want to use them elsewhere, we should make these helper
functions. It also makes the code look a lot neater. Unfortunately, it
adds a bit more indirection, but this isn't a critical flaw here.
2021-05-04 05:30:27 -04:00
James Shubin
cdc5ca8854 util: Add a simple log wrapper for io.Writer
This lets logger interfaces that use io.Writer get met by our logging
interface.
2021-05-04 05:27:07 -04:00
James Shubin
44e1e41266 lang: types: Improve documentation for ValueOf functions
A reminder that nil's in golang don't map to anything in mcl.
2021-05-04 05:27:07 -04:00
James Shubin
33fda8605a lang: types: Add new ValueOf tests
Hopefully this makes all of this a bit more obvious.
2021-05-04 04:27:33 -04:00
Joe Groocock
5f9ed69299 misc: Replace go-bindata with maintained fork
As per [1] go-bindata was removed from GitHub and later replaced by the
community. jteeuwen/go-bindata has since been archived to represent this
state and now most communities use kevinburke/go-bindata instead as it
is more actively maintained.

[1]: https://github.com/jteeuwen/go-bindata/issues/5

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-04 04:10:05 -04:00
Joe Groocock
7f1baea3b0 engine: resources: docker: Replace deprecated NewClient() with NewClientWithOpts()
docker/client.NewClient() is deprecated in favour of NewClientWithOpts()
which takes a series of client.Opt functions to configure the API
client. As mgmt only passes the API version through, this simplifies the
NewClient() calls.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-02 10:39:04 +01:00
James Shubin
f75026e4b2 lang: unification: Teach the solver about new invariants
This extends our simple solver so that it can understand the new
invariants. For the Value holding invariants, it doesn't do much-- those
are expected to be used within the execution of the GeneratorInvariant
so they are passed through untouched. For the GeneratorInvariant it must
actually try running this periodically to see if it produces new
invariants that are helpful for the whole solution. Since this could get
expensive quickly, the logic is to only try running these once we've
entered steady state, but before we've tried to reach for the
ExclusiveInvariant's. The exclusives are the most expensive so we run
these last, and the generators are run late because they won't usually
produce anything helpful unless some of the basic solving has already
happened. If they could produce useful things right away, then there
wouldn't be a need for them!
2021-05-02 00:52:57 -04:00
James Shubin
ce7a1a9c67 lang: Add a CallFuncArgsValueInvariant invariant
This is a new invariant that I realized might be useful. It's not
guaranteed that it will be used or useful, but I wanted to get it out of
my WIP branch, to keep that work cleaner.
2021-05-02 00:52:57 -04:00
James Shubin
a62056fb19 lang: Add a GeneratorInvariant invariant
This is a new invariant that I realized might be useful. It's not
guaranteed that it will be used or useful, but I wanted to get it out of
my WIP branch, to keep that work cleaner.
2021-05-02 00:52:57 -04:00
James Shubin
f3434a8155 lang: Add a ValueInvariant invariant
This is a new invariant that I realized might be useful. It's not
guaranteed that it will be used or useful, but I wanted to get it out of
my WIP branch, to keep that work cleaner.
2021-05-02 00:52:57 -04:00
James Shubin
4e023ef517 lang: Move the ExprAny to the interfaces package
Having this special "placeholder" interface is useful for more than one
package.
2021-05-02 00:52:57 -04:00
James Shubin
97b80cb930 lang: unification: Move the InvariantSolution struct
We are just relocating this in the file for consistency.
2021-05-02 00:52:57 -04:00
James Shubin
525b4e6a53 lang: Move core unification structs into shared interfaces package
We should probably move these into the central interfaces package so
that these can be used from multiple places. They don't have any
dependencies, and it doesn't make sense to have the solver code mixed in
to the same package. Overall the interface being implemented here could
probably be improved, but that's a project for another day.
2021-05-02 00:52:57 -04:00
James Shubin
054eaf65b8 util: safepath: Add a new safe path helper library
This is a new path manipulation library that is designed to be safer
than using simple strings for everything. It is more work to use, but it
can help you keep track of the different path types.

It has been sitting unused in a git branch for too long, and I figured
it should see the light of day in case someone wants to start using it.
2021-05-02 00:52:57 -04:00
James Shubin
48fa796ab1 test: Disable failing test
Hit another intermittent failed test in GH CI.
2021-03-10 03:36:27 -05:00
Jean-Philippe Evrard
1873e022cc test: Add guard when no commit needs testing
Without this patch, github actions fail.

It's a temporary workaround until [1] is done.

[1]: https://github.com/purpleidea/mgmt/issues/643
2021-03-03 10:49:22 +01:00
James Shubin
35a8062b58 test: Disable travis IRC notifications for now
One of our contributors is unusually annoyed by them, and it's important
to keep your contributors happy!
2021-03-03 04:28:53 -05:00
Jean-Philippe Evrard
636248ad67 test: Ensure branches are also testable
test-commit-message runs on PR, but also on push in other branches
which aren't PRs. We need to test those too.

This is fixed by ensuring the same kind of behaviour than travis CI:
When a patch is put on a branch, it's using the branchname for
testing [1].

[1]:
https://docs-staging.travis-ci.com/user/environment-variables/#default-environment-variables
2021-03-03 09:52:49 +01:00
Jean-Philippe Evrard
4511c54fad test: Ensure github CI tests commit messages
Without this patch, the travis var is empty, and we just pass.
This is a problem, as we are using github CI nowadays.

This should fix it.
2021-03-03 02:47:30 -05:00
James Shubin
7f3970541b test: Skip more tests
I think some of these fail due to shared environments and noisy
neighbours in github. We'll have to fix that eventually or test
elsewhere.
2021-03-02 13:41:00 -05:00
James Shubin
4040f4d151 test: Skip yet another intermittent test
We shall not have intermittent tests!
2021-03-02 12:48:44 -05:00
James Shubin
887d374c53 lang: funcs: Catch simple function api usage without types
In case a programmer makes a mistake and passes in a function using the
simple function API without a type or even without the entire value,
we'll now return a sensible error message and panic in init() instead of
requiring a test to catch this alone.
2021-02-28 22:50:51 -05:00
James Shubin
be4b87155d test: Skip another intermittent test
I think this might be related to multiple jobs running at the same time
on the same host. Not sure though.
2021-02-24 04:00:34 -05:00
James Shubin
b987a7da4c examples: tftp: Fix missing error checking in example 2021-02-20 13:31:45 -05:00
James Shubin
7153fe5ad2 test: Skip intermittent tests
It would be great to fix some rare races or debug what's wrong in CI,
but for now let's get rid of these fails so that we can get better data
for when we break something more serious. We'll need to revisit all of
this for sure.
2021-02-19 21:17:57 -05:00
James Shubin
ccd8ba44d9 test: Exclude generated ragel parser from golint 2021-02-17 22:01:34 -05:00
James Shubin
e7ef0f7a6c test: Set default column size if $TERM env var isn't set
Seems Github actions breaks or unsets this, leading to the errors:

tput: No value for $TERM and no -T specified
seq: missing operand
Try 'seq --help' for more information.

Hopefully this makes things a bit more robust.
2021-02-17 04:07:30 -05:00
James Shubin
400b58c0e9 lang: Improve string interpolation
The original string interpolation was based on hil which didn't allow
proper escaping, since they used a different escape pattern. Secondly,
the golang Unquote function didn't deal with the variable substitution,
which meant it had to be performed in a second step.

Most importantly, because we did this partial job in Unquote (the fact
that is strips the leading and trailing quotes tricked me into thinking
I was done with interpolation!) it was impossible to remedy the
remaining parts in a second pass with hil. Both operations needs to be
done in a single step. This is logical when you aren't tunnel visioned.

This patch replaces both of these so that string interpolation works
properly. This removes the ability to allow inline function calls in a
string, however this was an incidental feature, and it's not clear that
having it is a good idea. It also requires you wrap the var name with
curly braces. (They are not optional.)

This comes with a load of tests, but I think I got some of it wrong,
since I'm quite new at ragel. If you find something, please say so =D In
any case, this is much better than the original hil implementation, and
easy for a new contributor to patch to make the necessary fixes.
2021-02-17 03:35:12 -05:00
James Shubin
5257496214 test: Make a few cosmetic changes and enable race testing 2021-02-17 02:45:47 -05:00
Jean-Philippe Evrard
e1bfe4a3ce test: Add GitHub Actions test support
Authored-By: Jean-Philippe Evrard <open-source@a.spamming.party>
Co-Authored-By: James Shubin <james@shubin.ca>
Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-14 21:41:02 -05:00
James Shubin
f31cce8ec2 test: misc: Clarify golang wording 2021-02-14 21:39:01 -05:00
Joe Groocock
169ebfa72c test: make-deps: Add folds around tests and dep blocks
Improves readability of CI test output and hides away the complexity
when in most cases it is not required. Retain fold behaviour for both
Travis and GitHub Actions in case both are used in any capacity.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-12 16:44:12 +00:00
Joe Groocock
7cace52ab5 test: prevent LinuxBrew in GitHub Actions CI
Ubuntu-latest in GitHub Actions provides linuxbrew, so the tests install
both the native Debian dependency packages, and also the linuxbrew
variants which is slower and entirely redundant.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-12 16:44:12 +00:00
Joe Groocock
95b93c60d9 test: Invert negative bash assertions
In bash `-n` is `non zero length` which is the opposite of `-z` meaning
`zero length`. `-n` is semantically identical to `! -z` but `-n`.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-12 16:44:12 +00:00
Joe Groocock
5af1dcb8b1 test: Add in_ci utility test function
in_ci checks for environment variables set by a selection of CI systems
and returns true if the test appears to be running in CI. Additionally
it can test for specific CI systems, and returns true if the CI system
is listed.

Deduplicate existing environment checks for Travis and Jenkins.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-12 16:44:11 +00:00
Joe Groocock
6a61774fb7 docker: Bump docker dependencies, add containerd
These dependencies are maintained because the upstream repos bundle
vendor directories into the repos, which cause namespacing issues during
build. Git submodules don't strip the vendor directory whereas most
vendoring tools would.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-09 21:14:34 +00:00
James Shubin
ccbaca24f1 gopath: Remove this unused directory
I had this symlink hack a long time ago. I don't think it's being used
anymore.
2021-02-07 21:23:49 -05:00
Joe Groocock
07b6048dc5 etcd: Bump etcd + friends to the latest upstream version
This allows dropping the pinned grpc-prometheus and grpc-gateway
libraries as git master works fine for now.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-07 12:55:02 +00:00
Joe Groocock
60dd34d066 make: Drop support for Go 1.9 in make build
docs/development.md says the minimum required Golang version is 1.13 at
the time of writing.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-02-07 00:08:51 -05:00
James Shubin
28451d1e14 lang: funcs: Fixup race in vumeter example
The vumeter example was written quickly and without much care. This
fixes a possible race (panic) and also removes the busy loop that wastes
CPU while we're waiting for the first value to come in.
2021-02-06 23:59:06 -05:00