Commit Graph

1380 Commits

Author SHA1 Message Date
James Shubin
97baad4cb1 lang: funcs: Add Unify method for maplookup function
This also adds a few tests.
2021-05-23 20:03:10 -04:00
James Shubin
fbd93ecf0d lang: funcs: Add Unify method for structlookup function
This also adds a few tests.
2021-05-23 20:03:10 -04:00
James Shubin
e941ccea92 lang: funcs: Add Unify method for the simplepoly API
This is an implementation of the Unify approach for the simplepoly
function API, which wraps the full function API. It is unique in that a
lot of different functions use it, and it is easy to build functions
with it. It needs to use exclusives to represent the different options,
but at least it filters out any that aren't viable.

The Unify implementation here is fairly similar to the patterns in the
template() function.

To improve the filtering, it would be excellent if we could examine the
return type in `solved` somehow (if it is known) and use that to trim
our list of exclusives down even further! The smaller exclusives are,
the faster everything in the solver can run.
2021-05-23 20:03:10 -04:00
James Shubin
d692483bc3 lang: funcs: Add Unify method for operator function
This is an implementation of the Unify approach for the operator
function. It is unique in that it is a wrapper around the simple
operator function API.

To improve the filtering, it would be excellent if we could examine the
return type in `solved` somehow (if it is known) and use that to trim
our list of exclusives down even further! The smaller exclusives are,
the faster everything in the solver can run.
2021-05-23 20:03:10 -04:00
James Shubin
95cfbd0fff lang: funcs: Ensure that Info sig's are invalid if not built yet
In case something in the type unification tries to speculatively call
Info before it's ready to produce a valid sig, make sure we only return
a definitive answer (non-nil, and no variant types) once we've
conclusively finished defining the signature.
2021-05-23 20:03:10 -04:00
James Shubin
b3d1ed9e65 lang: funcs: core: math: Add a fortytwo function
This is mainly meant as a useful test case, but might as well have it be
fun too. As an aside, it taught me a surprising result about the %v verb
in printf, and we'll have to decide if it's an issue we care about.

https://github.com/golang/go/issues/46118

The interesting thing about this method is that it uses the simplepoly
API but has no input args-- only the output types are different. If it
had identical types in the input args, that might also have been
interesting, but it's more rare to have none. Hopefully this exercises
our type unification logic.
2021-05-12 03:30:25 -04:00
Joe Groocock
fe2b8c9fee engine: resources: exec: AutoEdge to User/Group/File
Fixes https://github.com/purpleidea/mgmt/issues/221

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-11 16:51:02 -04:00
James Shubin
2d7deef4e2 lang: unification: Don't stall the solver over generators
If we have a solution, and all that remains are generators, then feel
free to remove them and win.
2021-05-11 05:23:00 -04:00
James Shubin
b4a70b02e3 lang: funcs: Add Unify method for contains function
This is an implementation of the Unify approach for the contains
function. It is unique in that its generator invariant can recursively
generate a new generator invariant once.
2021-05-11 04:41:32 -04:00
James Shubin
c5c2364ed4 lang: funcs: core: fmt: Add an additional invariant to printf
This adds an invariant for printf that I might have missed. It may be
redundant, or it may help.
2021-05-11 03:23:33 -04:00
James Shubin
efcc4291a3 lang: funcs: core: Add Unify method for template function
This is an implementation of the Unify approach for the template
function.
2021-05-11 03:22:27 -04:00
James Shubin
6ea6ee264d lang: Add new unification rules for functions
This is meant as an incremental step into the new unification. Hopefully
it doesn't break anything and that we can rip out the old polymorphisms
work soon.
2021-05-11 02:52:35 -04:00
James Shubin
2865ba7632 lang: unification: Improve our simple solver
This removed a bug in the InvariantCall stuff, and also hopefully made
it more robust to actually solving when it had a solution.
2021-05-11 02:47:24 -04:00
James Shubin
2bed668d31 lang: interfaces: Small fixups to make unification work for now
This is all hacks until it works. Sorry that I am not a type unification
expert. If you are, please send us some patches =D
2021-05-11 01:31:10 -04:00
James Shubin
9dc24860f3 lang: interfaces: Add a new poly func interface
This new interface is subject to change and will probably be renamed if
we decide to keep it.
2021-05-11 00:45:25 -04:00
James Shubin
f01377b3bc lang: funcs: core: fmt: Add Unify method for printf
This is an implementation of the Unify approach for the printf function.
2021-05-11 00:33:50 -04:00
Joe Groocock
7443dfac4c misc: Run apt update before installing packages
Sometimes the package repo may be out of date and installing required
packages can return 404 because the version in the stale database has
been removed.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-09 19:22:40 +01:00
James Shubin
e6408e187c lang: Rename old fail5 and fail6 variables
Last of the numbered error scenario cleanups...
2021-05-08 05:19:36 -04:00
James Shubin
a02d282d3e lang: Rename old fail8 variable to failInterpolate
Eight... Interpolate?

Cleanups...
2021-05-08 04:30:17 -04:00
James Shubin
f778f53744 lang: Rename old fail9 variable to failInit
Cleanups...
2021-05-08 04:25:56 -04:00
James Shubin
95ea93564e lang: Rename old fail4 variable to failGraph
Cleanups...
2021-05-08 04:20:05 -04:00
James Shubin
d51029e86c lang: Rename old fail3 variable to failUnify
Cleanups...
2021-05-08 04:17:21 -04:00
James Shubin
1016699c94 lang: Rename old fail2 variable to failSetScope
Cleanups...
2021-05-08 04:13:33 -04:00
James Shubin
63f63955e7 lang: Replace numbered errors with named ones
This makes the tests easier to read and modify without having out of
order numbers. When writing the tests, you'll remember more easily which
section you're erroring in too!
2021-05-07 23:41:00 -04:00
James Shubin
37be9fda9f lang: Catch duplicate resource fields or meta entries statically
This teaches the compiler to catch entries with duplicate fields, and
duplicate meta entries, because it could be ambiguous to determine which
should take precedence. For example, if you specified `content` to a
file resource twice, this should error. This is known statically, so we
can catch it. If you specified two `Meta:noop` entries, this can also be
caught.

The interesting part happens when you specify one `Meta:noop` entry, and
one `Meta` entry which happens to contain a noop field in the struct.
For this, we actually have to wait until type unification is finished,
and catch the error there. This is because after type unification we
will know the precise type of the struct being passed to `Meta`, and so
we can look at its field names, even if their values aren't yet known
because the graph hasn't run yet.
2021-05-07 23:41:00 -04:00
James Shubin
0756133a7e lang: Add a named error for catching errors in test on Init
This makes it so that we can catch errors that happen in Init. We also
name the errors so that number sequence doesn't matter.
2021-05-07 23:14:49 -04:00
Joe Groocock
83c5ab318b lang: types: Clear map/list types during Into()
Map and list types are now unconditionally initialised during an Into()
call to ensure that the only data within them after the operation is
that added by the Into() function.

Prior to this change, map/list types would likely not be cleared prior
to the data being inserted into them with a few exceptions. Nil
pointers or maps/lists that were not sufficient in capacity would be
reinitialised and used to replace the existing backing data store. In
some cases this wouldn't occur meaning any residual data existing in the
container before the Into() call could persist after the data copy
completes. This behaviour is wildly inconsistent and not ideal in the
vast majority of cases. It should be assumed that the Into() call will
preserve nothing and always produce a consistent and deterministic
output.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-05 10:41:48 +01:00
Joe Groocock
0c28957016 lang: funcs: Funcs that never load are fatal
If there is a programming error in any func Stream() implementation then
the node could never output anything, causing the engine to hang
indefinitely waiting for an initial value that will never come,

Nodes keep track of whether they are loaded, so testing for this
occurence is pretty simple. Any nodes that do not return output at least
once before they close their output channel can be considered a fatal
error on which the engine will exit.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-04 11:39:34 -04:00
James Shubin
959084040d lang: Don't block the engine for empty values
If the user passed an empty list or map, we should send that and not
block. This also includes a simple test to ensure this keeps working.
2021-05-04 11:27:58 -04:00
James Shubin
8a428c6936 lang: Add a test timeout to catch blocked cases
This should catch any blocked tests and report an error. The timeout is
arbitrary.
2021-05-04 11:26:17 -04:00
Joe Groocock
48da23226c project: Add frebib to AUTHORS
Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-04 14:16:54 +01:00
James Shubin
5f0c6e5102 lang: types: Add extra ValueOf test
Just to double check weird behaviours of the golang reflect lib.
2021-05-04 06:26:33 -04:00
Joe Groocock
29f1c6f50e lang: types: Fix ValueOf() panic with nil pointer values
Some forms of reflect.Value can cause ValueOf() to panic when there is a
nil pointer somewhere within the reflect.Value, whether that be a
container type like a struct, list or map, or just a raw nil pointer.

In these cases, ValueOf() attempted to dereference the pointer without
ever checking if it was nil. mgmt lang doesn't have pointers of any
kind, so these Golang values cannot be represented in mcl types in the
current form so return a helpful error to the user.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-04 06:26:23 -04:00
James Shubin
4d187419ac engine: Small typo/cleanups in autogrouping code 2021-05-04 05:30:27 -04:00
James Shubin
58998f9cab engine: Transform the send/recv init functions into helpers
Since we'll want to use them elsewhere, we should make these helper
functions. It also makes the code look a lot neater. Unfortunately, it
adds a bit more indirection, but this isn't a critical flaw here.
2021-05-04 05:30:27 -04:00
James Shubin
cdc5ca8854 util: Add a simple log wrapper for io.Writer
This lets logger interfaces that use io.Writer get met by our logging
interface.
2021-05-04 05:27:07 -04:00
James Shubin
44e1e41266 lang: types: Improve documentation for ValueOf functions
A reminder that nil's in golang don't map to anything in mcl.
2021-05-04 05:27:07 -04:00
James Shubin
33fda8605a lang: types: Add new ValueOf tests
Hopefully this makes all of this a bit more obvious.
2021-05-04 04:27:33 -04:00
Joe Groocock
5f9ed69299 misc: Replace go-bindata with maintained fork
As per [1] go-bindata was removed from GitHub and later replaced by the
community. jteeuwen/go-bindata has since been archived to represent this
state and now most communities use kevinburke/go-bindata instead as it
is more actively maintained.

[1]: https://github.com/jteeuwen/go-bindata/issues/5

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-04 04:10:05 -04:00
Joe Groocock
7f1baea3b0 engine: resources: docker: Replace deprecated NewClient() with NewClientWithOpts()
docker/client.NewClient() is deprecated in favour of NewClientWithOpts()
which takes a series of client.Opt functions to configure the API
client. As mgmt only passes the API version through, this simplifies the
NewClient() calls.

Signed-off-by: Joe Groocock <me@frebib.net>
2021-05-02 10:39:04 +01:00
James Shubin
f75026e4b2 lang: unification: Teach the solver about new invariants
This extends our simple solver so that it can understand the new
invariants. For the Value holding invariants, it doesn't do much-- those
are expected to be used within the execution of the GeneratorInvariant
so they are passed through untouched. For the GeneratorInvariant it must
actually try running this periodically to see if it produces new
invariants that are helpful for the whole solution. Since this could get
expensive quickly, the logic is to only try running these once we've
entered steady state, but before we've tried to reach for the
ExclusiveInvariant's. The exclusives are the most expensive so we run
these last, and the generators are run late because they won't usually
produce anything helpful unless some of the basic solving has already
happened. If they could produce useful things right away, then there
wouldn't be a need for them!
2021-05-02 00:52:57 -04:00
James Shubin
ce7a1a9c67 lang: Add a CallFuncArgsValueInvariant invariant
This is a new invariant that I realized might be useful. It's not
guaranteed that it will be used or useful, but I wanted to get it out of
my WIP branch, to keep that work cleaner.
2021-05-02 00:52:57 -04:00
James Shubin
a62056fb19 lang: Add a GeneratorInvariant invariant
This is a new invariant that I realized might be useful. It's not
guaranteed that it will be used or useful, but I wanted to get it out of
my WIP branch, to keep that work cleaner.
2021-05-02 00:52:57 -04:00
James Shubin
f3434a8155 lang: Add a ValueInvariant invariant
This is a new invariant that I realized might be useful. It's not
guaranteed that it will be used or useful, but I wanted to get it out of
my WIP branch, to keep that work cleaner.
2021-05-02 00:52:57 -04:00
James Shubin
4e023ef517 lang: Move the ExprAny to the interfaces package
Having this special "placeholder" interface is useful for more than one
package.
2021-05-02 00:52:57 -04:00
James Shubin
97b80cb930 lang: unification: Move the InvariantSolution struct
We are just relocating this in the file for consistency.
2021-05-02 00:52:57 -04:00
James Shubin
525b4e6a53 lang: Move core unification structs into shared interfaces package
We should probably move these into the central interfaces package so
that these can be used from multiple places. They don't have any
dependencies, and it doesn't make sense to have the solver code mixed in
to the same package. Overall the interface being implemented here could
probably be improved, but that's a project for another day.
2021-05-02 00:52:57 -04:00
James Shubin
054eaf65b8 util: safepath: Add a new safe path helper library
This is a new path manipulation library that is designed to be safer
than using simple strings for everything. It is more work to use, but it
can help you keep track of the different path types.

It has been sitting unused in a git branch for too long, and I figured
it should see the light of day in case someone wants to start using it.
2021-05-02 00:52:57 -04:00
James Shubin
48fa796ab1 test: Disable failing test
Hit another intermittent failed test in GH CI.
2021-03-10 03:36:27 -05:00
Jean-Philippe Evrard
1873e022cc test: Add guard when no commit needs testing
Without this patch, github actions fail.

It's a temporary workaround until [1] is done.

[1]: https://github.com/purpleidea/mgmt/issues/643
2021-03-03 10:49:22 +01:00