Commit Graph

1674 Commits

Author SHA1 Message Date
James Shubin
c4a9560d53 engine: local: Fix benign race
Use the mutex in a safer manner to eliminate the benign data race.
2024-01-05 15:50:43 -05:00
James Shubin
bc63b7608e engine: graph: Use an atomic bool instead of a mutex
The isStateOK variable can be accessed concurrently as these are
supposed to be "benign" races. As such, they need to be labelled as such
so that we don't hit some undefined compiler behaviour.

Here are five good references relating to "benign" data races in golang.

1) https://web.archive.org/web/20181022150257/https://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong

2) https://go.dev/ref/mem - "Informal Overview" section.

3) https://docs.oracle.com/cd/E19205-01/820-0619/gecqt/index.html

4) https://www.usenix.org/legacy/event/hotpar11/tech/final_files/Boehm.pdf

5) https://go.dev/doc/articles/race_detector

TL;DR: wrap your benign races with sync/atomic or eliminate them.
2024-01-05 15:50:42 -05:00
James Shubin
c2f508e261 engine: graph: Use an rwmutex around read/write of vertex state
This fixes two small races we had while simultaneously reading from and
writing to the vertex timestamp, and simultaneously writing to the
vertex isStateOK (dirty) flag.

They were actually "safe" races, in that it doesn't matter if the
read/write race got the old or new value, or that the double write
happened. The time sequencing was correct (I believe) in both cases, but
this triggers the race detector now that we have tests for it.
2024-01-02 18:18:34 -05:00
James Shubin
a07dc0a511 lang: interpolate, funcs: Add a concat function for faster interpolation
This adds a concat function which can be used directly by string
interpolation to avoid having to constantly unify the plus operator
which is much slower at this time.

The new monomorphisms changes caused type unification of a notable
example to go from ~25s to ~5m30s which was obviously not bearable. With
this fix, things are now down to ~6s.

This is an important optimization, but it's also a good reminder that
type unification of polymorphic functions needs to be improved in
general too.
2024-01-02 15:23:40 -05:00
James Shubin
d8db320722 lang: gapi: Prettier printing of type unification timing
This makes things a little nicer to visualize.
2024-01-02 13:26:51 -05:00
James Shubin
24054f905f lib, lang, docs: Add --only-unify option
This adds a new run flag for the lang frontend to exit immediately
following type unification. This makes it easier to use this as a step
in CI, and also to type the execution for performance comparison
reasons.
2024-01-02 13:06:19 -05:00
James Shubin
32ca815896 lib, docs: Small cleanups
Formatting and removing of old docs.
2024-01-02 13:04:49 -05:00
James Shubin
fa5949e191 lang: funcs: core: test: Make a new instance for each test
Since this special one_instance function uses global state, if it's
re-used in more than one test, this won't work since they still all use
the whole global state. Make new ones for each test.

This also breaks the count=2 feature (any number other than 1) when
running these, which is not ideal. Create a cleanup API that we can run
between tests to reset the global state.
2023-12-27 18:25:53 -05:00
Samuel Gélineau
1c0a98a0cc lang: ast: ExprBind is now monomorphic
This adds ExprTopLevel and ExprSingleton and ensures that ExprBind is
now monomorphic.

This corrects a previous design bug where it was not monomorphic and
would thus cause spawning of many more copies than necessary. In most
cases this was only harmful to memory and performance, and not
behaviour, since these functions were pure, and we didn't have a test
for this.

This also adds a bunch more tests. Most notably, the graph shape tests
generally produce smaller graphs now.

Lastly, a lambda cannot have two different types when used at two
different call sites. It is rare that this would be used, and when it
would make sense, there are easy workarounds to accomplish equivalent
goals.

This was mostly authored by Sam, James helped with some cleanup and
debugging.

Co-authored-by: James Shubin <james@shubin.ca>
2023-12-27 16:33:38 -05:00
James Shubin
9d208e8795 lang: Test more iter funcs without polymorphic id function
This tests the lambdas in more ways so that we are sure non-polymorphic
$id functions work the way we want.
2023-12-27 16:33:38 -05:00
James Shubin
72fe0cd6db lang: Polymorphic lambda functions aren't allowed for now
We don't allow a lambda variable to be polymorphic anymore. Of course
this isn't bad, but it makes things too difficult, at least for now.
It's also likely that we don't need this specific feature very often I
think.
2023-12-27 16:33:38 -05:00
James Shubin
734590b6bd lang: Add a test for duplicate functions called
We should not call either of these functions more than once for their
values. If we do, it means we have made a mistake with a compiler
optimization.

This is important, because otherwise if you had code like:

$x = random_password()

Then this would obviously be a problem. Thankfully, the situations where
functions generate unique data is rare, but it's probably something we
should take care of.
2023-12-27 16:33:38 -05:00
James Shubin
7cc231e8b9 lang: unification, interfaces: Add a skip invariant
This is a cleaner way of telling the type unifier that we don't want
this particular expression in the solution set at the end.
2023-12-27 16:33:38 -05:00
James Shubin
c2bf4ef7d4 lang: Test a top-level var explicitly
This makes sure the top-level scope is really seen.
2023-12-27 16:33:38 -05:00
James Shubin
439179e37f lang: Remove duplicate test
It's contents were identical to the stmtfunc-recursive-double.txtar
test.
2023-12-27 16:33:38 -05:00
James Shubin
c333cb542c lang: ast: Expr Param and Poly should not have values
Sometimes a recursive call through ExprVar's Value method would hit one
of these and return (nil, nil) which would throw off things.
2023-12-17 21:39:06 -05:00
James Shubin
6a6546db8d lang: ast: Save the non-polyfunc args
We probably want to save these args too in case.
2023-12-16 14:26:29 -05:00
James Shubin
a6d22a5a4b misc: Add editorconfig info for txtar tests 2023-12-12 00:24:41 -05:00
James Shubin
da5f94e62c test: Add a simple mcl formatting test
This should definitely get expanded overtime when we can have something
decent for formatting mcl code. For now, catch the easy things.
2023-12-12 00:13:27 -05:00
James Shubin
1d886f2995 test: Improve bash fmt test
This catches slightly more issues.
2023-12-12 00:13:27 -05:00
James Shubin
a6b6aa570e test: Fix small test warnings
These were issues we should be catching.
2023-12-12 00:01:29 -05:00
James Shubin
32b26a09c2 test: Don't error on unbound variable
We would get this error instead of a test error:

	test/util.sh: line 32: GITHUB_ACTION: unbound variable

This would happen when run locally.
2023-12-12 00:00:19 -05:00
James Shubin
8fcaa4abf2 lang: Output of tests should also show autogrouped resources
This makes our test cases slightly more complete.
2023-12-08 18:25:54 -05:00
James Shubin
18e1f08156 engine: graph: Allow send/recv to work with autogrouped resources
We've previously not received a value from within an autogrouped
resource. It turns out this would be quite useful, and so this patch
implements the additional plumbing and testing so that this works!

Testing that an autogrouped resource can still send values has not been
done at this time.
2023-12-08 18:18:17 -05:00
James Shubin
bf5cc63bc5 misc: Move to golang 1.20
Make sure to quote your 1.20 otherwise it shows up as 1.2 which is very
old!
2023-12-03 21:40:48 -05:00
James Shubin
271a94e0c7 engine: resources: Improve logging on resource errors
When a subcommand errors, it's helpful to see what it was.
2023-12-03 21:21:55 -05:00
James Shubin
c3f34db81e test: Remove new grep warning
With newer versions of grep, you now see:

	grep: warning: stray \ before "

This is because we had an erroneous backslash escaping a quote. Remove
it.
2023-12-03 18:24:20 -05:00
James Shubin
b7d8a769db lang: funcs: core: Add value functions
This adds a new series of "get*" functions which can read values from
the associated "value" resources. The key name of the function must
match the name value of the resource for things to work.

Type unification isn't yet perfect in these scenarios, so you should use
casually and with caution.
2023-12-03 18:24:20 -05:00
James Shubin
c05af3b9b3 lib, lang, engine: local: Add Value mechanisms to local API
This expands the Local API with the first (and in theory, only ever) API
for reading and writing simple values. This is a coordination point for
resources and functions to share things directly.
2023-12-03 18:24:20 -05:00
James Shubin
3e37a60635 lang: funcs: core: test: Fix small formatting issue 2023-12-03 18:24:20 -05:00
James Shubin
9d47b6843f engine, gapi, lang, lib: Plumb through new local API
This is a new API that is similar in spirit and plumbing to the World
API, but it intended for all local machine operations and will likely
only ever have one implementation.
2023-12-03 18:24:20 -05:00
James Shubin
12ffac1f06 lang: Add placeholders for future prefix passing 2023-12-03 18:02:00 -05:00
James Shubin
7991b4ab25 engine, lang, lib: Re-order for consistency
Some small cleanups to make room for future patches.
2023-12-03 18:02:00 -05:00
James Shubin
984fb702e5 engine: resources: Make test resource easier for debugging
Let some fields not get printed if we request it.
2023-12-03 18:02:00 -05:00
James Shubin
019b88cedf util: Add value to B64 serialization functions
These small utility functions are useful.
2023-12-03 16:52:01 -05:00
Samuel Gélineau
55932cf3c7 lang: ast: Clear env on ExprCall.Graph()
Fix the same bug in ExprCall.Graph().
2023-12-02 01:19:31 -05:00
Samuel Gélineau
1ff2c9bbd9 lang: ast: Clear env in ExprVar.Graph()
Fix the bug in ExprVar.Graph() described in the previous commit.
2023-12-02 01:19:31 -05:00
Samuel Gélineau
72235b0fd4 lang: ast: New test "clear-env-on-var"
This test detects a mistake which is easy to make: when making a
recursive call to the target of an ExprVar, it would be easy to
accidentally pass the environment, like we usually do with every other
recursive call. For variables, this is a mistake, because the lambda
parameters which are in scope where the variable is used must not be in
scope where the variable is defined.

In fact, ExprVar.Graph() currently makes this mistake. The test passes
anyway, because an earlier phase (SetScope) correctly clears the
environment and detects the problem before the Graph phase. Thus, this
test does not guarantee that all the phases correctly clear their
environment, it merely detects the unlikely case in which all the phases
make the same mistake.
2023-12-02 01:19:21 -05:00
James Shubin
b8d391024a engine: resources: Add a DoneCmd feature for exec resource
This makes a common pattern of `$cmd && echo done > /tmp/donefile` much
clearer and easier.
2023-11-28 15:24:22 -05:00
James Shubin
598bec0eab engine: resources: Clean up var naming and misc in exec 2023-11-28 15:24:11 -05:00
James Shubin
709bf7b246 engine: resources: Add missing WatchCmd UID
Not sure if this is even used anywhere, but might as well add it for
now.
2023-11-28 15:24:11 -05:00
James Shubin
8251c8f259 lang: funcs: core: Add a second panic signature
This modifies the panic feature to accept a boolean or a string. If true
or not empty, then it will cause the panic. This makes some of the error
code a little less ugly.
2023-11-28 14:43:39 -05:00
James Shubin
9c0bde0b29 lang: funcs, parser: Add improved panic magic
This is a newer implementation of the panic magic. I kept the old commit
in for posterity and to show the difference. The two versions are
identical to the end-user with one exception: the newer version doesn't
include a useless panic resource in the graph when there is no panic. In
this version, the panic function returns false and the if statement it's
the condition of, doesn't produce the resource within. On error, we
still consume the function in the if expression, and doing so causes
everything to shutdown.

The other benefit is that the implementation is much cleaner and doesn't
need the interpolate hack.
2023-11-28 14:40:21 -05:00
James Shubin
2cbce963b7 engine: resources, lang: funcs, parser: Add panic magic
It's valuable to check your runtime values and to shut down the entire
engine in case something doesn't match. This patch adds some magic
plumbing to support a "panic" mechanism.

A new "panic" statement gets transparently converted into a panic
function and panic resource. The former errors if the input is not
empty. The latter must be present to consume the value, but doesn't
actually do anything.
2023-11-28 13:49:31 -05:00
James Shubin
64e6e686e0 lang: funcs: core: math: Add minus1 function
This is a useful template hack until we get something better.
2023-11-28 11:28:49 -05:00
James Shubin
dad3458ece mod: Try and update things
No idea why things break. go.mod sucks.
2023-11-27 21:00:03 -05:00
James Shubin
e727e7adb6 lang: funcs: core: net: Add more mac fmt functions 2023-11-27 21:00:03 -05:00
James Shubin
af1c952700 lang: funcs: funcgen: Allow []string as inputs
This allows us to have the strings.Join function generated.
2023-11-27 21:00:03 -05:00
James Shubin
ce2f7112a3 lang: funcs: funcgen: Don't nest unnecessarily
Clean up the code.
2023-11-27 19:57:54 -05:00
James Shubin
4650ed01eb lang: funcs: funcgen: Refactor slightly for list input
This code had some design issues. We'll rewrite this eventually, but for
now, let's allow list inputs. Refactor this to make it possible.
2023-11-27 19:57:54 -05:00