This mega patch primarily introduces a new function engine. The main
reasons for this new engine are:
1) Massively improved performance with lock-contended graphs.
Certain large function graphs could have very high lock-contention which
turned out to be much slower than I would have liked. This new algorithm
happens to be basically lock-free, so that's another helpful
improvement.
2) Glitch-free function graphs.
The function graphs could "glitch" (an FRP term) which could be
undesirable in theory. In practice this was never really an issue, and
I've not explicitly guaranteed that the new graphs are provably
glitch-free, but in practice things are a lot more consistent.
3) Simpler graph shape.
The new graphs don't require the private channels. This makes
understanding the graphs a lot easier.
4) Branched graphs only run half.
Previously we would run two pure side of an if statement, and while this
was mostly meant as an early experiment, it stayed in for far too long
and now was the right time to remove this. This also means our graphs
are much smaller and more efficient too.
Note that this changed the function API slightly. Everything has been
ported. It's possible that we introduce a new API in the future, but it
is unexpected to cause removal of the two current APIs.
In addition, we finally split out the "schedule" aspect from
world.schedule(). The "pick me" aspects now happen in a separate
resource, rather than as a yucky side-effect in the function. This also
lets us more precisely choose when we're scheduled, and we can observe
without being chosen too.
As usual many thanks to Sam for helping through some of the algorithmic
graph shape issues!
This removes the secret channel from the call function. Having it made
it more complicated to write new function engines, and it's not clear
why it was even needed in the first place. It seems that even the
current generation of function engines work just fine without it.
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
Resources that can be grouped into the http:server resource must have
that prefix. Grouping is basically hierarchical, and without that common
prefix, it means we'd have to special-case our grouping algorithm.
I think this is what I want in most scenarios, is there a reason to do
otherwise? This is because we may wish to export incomplete resources,
where the remaining necessary fields for validation happens on collect.
Many years ago I built and demoed a prototype of a simple web ui with a
slider, and as you moved it left and right, it started up or shutdown
some number of virtual machines.
The webui was standalone code, but the rough idea of having events from
a high-level overview flow into mgmt, was what I wanted to test out. At
this stage, I didn't even have the language built yet. This prototype
helped convince me of the way a web ui would fit into everything.
Years later, I build an autogrouping prototype which looks quite similar
to what we have today. I recently picked it back up to polish it a bit
more. It's certainly not perfect, and might even be buggy, but it's
useful enough that it's worth sharing.
If I had more cycles, I'd probably consider removing the "store" mode,
and replace it with the normal "value" system, but we would need the
resource "mutate" API if we wanted this. This would allow us to directly
change the "value" field, without triggering a graph swap, which would
be a lot less clunky than the "store" situation.
Of course I'd love to see a GTK version of this concept, but I figured
it would be more practical to have a web ui over HTTP.
One notable missing feature, is that if the "web ui" changes (rather
than just a value changing) we need to offer to the user to reload it.
It currently doesn't get an event for that, and so don't confuse your
users. We also need to be better at validating "untrusted" input here.
There's also no major reason to use the "gin" framework, we should
probably redo this with the standard library alone, but it was easier
for me to push out something quick this way. We can optimize that later.
Lastly, this is all quite ugly since I'm not a very good web dev, so if
you want to make this polished, please do! The wasm code is also quite
terrible due to limitations in the compiler, and maybe one day when that
works better and doesn't constantly deadlock, we can improve it.
Most of the time, we don't need to have a dynamic call sub graph, since
the actual function call could be represented statically as it
originally was before lambda functions were implemented. Simplifying the
graph shape has important performance benefits in terms of both keep the
graph smaller (memory, etc) and in avoiding the need to run transactions
at runtime (speed) to reshape the graph.
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
This commit implements a range function that mimicks python's range
built-in by having a start, stop, and range argument. There's also
a few examples and tests to mimick Python's examples to guarantee
we're consistent with their behaviour.
/ Test (basic) on ubuntu-latest with golang 1.23 (push) Has been cancelled
/ Test (race) on ubuntu-latest with golang 1.23 (push) Has been cancelled
/ Test (shell) on ubuntu-latest with golang 1.23 (push) Has been cancelled
This lets us look at the available resource data for collection, and to
filter it so we can decide what we want to collect on our machine.
Other types of collect functions could be added in the future.
I've been waiting to write this patch for a long time. I firmly believe
that the idea of "exported resources" was truly a brilliant one, but
which was never even properly understood by its original inventors! This
patch set aims to show how it should have been done.
The main differences are:
* Real-time modelling, since "once per run" makes no sense.
* Filter with code/functions not language syntax.
* Directed exporting to limit the intended recipients.
The next step is to add more "World" reading and filtering functions to
make it easy and expressive to make your selection of resources to
collect!
There are some rare situations with completely symmetrical graphs which
mean that there isn't a "more correct" error. This is due to the
annoying map iteration non-determinism, and so instead of fighting to
remove every bit of that, let's just accept more than one error here.
This adds a forkv statement which is used to iterate over a map with a
body of statements. This is an important data transformation tool which
should be used sparingly, but is important to have.
An import statement inside of a forkv loop is not currently supported.
We have a simple hack to detect the obvious cases, but more deeply
nested scenarios probably won't be caught, and you'll get an obscure
error message if you try to do this.
This was incredibly challenging to get right, and it's all thanks to Sam
for his brilliance.
Note, I couldn't think of a better keyword that "forkv" but suggestions
are welcome if you think you have a better idea. Other ideas were formap
and foreach, but neither got me very excited.
This adds a for statement which is used to iterate over a list with a
body of statements. This is an important data transformation tool which
should be used sparingly, but is important to have.
An import statement inside of a for loop is not currently supported. We
have a simple hack to detect the obvious cases, but more deeply nested
scenarios probably won't be caught, and you'll get an obscure error
message if you try to do this.
This was incredibly challenging to get right, and it's all thanks to Sam
for his brilliance.
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
This adds an initial implementation of printing line numbers on type
unification errors. It also attempts to print a visual position
indicator for most scenarios.
This patch was started by Felix Frank and finished by James Shubin.
Co-authored-by: Felix Frank <Felix.Frank.de@gmail.com>
This is a first attempt to add a new function for URL parsing, using
go's net/url package and the simple API. This is still a barebones
implementation, there's possibility to expose more information. It also
includes simple tests.
It would be a likely mistake to create a self-referential frag, and mgmt
would spin forever updating the file... We probably don't want this, so
let's just catch this case in Validate.
Of course you could get around this with multiple files, and a fancier
search could statically check the graph, but the goal isn't to prevent
any bad code, since that's not likely to be possible.
We forgot to reject this corner case which could lead to a runtime error
since the expected type from the incoming struct would not match what
we're handling.
If more than one star import is present in the same scope, allow it. If
one star import could overwrite something, ordering is not guaranteed.
We allow this for now, but we might create a compiler fix to stop it.
This adds a test to notice both of these behaviours.
Even if they have no side effect, they aren't legal, since it would be
surprising if suddenly something new got added in one which then broke
the imports.
If you had ambiguous code, and specified an invalid type, this could
sneak through and become a runtime error, instead of a compile-time
error. We fix this and add a test.
This adds a modern type unification algorithm, which drastically
improves performance, particularly for bigger programs.
This required a change to the AST to add TypeCheck methods (for Stmt)
and Infer/Check methods (for Expr). This also changed how the functions
express their invariants, and as a result this was changed as well.
This greatly improves the way we express these invariants, and as a
result it makes adding new polymorphic functions significantly easier.
This also makes error output for the user a lot better in pretty much
all scenarios.
The one downside of this patch is that a good chunk of it is merged in
this giant single commit since it was hard to do it step-wise. That's
not the end of the world.
This couldn't be done without the guidance of Sam who helped me in
explaining, debugging, and writing all the sneaky algorithmic parts and
much more. Thanks again Sam!
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
This removes the exclusive from the res names and edge names. We now
require that the names should be lists of strings, however they can
still be single strings if that can be determined statically.
Programmers should explicitly wrap their variables in a string by
interpolation to force this, or in square brackets to force a list. The
former is generally preferable because it generates a small function
graph since it doesn't need to build a list.
If we had a single list wrapped in an interpolated string, it could
sneak through type unification, which is not correct. Wrapping a
variable by interpolation in a string, must force it to be a string.
A regression in 4b0cdf9123 caused the
basic send/recv functionality to break for simple scenarios. This was
due to inadequate testing, and a partial misunderstanding of the
situation.
New testing should hopefully catch more cases, but send/recv and
compile-time checks are still not as complete as is probably possible.
This improves the autogrouping algorithm to support hierarchical
autogrouping. It's not guaranteed to work if we replace the reachability
grouper with something more efficient, but it's good enough for now.
Some of our special tests can only be run once per `go test` invocation.
That is, using the test -count flag will cause a guaranteed failure
since we depend on a global being initialized only once as part of that
test.
This adds a per-test config option so that a user can specify to never
run a particular test more than once. This lets us continue to use the
-count flag with the test suite, without it causing some tests to fail.
The Ordering and DAG detection code is challenging because we need
Ordering to do SetScope, but Ordering itself needs to know about scopes.
This improved variant should hopefully catch all the scenarios of
identically named variables causing invalid loops.
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
We are planning to implement an optimization in which some function
calls are compiled to a single static graph rather than to a CallFunc
which dynamically creates a sub-graph at runtime. These test cases
exercise corner cases for which it would be theoretically possible to
use a static graph, but which we might not be able to optimize.
This implements a new type of syntactic sugar for the common pattern of
a base class which returns a child class, and so on. Instead of needing
to repeatedly indent the child classes, we can instead prefix them at
the definition site (where created with the class keyword) with the name
of the parent class, followed by a colon, to get the desired embedded
sugar.
For example, instead of writing:
class base() {
class inner() {
class deepest() {
}
}
}
You can instead write:
class base() {
}
class base:inner() {
}
class base:inner:deepest() {
}
Of course, you can only access any of the inner classes by first
including (with the include keyword) a parent class, and then
subsequently including the inner one.
This adds support for `include as <identifier>` type statements which in
addition to pulling in any defined resources, it also makes the contents
of the scope of the class available to the scope of the include
statement, but prefixed by the identifier specified.
This makes passing data between scopes much more powerful, and it also
allows classes to return useful classes for subsequent use.
This also improves the SetScope procedure and adds to the Ordering
stage. It's unclear if the current Ordering stage can handle all code,
or if there exist corner-cases which are valid code, but which would
produce a wrong or imprecise topological sort.
Some extraneous scoping bugs still exist, which expose certain variables
that we should not depend on in future code.
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
