I've been waiting to write this patch for a long time. I firmly believe
that the idea of "exported resources" was truly a brilliant one, but
which was never even properly understood by its original inventors! This
patch set aims to show how it should have been done.
The main differences are:
* Real-time modelling, since "once per run" makes no sense.
* Filter with code/functions not language syntax.
* Directed exporting to limit the intended recipients.
The next step is to add more "World" reading and filtering functions to
make it easy and expressive to make your selection of resources to
collect!
This adds a for statement which is used to iterate over a list with a
body of statements. This is an important data transformation tool which
should be used sparingly, but is important to have.
An import statement inside of a for loop is not currently supported. We
have a simple hack to detect the obvious cases, but more deeply nested
scenarios probably won't be caught, and you'll get an obscure error
message if you try to do this.
This was incredibly challenging to get right, and it's all thanks to Sam
for his brilliance.
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
This adds an initial implementation of printing line numbers on type
unification errors. It also attempts to print a visual position
indicator for most scenarios.
This patch was started by Felix Frank and finished by James Shubin.
Co-authored-by: Felix Frank <Felix.Frank.de@gmail.com>
Instead of constantly making these updates, let's just remove the year
since things are stored in git anyways, and this is not an actual modern
legal risk anymore.
This adds a modern type unification algorithm, which drastically
improves performance, particularly for bigger programs.
This required a change to the AST to add TypeCheck methods (for Stmt)
and Infer/Check methods (for Expr). This also changed how the functions
express their invariants, and as a result this was changed as well.
This greatly improves the way we express these invariants, and as a
result it makes adding new polymorphic functions significantly easier.
This also makes error output for the user a lot better in pretty much
all scenarios.
The one downside of this patch is that a good chunk of it is merged in
this giant single commit since it was hard to do it step-wise. That's
not the end of the world.
This couldn't be done without the guidance of Sam who helped me in
explaining, debugging, and writing all the sneaky algorithmic parts and
much more. Thanks again Sam!
Co-authored-by: Samuel Gélineau <gelisam@gmail.com>
This adds a unification optimizations API, and uses it to optimize the
embedded provisioner. With these turned on, type unification drops from
around 1m45s to 2.5s which is a 40x speedup.
This is a bit of a hack until we improve the GAPI a bit, but will let us
shut down type unification a bit faster if we want to interrupt a long
running operation. Hopefully our future algorthmic performance
improvements will obliviate the need for this to be a common issue.
With the recent merging of embedded package imports and the entry CLI
package, it is now possible for users to build in mcl code into a single
binary. This additional permission makes it explicitly clear that this
is permitted to make it easier for those users. The condition is phrased
so that the terms can be "patched" by the original author if it's
necessary for the project. For example, if the name of the language
(mcl) changes, has a differently named new version, someone finds a
phrasing improvement or a legal loophole, or for some other
reasonable circumstance. Now go write some beautiful embedded tools!
This simplifies the API by passing through the filesystem so that
function signatures don't need to be as complicated, and furthermore use
that consistently throughout.
If we don't startup fast enough, print some debugging information. We
should eventually change this to print the list of functions that aren't
started yet, and also to give each function entry a better String method
so that we have a better idea of what everything is.
This expands the Local API with the first (and in theory, only ever) API
for reading and writing simple values. This is a coordination point for
resources and functions to share things directly.
This is a new API that is similar in spirit and plumbing to the World
API, but it intended for all local machine operations and will likely
only ever have one implementation.
This lets us add a resource that has an implementation with a field
whose type is determined at compile time. This let's us write more
flexible resources.
What's missing is additional type checking so that we guarantee that a
specific resource doesn't change types during run-time.
This removes the calling of SetValue from the engine, and instead
replaces it with the Table() API. The downside is that this is likely
slower, and the current API with locking being exposed publicly is kind
of ugly. The upside is that this might make building the new engine
easier.
Future versions might remove locking from the API if we can avoid making
any accesses to expressions. Currently this happens within Logf/SafeLogf
which is our main (only?) usage at the moment. Logging could become
smarter perhaps. Alternatively, we might pass in a "setter" function
that gets called safely from within the engine. This could wrap SetValue
and the locking functions wouldn't be part of the public API.
There were a bunch of packages that weren't well documented. With the
recent split up of the lang package, I figured it would be more helpful
for new contributors who want to learn the structure of the project.
This is a giant refactor to split the giant lang package into many
subpackages. The most difficult piece was figuring out how to extract
the extra ast structs into their own package, because they needed to
call two functions which also needed to import the ast.
The solution was to separate out those functions into their own
packages, and to pass them into the ast at the root when they're needed,
and to let the relevant ast portions call a handle.
This isn't terribly ugly because we already had a giant data struct
woven through the ast.
The bad part is rebasing any WIP work on top of this.
Since we focus on safety, it would be nice to reduce the chance of any
runtime errors if we made a typo for a resource parameter. With this
patch, each resource can export constants into the global namespace so
that typos would cause a compile error.
Of course in the future if we had a more advanced type system, then we
could support precise types for each individual resource param, but in
an attempt to keep things simple, we'll leave that for another day. It
would add complexity too if we ever wanted to store a parameter
externally.
Lastly, we might consider adding "special case" parsing so that directly
specified fields would parse intelligently. For example, we could allow:
file "/tmp/hello" {
state => exists, # magic sugar!
}
This isn't supported for now, but if it works after all the other parser
changes have been made, it might be something to consider.
This should give us options as to how a function should interact with an
FS. I feel like it's cleaner to go through the World API, and passing in
the FsURI lets us do that, but I passed in the Fs at the same time in
case it's useful for some reason. I think using it is a boundary
violation, but it's just a hunch. Does anything break when we move from
one deploy to the next?
This adds a giant missing piece of the language: proper function values!
It is lovely to now understand why early programming language designers
didn't implement these, but a joy to now reap the benefits of them. In
adding these, many other changes had to be made to get them to "fit"
correctly. This improved the code and fixed a number of bugs.
Unfortunately this touched many areas of the code, and since I was
learning how to do all of this for the first time, I've squashed most of
my work into a single commit. Some more information:
* This adds over 70 new tests to verify the new functionality.
* Functions, global variables, and classes can all be implemented
natively in mcl and built into core packages.
* A new compiler step called "Ordering" was added. It is called by the
SetScope step, and determines statement ordering and shadowing
precedence formally. It helped remove at least one bug and provided the
additional analysis required to properly capture variables when
implementing function generators and closures.
* The type unification code was improved to handle the new cases.
* Light copying of Node's allowed our function graphs to be more optimal
and share common vertices and edges. For example, if two different
closures capture a variable $x, they'll both use the same copy when
running the function, since the compiler can prove if they're identical.
* Some areas still need improvements, but this is ready for mainstream
testing and use!
The simple type unification algorithm suffered from some serious
performance and memory problems when used with certain code bases. This
adds some crucial optimizations that improve performance drastically.
This enables imports in mcl code, and is one of last remaining blockers
to using mgmt. Now we can start writing standalone modules, and adding
standard library functions as needed. There's still lots to do, but this
was a big missing piece. It was much harder to get right than I had
expected, but I think it's solid!
This unfortunately large commit is the result of some wild hacking I've
been doing for the past little while. It's the result of a rebase that
broke many "wip" commits that tracked my private progress, into
something that's not gratuitously messy for our git logs. Since this was
a learning and discovery process for me, I've "erased" the confusing git
history that wouldn't have helped. I'm happy to discuss the dead-ends,
and a small portion of that code was even left in for possible future
use.
This patch includes:
* A change to the cli interface:
You now specify the front-end explicitly, instead of leaving it up to
the front-end to decide when to "activate". For example, instead of:
mgmt run --lang code.mcl
we now do:
mgmt run lang --lang code.mcl
We might rename the --lang flag in the future to avoid the awkward word
repetition. Suggestions welcome, but I'm considering "input". One
side-effect of this change, is that flags which are "engine" specific
now must be specified with "run" before the front-end name. Eg:
mgmt run --tmp-prefix lang --lang code.mcl
instead of putting --tmp-prefix at the end. We also changed the GAPI
slightly, but I've patched all code that used it. This also makes things
consistent with the "deploy" command.
* The deploys are more robust and let you deploy after a run
This has been vastly improved and let's mgmt really run as a smart
engine that can handle different workloads. If you don't want to deploy
when you've started with `run` or if one comes in, you can use the
--no-watch-deploy option to block new deploys.
* The import statement exists and works!
We now have a working `import` statement. Read the docs, and try it out.
I think it's quite elegant how it fits in with `SetScope`. Have a look.
As a result, we now have some built-in functions available in modules.
This also adds the metadata.yaml entry-point for all modules. Have a
look at the examples or the tests. The bulk of the patch is to support
this.
* Improved lang input parsing code:
I re-wrote the parsing that determined what ran when we passed different
things to --lang. Deciding between running an mcl file or raw code is
now handled in a more intelligent, and re-usable way. See the inputs.go
file if you want to have a look. One casualty is that you can't stream
code from stdin *directly* to the front-end, it's encapsulated into a
deploy first. You can still use stdin though! I doubt anyone will notice
this change.
* The scope was extended to include functions and classes:
Go forth and import lovely code. All these exist in scopes now, and can
be re-used!
* Function calls actually use the scope now. Glad I got this sorted out.
* There is import cycle detection for modules!
Yes, this is another dag. I think that's #4. I guess they're useful.
* A ton of tests and new test infra was added!
This should make it much easier to add new tests that run mcl code. Have
a look at TestAstFunc1 to see how to add more of these.
As usual, I'll try to keep these commits smaller in the future!
