Commit Graph

75 Commits

Author SHA1 Message Date
James Shubin
3ad7097c8a lang: Add internal, resource specific edges
This adds the ability to specify internal, resource specific edges, with
and without notifications. We use the special words: "Notify", "Before",
"Listen", and "Depend". They must have the first character capitalized.
They also support the "elvis" operator.
2018-02-27 23:26:25 -05:00
James Shubin
8e01b6db48 lang: Add a resource-specific elvis operator
This allows you to omit a resource parameter programmatically, and
avoids the need of an `undef` or `nil` in our language, which would
contribute to programming errors, crashes, and overall reduced safety.
2018-02-27 17:29:49 -05:00
James Shubin
df1e50e599 lang: funcs: Add math pow function and a few examples
Just a few small things I think should be committed.
2018-02-25 19:48:25 -05:00
James Shubin
6370f0cb95 lang: Add edges to lexer and parser
This adds some initial syntax for external edges to the language.

There are still improvements which are necessary for send/recv.
2018-02-25 19:29:27 -05:00
James Shubin
80784bb8f1 lang: types, funcs: Add simple polymorphic function API
This adds a simple API for adding static, polymorphic, pure functions.
This lets you define a list of type signatures and the associated
implementations to overload a particular function name. The internals of
this API then do all of the hard work of matching the available
signatures to what statically type checks, and then calling the
appropriate implementation.

While this seems as if this would only work for function polymorphism
with a finite number of possible types, while this is mostly true, it
also allows you to add the `variant` "wildcard" type into your
signatures which will allow you to match a wider set of signatures.

A canonical use case for this is the len function which can determine
the length of both lists and maps with any contained type. (Either the
type of the list elements, or the types of the map keys and values.)

When using this functionality, you must be careful to ensure that there
is only a single mapping from possible type to signature so that the
"dynamic dispatch" of the function is unique.

It is worth noting that this API won't cover functions which support an
arbitrary number of input arguments. The well-known case of this,
printf, is implemented with the more general function API which is more
complicated.

This patch also adds some necessary library improvements for comparing
types to partial types, and to types containing variants.

Lastly, this fixes a bug in the `NewType` parser which parsed certain
complex function types wrong.
2018-02-25 02:17:13 -05:00
James Shubin
40dcd6ec99 all: Misc fixes and test fixes 2018-02-25 02:13:51 -05:00
karimb
06f2d65500 docs: Add docs for docker usage 2018-02-24 12:04:23 +01:00
James Shubin
46da3a34a0 docs: Add two new faq entries 2018-02-23 19:44:54 -05:00
Johan Bloemberg
06d33a45f5 docs, misc: Add tool references, .editorconfig for mcl 2018-02-22 17:23:11 -05:00
Johan Bloemberg
9544ab2e02 recwatch: Fix watching newly created files on macOS
Fixes: https://github.com/purpleidea/mgmt/issues/33
2018-02-22 16:52:26 -05:00
James Shubin
5597183391 docs: Add two faq entries about the type system 2018-02-22 16:45:54 -05:00
James Shubin
05c60d9a59 test, docs: Restrict long lines in markdown linter
It's getting out of hand...
2018-02-22 16:19:23 -05:00
James Shubin
cffdb06181 test, docs: Add a linter for testing markdown, and fix up our docs
While writing docs, I couldn't remember what the correct style was
supposed to be, and I remember someone complaining about this
previously, so I decided to add a linter! I excluded a bunch of annoying
style rules, but if we find more we can add those to the list too.

Hopefully this gives us a more consistent feel throughout.
2018-02-21 22:52:41 -05:00
James Shubin
837388ae4e lang: types, funcs: Add simple function API
This patch adds a simple function API for writing simple, pure
functions. This should reduce the amount of boilerplate required for
most functions, and make growing a stdlib significantly easier. If you
need to build more complex, event-generating functions, or statically
polymorphic functions, then you'll still need to use the normal API for
now.

This also makes all of these pure functions available automatically
within templates. It might make sense to group these functions into
packages to make their logical organization easier, but this is a good
enough start for now.

Lastly, this added some missing pieces to our types library. You can now
use `ValueOf` to convert from a `reflect.Value` to the corresponding
`Value` in our type system, if an equivalent exists.

Unfortunately, we're severely lacking in tests for these new types
library additions, but look forward to growing some in the future!
2018-02-21 21:32:31 -05:00
James Shubin
4a623c1891 docs: Add an entry to the faq about converged timeouts 2018-02-18 16:28:50 -05:00
James Shubin
bc91b03276 docs: Add faq entry about production readiness 2018-02-18 13:34:21 -05:00
James Shubin
9ba893c06c etcd: Bump to etcd v3.3 and golang 1.9
This moves us to etcd v3.3 (a new major release) which has some useful
features but that requires version 1.9 of golang.
2018-02-15 18:47:55 -05:00
Johan Bloemberg
b34b359860 test: Streamline test suite a little
This change aims to streamline the integrationtest suite and reduce friction when running (parts of) test suites.

Changes:
- add `test-testname` to makefile to easily run one suite
- made skipping tests first class citizen in test.sh (all available testsuites and the reasons they are skipped are now better exposed and discovered)
- suppress some output of gotest unless there is an error
- no longer build binary for examples and gotest suites
- removed .SILENT from makefile as it being applied to only some targets makes it feel weird (I just learned about this option btw, feel free to comment on this change)
- move individual tests out of `test.sh` and into `test-misc.sh`
- introduced the concept of testsuites to `test.sh`
2018-02-15 17:21:49 +01:00
James Shubin
3af7e815d0 docs: Add newly recorded talks and blog posts about mgmt 2018-02-14 15:45:40 -05:00
James Shubin
1a2918082d docs: Add FAQ entry about vendoring dependencies 2018-02-14 14:01:10 -05:00
Wim
6c9e0ff974 docs: Add GOPATH/bin to PATH 2018-02-12 15:09:32 -05:00
Johan Bloemberg
ba21554c5f build, docs: Improve macOS building
- New docker command for quickly running tasks in a Linux environment.
- Updated docs with macOS specific details.
- Fixed some test issues.
- Add (fallible) macOS test target for Travis.
2018-02-10 05:01:12 -05:00
James Shubin
b19583e7d3 lang: Initial implementation of the mgmt language
This is an initial implementation of the mgmt language. It is a
declarative (immutable) functional, reactive, domain specific
programming language. It is intended to be a language that is:

* safe
* powerful
* easy to reason about

With these properties, we hope this language, and the mgmt engine will
allow you to model the real-time systems that you'd like to automate.

This also includes a number of other associated changes. Sorry for the
large size of this patch.
2018-01-20 08:09:29 -05:00
James Shubin
12fce52cd7 legal: Happy 2018 everyone...
Done with:

ack '2017+' -l | xargs sed -i -e 's/2017+/2018+/g'

Checked manually with:

git add -p

Hello to future James from 2019, and Happy Hacking!
2018-01-03 21:22:07 -05:00
James Shubin
a76dce8b15 docs: Add missing blog post about augeas resource 2017-11-26 17:15:49 -05:00
James Shubin
b01d453ae3 docs: Refresh documentation to provide a better new user experience
This does some cleanups and moves some things around for a better
experience. If you're an expert in this area, or are a new user who has
some feedback about their first impressions and experiences, please let
us know!
2017-11-25 20:45:57 -05:00
Julien Pivotto
472663193a prometheus: Initialize all metrics
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2017-11-24 11:02:36 +01:00
Julien Pivotto
23e167616f doc: Fix link to the prometheus wiki
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2017-11-23 09:52:28 +01:00
Juan-Luis de Sousa-Valadas Castaño
9f66962bfb docs: Change go required version to 1.8 2017-08-31 23:56:16 +02:00
James Shubin
b8ff6938df resources: Unify resource creation and kind setting
This removes the duplication of the kind string and cleans up things for
resource creation.
2017-06-07 03:07:02 -04:00
James Shubin
4d9d0d4548 resources: Improve AutoEdge API and pkg breakage
I previously broke the pkg auto edges because the package list wasn't
available by the time it was called. This fixes the pkg resource so that
it gets the necessary list of packages when needed. Since this means
that a possible failure could happen, we also update the AutoEdges API
to support errors. Errors can only be generated at AutoEdge struct
creation, once the struct has been returned (right before modification
of the graph structure) there is no possibility to return any errors.

It's important to remember that the AutoEdges stuff gets called because
the Init of each resource, so make sure it doesn't depend on anything
that happens there or that gets cached as a result of Init.

This is all much nicer now and has a test too :)
2017-06-02 22:15:28 -04:00
James Shubin
6e503cc79b resources: Simplify the resource Compare functions
This removes one level of indentation and simplifies the code.
2017-05-31 17:27:34 -04:00
James Shubin
a87288d519 pgraph, resources: Major refactoring continued
There was simply some technical debt I needed to kill off. Sorry for not
splitting this up into more patches.
2017-05-31 15:36:14 -04:00
James Shubin
9b9ff2622d resources: Make resource kind and baseuid fields public
This is required if we're going to have out of package resources. In
particular for third party packages, and also for if we decide to split
out each resource into a separate sub package.
2017-04-11 01:52:21 -04:00
James Shubin
64dc47d7e9 misc: Fixup documentation 2017-03-20 17:11:51 -04:00
James Shubin
22b48e296a resources, yamlgraph: Drop the kind capitalization
This stopped making sense now that we have a resource with two primary
capitals. It was just a silly formatting hack anyways. Welcome kv!
2017-03-09 02:50:55 -05:00
James Shubin
8d94be8924 resources: kv: Add new KV resource which sets key value pairs
This is a new resource for setting key value pairs in our global world
database. Currently only etcd is supported. Some of the implications and
possibilities of this resource will become more obvious with future
commits!

You can bother/test this resource with these commands:

ETCDCTL_API=3 etcdctl get "/_mgmt/strings/" --prefix=true
ETCDCTL_API=3 etcdctl put "/_mgmt/strings/KEY/HOSTNAME" 42

Replace the KEY and HOSTNAME variables with the actual values you'd like
to use. The 42 is the value that is set.
2017-03-08 19:23:33 -05:00
James Shubin
be60440b20 readme: Add new blog post about metaparameters
This also documents the recent semaphore work.
2017-03-01 16:18:14 -05:00
James Shubin
d8e19cd79a semaphore: Create a semaphore metaparam
This adds a P/V style semaphore mechanism to the resource graph. This
enables the user to specify a number of "id:count" tags associated with
each resource which will reduce the parallelism of the CheckApply
operation to that maximum count.

This is particularly interesting because (assuming I'm not mistaken) the
implementation is dead-lock free assuming that no individual resource
permanently ever blocks during execution! I don't have a formal proof of
this, but I was able to convince myself on paper that it was the case.

An actual proof that N P/V counting semaphores in a DAG won't ever
dead-lock would be particularly welcome! Hint: the trick is to acquire
them in alphabetical order while respecting the DAG flow. Disclaimer,
this assumes that the lock count is always > 0 of course.
2017-02-27 02:57:06 -05:00
Julien Pivotto
7d92ab335a prometheus: Add mgmt_pgraph_start_time_seconds metric
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2017-02-26 15:28:43 +01:00
James Shubin
fccf508dde resources, pgraph: Refactor Worker and simplify API
I'm still working on reducing the size of the monster patches that I
land, but I'm exercising the priviledge as the initial author. In any
case, this refactors worker into two, and cleans up the passing around
of the processChan. This puts common code into Init and Close.
2017-02-21 18:42:07 -05:00
James Shubin
bec7f1726f resources: virt: Allow hotplugging
This allows hot (un)plugging of CPU's! It also includes some general
cleanups which were necessary to support this as well as some other
features to the virt resource. Hotunplug requires Fedora 25.

It also comes with a mini shell script to help demo this capability.

Many thanks to pkrempa for his help with the libvirt API!
2017-02-21 18:42:07 -05:00
James Shubin
a981cfa053 legal: Oh yeah, it is 2017 2017-02-16 01:34:32 -05:00
James Shubin
55290dd1e3 docs: Add missing newline and remove extra one 2017-02-16 01:13:42 -05:00
Julien Pivotto
9c4e255994 documentation: Implement sphinx documentation
Licence removed due to to a read the docs (or sphinx/recommonmark) bug:
everything after the comment is not rendered.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2017-02-16 07:08:44 +01:00
Julien Pivotto
49baea5f6a documentation: Embed command-line syntax in a code block
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2017-02-15 15:30:32 -05:00
Julien Pivotto
be5040e7a8 prometheus: Remove mgmt_process_start_time_seconds metric
That metric is useless as by default the prometheus golang client
provides the `process_start_time_seconds` metric.

This reverts commit 25e2af7c89.
2017-02-14 22:56:12 +01:00
Julien Pivotto
25e2af7c89 prometheus: Add mgmt_process_start_time_seconds metric 2017-02-14 22:14:59 +01:00
James Shubin
0e069f1e75 docs: Update the quick start guide with git clone and vagrant
There was a bit of a catch-22, since go get wouldn't succeed without
some system build libs installed first. We also added a Vagrant
environment.
2017-02-14 12:13:43 -05:00
Mildred Ki'Lya
8c2c552164 resources: file: Implement file attributes
Add owner which must be username or uid of the file owner, group which is
the group name or gid of the file, and mode which is the octal unix file
permissions.

Add separate implementation for Go 1.6 and lower.
2017-02-14 11:55:00 -05:00