From 01d4226c4ac98cb3012bff62cd85fc1bfc3fbb74 Mon Sep 17 00:00:00 2001 From: James Shubin Date: Mon, 6 May 2019 07:07:55 -0400 Subject: [PATCH] docs, readme: Improve new user experience This hopefully improves some docs for new users, and makes releases more easily available. --- Makefile | 6 +- README.md | 63 ++++++++++-- docs/development.md | 117 ++++++++++++++++++++- docs/quick-start-guide.md | 210 +++++++++++++------------------------- releases/README | 2 + 5 files changed, 245 insertions(+), 153 deletions(-) create mode 100644 releases/README diff --git a/Makefile b/Makefile index 48658ff8..a055c81c 100644 --- a/Makefile +++ b/Makefile @@ -16,7 +16,7 @@ # along with this program. If not, see . SHELL = /usr/bin/env bash -.PHONY: all art cleanart version program lang path deps run race bindata generate build build-debug crossbuild clean test gofmt yamlfmt format docs rpmbuild mkdirs rpm srpm spec tar upload upload-sources upload-srpms upload-rpms copr tag release funcgen +.PHONY: all art cleanart version program lang path deps run race bindata generate build build-debug crossbuild clean test gofmt yamlfmt format docs rpmbuild mkdirs rpm srpm spec tar upload upload-sources upload-srpms upload-rpms upload-releases copr tag release funcgen .SILENT: clean bindata # a large amount of output from this `find`, can cause `make` to be much slower! @@ -326,6 +326,10 @@ upload-rpms: rpmbuild/RPMS/ rpmbuild/RPMS/SHA256SUMS rpmbuild/RPMS/SHA256SUMS.as rsync -avz --prune-empty-dirs rpmbuild/RPMS/ $(SERVER):$(REMOTE_PATH)/RPMS/; \ fi +upload-releases: + echo Running releases/ upload... + rsync -avz --exclude '.mkdir' --exclude 'mgmt-release.url' releases/ $(SERVER):$(REMOTE_PATH)/releases/ + # # copr build # diff --git a/README.md b/README.md index ac6055a7..09248a42 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,56 @@ [![Patreon](https://img.shields.io/badge/patreon-donate-yellow.svg?style=flat-square)](https://www.patreon.com/purpleidea) [![Liberapay](https://img.shields.io/badge/liberapay-donate-yellow.svg?style=flat-square)](https://liberapay.com/purpleidea/donate) +## About: + +`Mgmt` is a real-time automation tool. It is familiar to existing configuration +management software, but is drastically more powerful as it can allow you to +build real-time, closed-loop feedback systems, in a very safe way, and with a +surprisingly small amout of our `mcl` code. For example, the following code will +ensure that your file server is set to read-only when it's friday. + +```mcl +import "datetime" +$is_friday = datetime.weekday(datetime.now()) == "friday" +file "/srv/files/" { + state => "exists", + mode => if $is_friday { # this updates the mode, the instant it changes! + "0550" + } else { + "0770" + }, +} +``` + +It can run continuously, intermittently, or on-demand, and in the first case, it +will guarantee that your system is always in the desired state for that instant! +In this mode it can run as a decentralized cluster of agents across your +network, each exchanging information with the others in real-time, to respond to +your changing needs. For example, if you want to ensure that some resource runs +on a maximum of two hosts in your cluster, you can specify that as well: + +```mcl +import "sys" +import "world" + +# we'll set a few scheduling options: +$opts = struct{strategy => "rr", max => 2, ttl => 10,} + +# schedule in a particular namespace with options: +$set = world.schedule("xsched", $opts) + +if sys.hostname() in $set { + # use your imagination to put something more complex right here... + print "i got scheduled" {} # this will run on the chosen machines +} +``` + +As you add and remove hosts from the cluster, the real-time `schedule` function +will dynamically pick up to two hosts from the available pool. These specific +functions aren't intrinsic to the core design, and new ones can be easily added. + +Please read on if you'd like to learn more... + ## Community: Come join us in the `mgmt` community! @@ -30,7 +80,7 @@ approach. The project contains an engine and a language. Mgmt is a fairly new project. It is usable today, but not yet feature complete. With your help you'll be able to influence our design and get us to 1.0 sooner! -Interested developers should read the [quick start guide](docs/quick-start-guide.md). +Interested users should read the [quick start guide](docs/quick-start-guide.md). ## Documentation: @@ -38,7 +88,7 @@ Please read, enjoy and help improve our documentation! | Documentation | Additional Notes | |---|---| -| [quick start guide](docs/quick-start-guide.md) | for mgmt developers | +| [quick start guide](docs/quick-start-guide.md) | for everyone | | [frequently asked questions](docs/faq.md) | for everyone | | [general documentation](docs/documentation.md) | for everyone | | [language guide](docs/language-guide.md) | for everyone | @@ -61,18 +111,13 @@ question, and a patch with the answer! Feel free to grab one of the straightforward [#mgmtlove](https://github.com/purpleidea/mgmt/labels/mgmtlove) issues if you're a first time contributor to the project or if you're unsure -about what to hack on! -Please see: [TODO.md](TODO.md) for a list of upcoming work and TODO items. -Please get involved by working on one of these items or by suggesting something -else! +about what to hack on! Please get involved by working on one of these items or +by suggesting something else! ## Bugs: Please set the `DEBUG` constant in [main.go](https://github.com/purpleidea/mgmt/blob/master/main.go) to `true`, and post the logs when you report the [issue](https://github.com/purpleidea/mgmt/issues). -Bonus points if you provide a [shell](https://github.com/purpleidea/mgmt/tree/master/test/shell) -or [OMV](https://github.com/purpleidea/mgmt/tree/master/test/omv) reproducible -test case. Feel free to read my article on [debugging golang programs](https://purpleidea.com/blog/2016/02/15/debugging-golang-programs/). ## Patches: diff --git a/docs/development.md b/docs/development.md index 61b052d2..696e7426 100644 --- a/docs/development.md +++ b/docs/development.md @@ -5,6 +5,118 @@ developing `mgmt`. Useful tools, conventions, etc. Be sure to read [quick start guide](quick-start-guide.md) first. +## Vagrant + +If you would like to avoid doing the above steps manually, we have prepared a +[Vagrant](https://www.vagrantup.com/) environment for your convenience. From the +project directory, run a `vagrant up`, and then a `vagrant status`. From there, +you can `vagrant ssh` into the `mgmt` machine. The `MOTD` will explain the rest. +This environment isn't commonly used by the `mgmt` developers, so it might not +be working properly. + +## Using Docker + +Alternatively, you can check out the [docker-guide](docker-guide.md) in order to +develop or deploy using docker. This method is not endorsed or supported, so use +at your own risk, as it might not be working properly. + +## Information about dependencies + +Software projects have a few different kinds of dependencies. There are _build_ +dependencies, _runtime_ dependencies, and additionally, a few extra dependencies +required for running the _test_ suite. + +### Build + +* `golang` 1.11 or higher (required, available in some distros and distributed +as a binary officially by [golang.org](https://golang.org/dl/)) + +### Runtime + +A relatively modern GNU/Linux system should be able to run `mgmt` without any +problems. Since `mgmt` runs as a single statically compiled binary, all of the +library dependencies are included. It is expected, that certain advanced +resources require host specific facilities to work. These requirements are +listed below: + +| Resource | Dependency | Version | Check version with | +|----------|-------------------|-----------------------------|-----------------------------------------------------------| +| augeas | augeas-devel | `augeas 1.6` or greater | `dnf info augeas-devel` or `apt-cache show libaugeas-dev` | +| file | inotify | `Linux 2.6.27` or greater | `uname -a` | +| hostname | systemd-hostnamed | `systemd 25` or greater | `systemctl --version` | +| nspawn | systemd-nspawn | `systemd ???` or greater | `systemctl --version` | +| pkg | packagekitd | `packagekit 1.x` or greater | `pkcon --version` | +| svc | systemd | `systemd ???` or greater | `systemctl --version` | +| virt | libvirt-devel | `libvirt 1.2.0` or greater | `dnf info libvirt-devel` or `apt-cache show libvirt-dev` | +| virt | libvirtd | `libvirt 1.2.0` or greater | `libvirtd --version` | + +For building a visual representation of the graph, `graphviz` is required. + +To build `mgmt` without augeas support please run: +`GOTAGS='noaugeas' make build` + +To build `mgmt` without libvirt support please run: +`GOTAGS='novirt' make build` + +To build `mgmt` without docker support please run: +`GOTAGS='nodocker' make build` + +To build `mgmt` without augeas, libvirt or docker support please run: +`GOTAGS='noaugeas novirt nodocker' make build` + +## OSX/macOS/Darwin development + +Developing and running `mgmt` on macOS is currently not supported (but not +discouraged either). Meaning it might work but in the case it doesn't you would +have to provide your own patches to fix problems (the project maintainer and +community are glad to assist where needed). + +There are currently some issues that make `mgmt` less suitable to run for +provisioning macOS. But as a client to provision remote servers it should run +fine. + +Since the primary supported systems are Linux and these are the environments +tested, it is wise to run these suites during macOS development as well. To ease +this, Docker can be leveraged ([Docker for Mac](https://docs.docker.com/docker-for-mac/)). + +Before running any of the commands below create the development Docker image: + +``` +docker/scripts/build-development +``` + +This image requires updating every time dependencies (`make-deps.sh`) changes. + +Then to run the test suite: + +``` +docker run --rm -ti \ + -v $PWD:/go/src/github.com/purpleidea/mgmt/ \ + -w /go/src/github.com/purpleidea/mgmt/ \ + purpleidea/mgmt:development \ + make test +``` + +For convenience this command is wrapped in `docker/scripts/exec-development`. + +Basically any command can be executed this way. Because the repository source is +mounted into the Docker container invocation will be quick and allow rapid +testing, for example: + +``` +docker/scripts/exec-development test/test-shell.sh load0.sh +``` + +Other examples: + +``` +docker/scripts/exec-development make build +docker/scripts/exec-development ./mgmt run --tmp-prefix lang examples/lang/load0.mcl +``` + +Be advised that this method is not supported and it might not be working +properly. + ## Testing This project has both unit tests in the form of golang tests and integration @@ -45,5 +157,6 @@ individual tests to run. ### IDE/Editor support -- Emacs: see `misc/emacs/` -- [Textmate](https://github.com/aequitas/mgmt.tmbundle) +* Emacs: see `misc/emacs/` +* [Textmate](https://github.com/aequitas/mgmt.tmbundle) +* [VSCode](https://github.com/aequitas/mgmt.vscode) diff --git a/docs/quick-start-guide.md b/docs/quick-start-guide.md index 38d80b8b..843cd911 100644 --- a/docs/quick-start-guide.md +++ b/docs/quick-start-guide.md @@ -2,16 +2,40 @@ ## Introduction -This guide is intended for developers. Once `mgmt` is minimally viable, we'll -publish a quick start guide for users too. If you're brand new to `mgmt`, it's -probably a good idea to start by reading the -[introductory article](https://purpleidea.com/blog/2016/01/18/next-generation-configuration-mgmt/) -or to watch an [introductory video](https://www.youtube.com/watch?v=LkEtBVLfygE&html5=1). -Once you're familiar with the general idea, please start hacking... +This guide is intended for users and developers. If you're brand new to `mgmt`, +it's probably a good idea to start by reading an +[introductory article about the engine](https://purpleidea.com/blog/2016/01/18/next-generation-configuration-mgmt/) +and an [introductory article about the language](https://purpleidea.com/blog/2018/02/05/mgmt-configuration-language/). +[There are other articles and videos available](on-the-web.md) if you'd like to +learn more or prefer different formats. Once you're familiar with the general +idea, or if you prefer a hands-on approach, please start hacking... -## Quick start +## Getting mgmt -### Installing golang +You can either build `mgmt` from source, or you can download a pre-built +release. There are also some distro repositories available, but they may not be +up to date. A pre-built release is the fastest option if there's one that's +available for your platform. If you are developing or testing a new patch to +`mgmt`, or there is not a release available for your platform, then you'll have +to build your own. + +### Downloading a pre-built release: + +The latest releases can be found [here](https://github.com/purpleidea/mgmt/releases/). +An alternate mirror is available [here](https://dl.fedoraproject.org/pub/alt/purpleidea/mgmt/releases/). + +Make sure to verify the signatures of all packages before you use them. The +signing key can be downloaded from [https://purpleidea.com/contact/#pgp-key](https://purpleidea.com/contact/#pgp-key) +to verify the release. + +If you've decided to install a pre-build release, you can skip to the +[Running mgmt](#running-mgmt) section below! + +### Building a release: + +You'll need some dependencies, including `golang`, and some associated tools. + +#### Installing golang * You need golang version 1.11 or greater installed. * To install on rpm style systems: `sudo dnf install golang` @@ -22,45 +46,64 @@ Once you're familiar with the general idea, please start hacking... * If your distro is tool old, you may need to [download](https://golang.org/dl/) a newer golang version. -### Setting up golang +#### Setting up golang -* If you do not have a GOPATH yet, create one and export it: +* You can skip this step, as your installation will default to using `~/go/`, +but if you do not have a `GOPATH` yet and want one in a custom location, create +one and export it: -``` +```shell mkdir $HOME/gopath export GOPATH=$HOME/gopath ``` * You might also want to add the GOPATH to your `~/.bashrc` or `~/.profile`. -* For more information you can read the [GOPATH documentation](https://golang.org/cmd/go/#hdr-GOPATH_environment_variable). +* For more information you can read the +[GOPATH documentation](https://golang.org/cmd/go/#hdr-GOPATH_environment_variable). -### Getting the mgmt code and dependencies +#### Getting the mgmt code and associated dependencies -* Download the `mgmt` code into the GOPATH, and switch to that directory: +* Download the `mgmt` code into the `GOPATH`, and switch to that directory: -``` -mkdir -p $GOPATH/src/github.com/purpleidea/ -cd $GOPATH/src/github.com/purpleidea/ +```shell +[ -z "$GOPATH" ] && mkdir ~/go/ || mkdir -p $GOPATH/src/github.com/purpleidea/ +cd $GOPATH/src/github.com/purpleidea/ || cd ~/go/ git clone --recursive https://github.com/purpleidea/mgmt/ -cd $GOPATH/src/github.com/purpleidea/mgmt +cd $GOPATH/src/github.com/purpleidea/mgmt/ || cd ~/go/src/github.com/purpleidea/mgmt/ ``` -* Add $GOPATH/bin to $PATH +* Add `$GOPATH/bin` to `$PATH` -``` +```shell export PATH=$PATH:$GOPATH/bin ``` * Run `make deps` to install system and golang dependencies. Take a look at -`misc/make-deps.sh` for details. -* Run `make build` to get a freshly built `mgmt` binary. +`misc/make-deps.sh` if you want to see the details of what it does. -### Running mgmt +#### Building mgmt -* Run `time ./mgmt run --tmp-prefix lang examples/lang/hello0.mcl` to try out a -very simple example! +* Now run `make` to get a freshly built `mgmt` binary. If this succeeds, you can +proceed to the [Running mgmt](#running-mgmt) section below! + +### Installing a distro release + +Installation of `mgmt` from distribution packages currently needs improvement. +They are not always up-to-date with git master and as such are not recommended. +At the moment we have: +* [COPR](https://copr.fedoraproject.org/coprs/purpleidea/mgmt/) (currently dead) +* [Arch](https://aur.archlinux.org/packages/mgmt/) (currently stale) + +Please contribute more and help improve these! We'd especially like to see a +Debian package! + +## Running mgmt + +* Run `mgmt run --tmp-prefix lang examples/lang/hello0.mcl` to try out a very +simple example! If you built it from source, you'll need to use `./mgmt` from +the project directory. * Look in that example file that you ran to see if you can figure out what it -did! +did! You can press `^C` to exit `mgmt`. * Have fun hacking on our future technology and get involved to shape the project! @@ -68,118 +111,3 @@ project! Please look in the [examples/lang/](../examples/lang/) folder for some more examples! - -## Vagrant - -If you would like to avoid doing the above steps manually, we have prepared a -[Vagrant](https://www.vagrantup.com/) environment for your convenience. From the -project directory, run a `vagrant up`, and then a `vagrant status`. From there, -you can `vagrant ssh` into the `mgmt` machine. The MOTD will explain the rest. - -## Using Docker - -Alternatively, you can check out the [docker-guide](docs/docker-guide.md) in -order to develop or deploy using docker. - -## Information about dependencies - -Software projects have a few different kinds of dependencies. There are _build_ -dependencies, _runtime_ dependencies, and additionally, a few extra dependencies -required for running the _test_ suite. - -### Build - -* `golang` 1.11 or higher (required, available in some distros and distributed -as a binary officially by [golang.org](https://golang.org/dl/)) - -### Runtime - -A relatively modern GNU/Linux system should be able to run `mgmt` without any -problems. Since `mgmt` runs as a single statically compiled binary, all of the -library dependencies are included. It is expected, that certain advanced -resources require host specific facilities to work. These requirements are -listed below: - -| Resource | Dependency | Version | Check version with | -|----------|-------------------|-----------------------------|-----------------------------------------------------------| -| augeas | augeas-devel | `augeas 1.6` or greater | `dnf info augeas-devel` or `apt-cache show libaugeas-dev` | -| file | inotify | `Linux 2.6.27` or greater | `uname -a` | -| hostname | systemd-hostnamed | `systemd 25` or greater | `systemctl --version` | -| nspawn | systemd-nspawn | `systemd ???` or greater | `systemctl --version` | -| pkg | packagekitd | `packagekit 1.x` or greater | `pkcon --version` | -| svc | systemd | `systemd ???` or greater | `systemctl --version` | -| virt | libvirt-devel | `libvirt 1.2.0` or greater | `dnf info libvirt-devel` or `apt-cache show libvirt-dev` | -| virt | libvirtd | `libvirt 1.2.0` or greater | `libvirtd --version` | - -For building a visual representation of the graph, `graphviz` is required. - -To build `mgmt` without augeas support please run: -`GOTAGS='noaugeas' make build` - -To build `mgmt` without libvirt support please run: -`GOTAGS='novirt' make build` - -To build `mgmt` without docker support please run: -`GOTAGS='nodocker' make build` - -To build `mgmt` without augeas, libvirt or docker support please run: -`GOTAGS='noaugeas novirt nodocker' make build` - -## Binary Package Installation - -Installation of `mgmt` from distribution packages currently needs improvement. -They are not always up-to-date with git master and as such are not recommended. -At the moment we have: -* [COPR](https://copr.fedoraproject.org/coprs/purpleidea/mgmt/) -* [Arch](https://aur.archlinux.org/packages/mgmt/) - -Please contribute more! We'd especially like to see a Debian package! - -## OSX/macOS/Darwin development - -Developing and running `mgmt` on macOS is currently not supported (but not -discouraged either). Meaning it might work but in the case it doesn't you would -have to provide your own patches to fix problems (the project maintainer and -community are glad to assist where needed). - -There are currently some issues that make `mgmt` less suitable to run for provisioning -macOS. But as a client to provision remote servers it should run fine. - -Since the primary supported systems are Linux and these are the environments -tested for it is wise to run these suites during macOS development as well. To -ease this Docker can be leveraged ([Docker for Mac](https://docs.docker.com/docker-for-mac/)). - -Before running any of the commands below create the development Docker image: - -``` -docker/scripts/build-development -``` - -This image requires updating every time dependencies (`make-deps.sh`) change. - -Then to run the test suite: - -``` -docker run --rm -ti \ - -v $PWD:/go/src/github.com/purpleidea/mgmt/ \ - -w /go/src/github.com/purpleidea/mgmt/ \ - purpleidea/mgmt:development \ - make test -``` - -For convenience this command is wrapped in `docker/scripts/exec-development`. - -Basically any command can be executed this way. Because the repository source is -mounted into the Docker container invocation will be quick and allow rapid -testing, example: - -``` -docker/scripts/exec-development test/test-shell.sh load0.sh -``` - -Other examples: - -``` -docker/scripts/exec-development make build -docker/scripts/exec-development ./mgmt run --tmp-prefix lang examples/lang/load0.mcl -``` diff --git a/releases/README b/releases/README new file mode 100644 index 00000000..7ed71416 --- /dev/null +++ b/releases/README @@ -0,0 +1,2 @@ +Releases from the mgmt project can be found here. Please see: +https://github.com/purpleidea/mgmt/ for more information.