Allow failures in 1.4.x because of etcd problem

The error seen is:

req.Cancel undefined
(type *http.Request has no field or method Cancel)

And is possibly the result of something etcd folks changed. Sadly, the
fast building 1.4.x won't work at the moment :(

.gitignore

@@ -1,3 +1,9 @@
+.omv/
+.ssh/
+.vagrant/
 mgmt-documentation.pdf
 old/
 tmp/
+*_stringer.go
+mgmt
+rpmbuild/

.travis.yml

@@ -0,0 +1,27 @@
+language: go
+go:
+  - 1.4.3
+  - 1.5.3
+  - 1.6
+  - tip
+sudo: false
+before_install: 'git fetch --unshallow'
+install: 'make deps'
+script: 'make test'
+matrix:
+  allow_failures:
+    - go: tip
+    - go: 1.4.3
+notifications:
+  irc:
+    channels:
+      - "irc.freenode.net#mgmtconfig"
+    template:
+      - "%{repository} (%{commit}: %{author}): %{message}"
+      - "More info : %{build_url}"
+    on_success: always
+    on_failure: always
+    use_notice: false
+    skip_join: false
+  email:
+    - travis-ci@shubin.ca

COPYRIGHT

@@ -1,5 +1,5 @@
 Mgmt
-Copyright (C) 2013-2015+ James Shubin and the project contributors
+Copyright (C) 2013-2016+ James Shubin and the project contributors
 Written by James Shubin <james@shubin.ca> and the project contributors
 
 This program is free software: you can redistribute it and/or modify

DOCUMENTATION.md

@@ -2,7 +2,7 @@
 
 <!--
 Mgmt
-Copyright (C) 2013-2015+ James Shubin and the project contributors
+Copyright (C) 2013-2016+ James Shubin and the project contributors
 Written by James Shubin <james@shubin.ca> and the project contributors
 
 This program is free software: you can redistribute it and/or modify
@@ -30,13 +30,16 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 1. [Overview](#overview)
 2. [Project description - What the project does](#project-description)
 3. [Setup - Getting started with mgmt](#setup)
-4. [Usage/FAQ - Notes on usage and frequently asked questions](#usage-and-frequently-asked-questions)
-5. [Reference - Detailed reference](#reference)
-	* [graph.yaml](#graph.yaml)
+4. [Features - All things mgmt can do](#features)
+	* [Autoedges - Automatic resource relationships](#autoedges)
+	* [Autogrouping - Automatic resource grouping](#autogrouping)
+5. [Usage/FAQ - Notes on usage and frequently asked questions](#usage-and-frequently-asked-questions)
+6. [Reference - Detailed reference](#reference)
+	* [Graph definition file](#graph-definition-file)
 	* [Command line](#command-line)
-6. [Examples - Example configurations](#examples)
-7. [Development - Background on module development and reporting bugs](#development)
-8. [Authors - Authors and contact information](#authors)
+7. [Examples - Example configurations](#examples)
+8. [Development - Background on module development and reporting bugs](#development)
+9. [Authors - Authors and contact information](#authors)
 
 ##Overview
 
@@ -49,6 +52,13 @@ The mgmt tool is a distributed, event driven, config management tool, that
 supports parallel execution, and librarification to be used as the management
 foundation in and for, new and existing software.
 
+For more information, you may like to read some blog posts from the author:
+
+* [Next generation config mgmt](https://ttboj.wordpress.com/2016/01/18/next-generation-configuration-mgmt/)
+* [Automatic edges in mgmt](https://ttboj.wordpress.com/2016/03/14/automatic-edges-in-mgmt/)
+
+There is also an [introductory video](https://www.youtube.com/watch?v=GVhpPF0j-iE&html5=1) available.
+
 ##Setup
 
 During this prototype phase, the tool can be run out of the source directory.
@@ -57,6 +67,51 @@ get started. Beware that this _can_ cause data loss. Understand what you're
 doing first, or perform these actions in a virtual environment such as the one
 provided by [Oh-My-Vagrant](https://github.com/purpleidea/oh-my-vagrant).
 
+##Features
+
+This section details the numerous features of mgmt and some caveats you might
+need to be aware of.
+
+###Autoedges
+
+Automatic edges, or AutoEdges, is the mechanism in mgmt by which it will
+automatically create dependencies for you between resources. For example,
+since mgmt can discover which files are installed by a package it will
+automatically ensure that any file resource you declare that matches a
+file installed by your package resource will only be processed after the
+package is installed.
+
+####Controlling autodeges
+
+Though autoedges is likely to be very helpful and avoid you having to declare
+all dependencies explicitly, there are cases where this behaviour is
+undesirable.
+
+Some distributions allow package installations to automatically start the
+service they ship. This can be problematic in the case of packages like MySQL
+as there are configuration options that need to be set before MySQL is ever
+started for the first time (or you'll need to wipe the data directory). In
+order to handle this situation you can disable autoedges per resource and
+explicitly declare that you want `my.cnf` to be written to disk before the
+installation of the `mysql-server` package.
+
+You can disable autoedges for a resource by setting the `autoedge` key on
+the meta attributes of that resource to `false`.
+
+###Autogrouping
+
+Automatic grouping or AutoGroup is the mechanism in mgmt by which it will
+automatically group multiple resource vertices into a single one. This is
+particularly useful for grouping multiple package resources into a single
+resource, since the multiple installations can happen together in a single
+transaction, which saves a lot of time because package resources typically have
+a large fixed cost to running (downloading and verifying the package repo) and
+if they are grouped they share this fixed cost. This grouping feature can be
+used for other use cases too.
+
+You can disable autogrouping for a resource by setting the `autogroup` key on
+the meta attributes of that resource to `false`.
+
 ##Usage and frequently asked questions
 (Send your questions as a patch to this FAQ! I'll review it, merge it, and
 respond by commit with the answer.)
@@ -67,6 +122,14 @@ I wanted a next generation config management solution that didn't have all of
 the design flaws or limitations that the current generation of tools do, and no
 tool existed!
 
+###Why did you use etcd? What about consul?
+
+Etcd and consul are both written in golang, which made them the top two
+contenders for my prototype. Ultimately a choice had to be made, and etcd was
+chosen, but it was also somewhat arbitrary. If there is available interest,
+good reasoning, *and* patches, then we would consider either switching or
+supporting both, but this is not a high priority at this time.
+
 ###You didn't answer my question, or I have a question!
 
 It's best to ask on [IRC](https://webchat.freenode.net/?channels=#mgmtconfig)
@@ -84,11 +147,11 @@ information on these options, please view the source at:
 If you feel that a well used option needs documenting here, please patch it!
 
 ###Overview of reference
-* [graph.yaml](#graph.yaml): Main graph definition file.
+* [Graph definition file](#graph-definition-file): Main graph definition file.
 * [Command line](#command-line): Command line parameters.
 
-###graph.yaml
-This is the compiled graph definition file. The format is currently
+###Graph definition file
+graph.yaml is the compiled graph definition file. The format is currently
 undocumented, but by looking through the [examples/](https://github.com/purpleidea/mgmt/tree/master/examples)
 you can probably figure out most of it, as it's fairly intuitive.
 
@@ -99,6 +162,13 @@ documentation, please run `mgmt --help`.
 ####`--file <graph.yaml>`
 Point to a graph file to run.
 
+####`--converged-timeout <seconds>`
+Exit if the machine has converged for approximately this many seconds.
+
+####`--max-runtime <seconds>`
+Exit when the agent has run for approximately this many seconds. This is not
+generally recommended, but may be useful for users who know what they're doing.
+
 ##Examples
 For example configurations, please consult the [examples/](https://github.com/purpleidea/mgmt/tree/master/examples) directory in the git
 source repository. It is available from:
@@ -117,7 +187,7 @@ To report any bugs, please file a ticket at: [https://github.com/purpleidea/mgmt
 
 ##Authors
 
-Copyright (C) 2013-2015+ James Shubin and the project contributors
+Copyright (C) 2013-2016+ James Shubin and the project contributors
 
 Please see the
 [AUTHORS](https://github.com/purpleidea/mgmt/tree/master/AUTHORS) file

Makefile

@@ -1,9 +1,42 @@
+# Mgmt
+# Copyright (C) 2013-2016+ James Shubin and the project contributors
+# Written by James Shubin <james@shubin.ca> and the project contributors
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 SHELL = /bin/bash
-.PHONY: all version run race build clean test format docs
+.PHONY: all version program path deps run race build clean test format docs rpmbuild mkdirs rpm srpm spec tar upload upload-sources upload-srpms upload-rpms copr
 .SILENT: clean
 
-VERSION := $(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --dirty)
-PROGRAM := $(notdir $(CURDIR))
+SVERSION := $(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --dirty --always)
+VERSION := $(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --abbrev=0)
+PROGRAM := $(shell echo $(notdir $(CURDIR)) | cut -f1 -d"-")
+OLDGOLANG := $(shell go version | grep -E 'go1.3|go1.4')
+ifeq ($(VERSION),$(SVERSION))
+	RELEASE = 1
+else
+	RELEASE = untagged
+endif
+ARCH = $(shell arch)
+SPEC = rpmbuild/SPECS/$(PROGRAM).spec
+SOURCE = rpmbuild/SOURCES/$(PROGRAM)-$(VERSION).tar.bz2
+SRPM = rpmbuild/SRPMS/$(PROGRAM)-$(VERSION)-$(RELEASE).src.rpm
+SRPM_BASE = $(PROGRAM)-$(VERSION)-$(RELEASE).src.rpm
+RPM = rpmbuild/RPMS/$(PROGRAM)-$(VERSION)-$(RELEASE).$(ARCH).rpm
+USERNAME := $(shell cat ~/.config/copr 2>/dev/null | grep username | awk -F '=' '{print $$2}' | tr -d ' ')
+SERVER = 'dl.fedoraproject.org'
+REMOTE_PATH = 'pub/alt/$(USERNAME)/$(PROGRAM)'
 
 all: docs
 
@@ -11,35 +44,179 @@ all: docs
 version:
 	@echo $(VERSION)
 
+program:
+	@echo $(PROGRAM)
+
+path:
+	./misc/make-path.sh
+
+deps:
+	./misc/make-deps.sh
+
 run:
-	find -maxdepth 1 -type f -name '*.go' -not -name '*_test.go' | xargs go run -ldflags "-X main.version $(VERSION) -X main.program $(PROGRAM)"
+	find -maxdepth 1 -type f -name '*.go' -not -name '*_test.go' | xargs go run -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION)"
 
-# include race test
+# include race flag
 race:
-	find -maxdepth 1 -type f -name '*.go' -not -name '*_test.go' | xargs go run -race -ldflags "-X main.version $(VERSION) -X main.program $(PROGRAM)"
+	find -maxdepth 1 -type f -name '*.go' -not -name '*_test.go' | xargs go run -race -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION)"
 
-build: mgmt
+build: $(PROGRAM)
 
-mgmt: main.go
-	go build -ldflags "-X main.version $(VERSION) -X main.program $(PROGRAM)"
+$(PROGRAM): main.go
+	@echo "Building: $(PROGRAM), version: $(SVERSION)..."
+	go generate
+ifneq ($(OLDGOLANG),)
+	@# avoid equals sign in old golang versions eg in: -X foo=bar
+	go build -ldflags "-X main.program $(PROGRAM) -X main.version $(SVERSION)" -o $(PROGRAM);
+else
+	go build -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION)" -o $(PROGRAM);
+endif
 
 clean:
-	[ ! -e mgmt ] || rm mgmt
+	[ ! -e $(PROGRAM) ] || rm $(PROGRAM)
+	rm -f *_stringer.go	# generated by `go generate`
 
 test:
 	./test.sh
-	./test/test-gofmt.sh
-	./test/test-yamlfmt.sh
-	go test
-	#go test ./pgraph
-	go test -race
-	#go test -race ./pgraph
 
 format:
 	find -type f -name '*.go' -not -path './old/*' -not -path './tmp/*' -exec gofmt -w {} \;
-	find -type f -name '*.yaml' -not -path './old/*' -not -path './tmp/*' -exec ruby -e "require 'yaml'; x=YAML.load_file('{}').to_yaml; File.open('{}', 'w').write x" \;
+	find -type f -name '*.yaml' -not -path './old/*' -not -path './tmp/*' -not -path './omv.yaml' -exec ruby -e "require 'yaml'; x=YAML.load_file('{}').to_yaml.each_line.map(&:rstrip).join(10.chr)+10.chr; File.open('{}', 'w').write x" \;
 
-docs: mgmt-documentation.pdf
+docs: $(PROGRAM)-documentation.pdf
 
-mgmt-documentation.pdf: DOCUMENTATION.md
-	pandoc DOCUMENTATION.md -o 'mgmt-documentation.pdf'
+$(PROGRAM)-documentation.pdf: DOCUMENTATION.md
+	pandoc DOCUMENTATION.md -o '$(PROGRAM)-documentation.pdf'
+
+#
+#	build aliases
+#
+# TODO: does making an rpm depend on making a .srpm first ?
+rpm: $(SRPM) $(RPM)
+	# do nothing
+
+srpm: $(SRPM)
+	# do nothing
+
+spec: $(SPEC)
+	# do nothing
+
+tar: $(SOURCE)
+	# do nothing
+
+rpmbuild/SOURCES/: tar
+rpmbuild/SRPMS/: srpm
+rpmbuild/RPMS/: rpm
+
+upload: upload-sources upload-srpms upload-rpms
+	# do nothing
+
+#
+#	rpmbuild
+#
+$(RPM): $(SPEC) $(SOURCE)
+	@echo Running rpmbuild -bb...
+	rpmbuild --define '_topdir $(shell pwd)/rpmbuild' -bb $(SPEC) && \
+	mv rpmbuild/RPMS/$(ARCH)/$(PROGRAM)-$(VERSION)-$(RELEASE).*.rpm $(RPM)
+
+$(SRPM): $(SPEC) $(SOURCE)
+	@echo Running rpmbuild -bs...
+	rpmbuild --define '_topdir $(shell pwd)/rpmbuild' -bs $(SPEC)
+	# renaming is not needed because we aren't using the dist variable
+	#mv rpmbuild/SRPMS/$(PROGRAM)-$(VERSION)-$(RELEASE).*.src.rpm $(SRPM)
+
+#
+#	spec
+#
+$(SPEC): rpmbuild/ spec.in
+	@echo Running templater...
+	#cat spec.in > $(SPEC)
+	sed -e s/__PROGRAM__/$(PROGRAM)/ -e s/__VERSION__/$(VERSION)/ -e s/__RELEASE__/$(RELEASE)/ < spec.in > $(SPEC)
+	# append a changelog to the .spec file
+	git log --format="* %cd %aN <%aE>%n- (%h) %s%d%n" --date=local | sed -r 's/[0-9]+:[0-9]+:[0-9]+ //' >> $(SPEC)
+
+#
+#	archive
+#
+$(SOURCE): rpmbuild/
+	@echo Running git archive...
+	# use HEAD if tag doesn't exist yet, so that development is easier...
+	git archive --prefix=$(PROGRAM)-$(VERSION)/ -o $(SOURCE) $(VERSION) 2> /dev/null || (echo 'Warning: $(VERSION) does not exist. Using HEAD instead.' && git archive --prefix=$(PROGRAM)-$(VERSION)/ -o $(SOURCE) HEAD)
+	# TODO: if git archive had a --submodules flag this would easier!
+	@echo Running git archive submodules...
+	# i thought i would need --ignore-zeros, but it doesn't seem necessary!
+	p=`pwd` && (echo .; git submodule foreach) | while read entering path; do \
+		temp="$${path%\'}"; \
+		temp="$${temp#\'}"; \
+		path=$$temp; \
+		[ "$$path" = "" ] && continue; \
+		(cd $$path && git archive --prefix=$(PROGRAM)-$(VERSION)/$$path/ HEAD > $$p/rpmbuild/tmp.tar && tar --concatenate --file=$$p/$(SOURCE) $$p/rpmbuild/tmp.tar && rm $$p/rpmbuild/tmp.tar); \
+	done
+
+# TODO: ensure that each sub directory exists
+rpmbuild/:
+	mkdir -p rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
+
+mkdirs:
+	mkdir -p rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
+
+#
+#	sha256sum
+#
+rpmbuild/SOURCES/SHA256SUMS: rpmbuild/SOURCES/ $(SOURCE)
+	@echo Running SOURCES sha256sum...
+	cd rpmbuild/SOURCES/ && sha256sum *.tar.bz2 > SHA256SUMS; cd -
+
+rpmbuild/SRPMS/SHA256SUMS: rpmbuild/SRPMS/ $(SRPM)
+	@echo Running SRPMS sha256sum...
+	cd rpmbuild/SRPMS/ && sha256sum *src.rpm > SHA256SUMS; cd -
+
+rpmbuild/RPMS/SHA256SUMS: rpmbuild/RPMS/ $(RPM)
+	@echo Running RPMS sha256sum...
+	cd rpmbuild/RPMS/ && sha256sum *.rpm > SHA256SUMS; cd -
+
+#
+#	gpg
+#
+rpmbuild/SOURCES/SHA256SUMS.asc: rpmbuild/SOURCES/SHA256SUMS
+	@echo Running SOURCES gpg...
+	# the --yes forces an overwrite of the SHA256SUMS.asc if necessary
+	gpg2 --yes --clearsign rpmbuild/SOURCES/SHA256SUMS
+
+rpmbuild/SRPMS/SHA256SUMS.asc: rpmbuild/SRPMS/SHA256SUMS
+	@echo Running SRPMS gpg...
+	gpg2 --yes --clearsign rpmbuild/SRPMS/SHA256SUMS
+
+rpmbuild/RPMS/SHA256SUMS.asc: rpmbuild/RPMS/SHA256SUMS
+	@echo Running RPMS gpg...
+	gpg2 --yes --clearsign rpmbuild/RPMS/SHA256SUMS
+
+#
+#	upload
+#
+# upload to public server
+upload-sources: rpmbuild/SOURCES/ rpmbuild/SOURCES/SHA256SUMS rpmbuild/SOURCES/SHA256SUMS.asc
+	if [ "`cat rpmbuild/SOURCES/SHA256SUMS`" != "`ssh $(SERVER) 'cd $(REMOTE_PATH)/SOURCES/ && cat SHA256SUMS'`" ]; then \
+		echo Running SOURCES upload...; \
+		rsync -avz rpmbuild/SOURCES/ $(SERVER):$(REMOTE_PATH)/SOURCES/; \
+	fi
+
+upload-srpms: rpmbuild/SRPMS/ rpmbuild/SRPMS/SHA256SUMS rpmbuild/SRPMS/SHA256SUMS.asc
+	if [ "`cat rpmbuild/SRPMS/SHA256SUMS`" != "`ssh $(SERVER) 'cd $(REMOTE_PATH)/SRPMS/ && cat SHA256SUMS'`" ]; then \
+		echo Running SRPMS upload...; \
+		rsync -avz rpmbuild/SRPMS/ $(SERVER):$(REMOTE_PATH)/SRPMS/; \
+	fi
+
+upload-rpms: rpmbuild/RPMS/ rpmbuild/RPMS/SHA256SUMS rpmbuild/RPMS/SHA256SUMS.asc
+	if [ "`cat rpmbuild/RPMS/SHA256SUMS`" != "`ssh $(SERVER) 'cd $(REMOTE_PATH)/RPMS/ && cat SHA256SUMS'`" ]; then \
+		echo Running RPMS upload...; \
+		rsync -avz --prune-empty-dirs rpmbuild/RPMS/ $(SERVER):$(REMOTE_PATH)/RPMS/; \
+	fi
+
+#
+#	copr build
+#
+copr: upload-srpms
+	./misc/copr-build.py https://$(SERVER)/$(REMOTE_PATH)/SRPMS/$(SRPM_BASE)
+
+# vim: ts=8

README.md

@@ -1,35 +1,73 @@
 # *mgmt*: This is: mgmt!
 
-[![Build Status](https://secure.travis-ci.org/purpleidea/mgmt.png)](http://travis-ci.org/purpleidea/mgmt)
+[![Build Status](https://secure.travis-ci.org/purpleidea/mgmt.png?branch=master)](http://travis-ci.org/purpleidea/mgmt)
+[![Documentation](https://img.shields.io/docs/markdown.png)](DOCUMENTATION.md)
+[![IRC](https://img.shields.io/irc/%23mgmtconfig.png)](https://webchat.freenode.net/?channels=#mgmtconfig)
+[![Jenkins](https://img.shields.io/jenkins/status.png)](https://ci.centos.org/job/purpleidea-mgmt/)
+[![COPR](https://img.shields.io/copr/builds.png)](https://copr.fedoraproject.org/coprs/purpleidea/mgmt/)
+
+## Community:
+Come join us on IRC in [#mgmtconfig](https://webchat.freenode.net/?channels=#mgmtconfig) on Freenode!
+You may like the [#mgmtconfig](https://twitter.com/hashtag/mgmtconfig) hashtag if you're on [Twitter](https://twitter.com/#!/purpleidea).
+
+## Questions:
+Please join the [#mgmtconfig](https://webchat.freenode.net/?channels=#mgmtconfig) IRC community!
+If you have a well phrased question that might benefit others, consider asking it by sending a patch to the documentation [FAQ](https://github.com/purpleidea/mgmt/blob/master/DOCUMENTATION.md#usage-and-frequently-asked-questions) section. I'll merge your question, and a patch with the answer!
+
+## Quick start:
+* Either get the golang dependencies on your own, or run `make deps` if you're comfortable with how we install them.
+* Run `make build` to get a freshly built `mgmt` binary.
+* Run `cd $(mktemp --tmpdir -d tmp.XXX) && etcd` to get etcd running. The `mgmt` software will do this automatically for you in the future.
+* Run `time ./mgmt run --file examples/graph0.yaml --converged-timeout=1` to try out a very simple example!
+* To run continuously in the default mode of operation, omit the `--converged-timeout` option.
+* Have fun hacking on our future technology!
+
+## Examples:
+Please look in the [examples/](examples/) folder for more examples!
 
 ## Documentation:
 Please see: [DOCUMENTATION.md](DOCUMENTATION.md) or [PDF](https://pdfdoc-purpleidea.rhcloud.com/pdf/https://github.com/purpleidea/mgmt/blob/master/DOCUMENTATION.md).
 
-## Questions:
-Come join us in [#mgmtconfig](https://webchat.freenode.net/?channels=#mgmtconfig) on Freenode!
+## Roadmap:
+Please see: [TODO.md](TODO.md) for a list of upcoming work and TODO items.
+Please get involved by working on one of these items or by suggesting something else!
+Feel free to grab one of the straightforward [#mgmtlove](https://github.com/purpleidea/mgmt/labels/mgmtlove) issues if you're a first time contributor to the project or if you're unsure about what to hack on!
 
-## Examples:
-Please look in the [examples/](examples/) folder for usage. If none exist, please contribute one!
-
-## Notes:
-* This is currently a research project into next generation config management technologies!
-* This is my first complex project in golang, please notify me of any issues.
-* I have some well thought out designs for the future of this project, which I'll try and write up clearly and publish as soon as possible.
-* Please don't expect stable interfaces, code, or any data safety.
-* This design is the result of ideas I've had from hacking on advanced config management projects.
-* I first started hacking on this in ~2013, even though I had very little time for it.
-* I couldn't think of a good name for the project, so it's now being called `mgmt` until someone contributes a better one!
-* I've published a number of articles about this tool:
-  * TODO
-* There are some screencasts available:
-  * TODO
+## Bugs:
+Please set the `DEBUG` constant in [main.go](https://github.com/purpleidea/mgmt/blob/master/main.go) to `true`, and post the logs when you report the [issue](https://github.com/purpleidea/mgmt/issues).
+Bonus points if you provide a [shell](https://github.com/purpleidea/mgmt/tree/master/test/shell) or [OMV](https://github.com/purpleidea/mgmt/tree/master/test/omv) reproducible test case.
+Feel free to read my article on [debugging golang programs](https://ttboj.wordpress.com/2016/02/15/debugging-golang-programs/).
 
 ## Dependencies:
-* golang (available in most distros)
-* pandoc (for building a pdf of the documentation)
+* golang 1.4 or higher (required, available in most distros)
+* golang libraries (required, available with `go get`)
+
+        go get github.com/coreos/etcd/client
+        go get gopkg.in/yaml.v2
+        go get gopkg.in/fsnotify.v1
+        go get github.com/codegangsta/cli
+        go get github.com/coreos/go-systemd/dbus
+        go get github.com/coreos/go-systemd/util
+
+* stringer (required for building), available as a package on some platforms, otherwise via `go get`
+
+        go get golang.org/x/tools/cmd/stringer
+
+* pandoc (optional, for building a pdf of the documentation)
+* graphviz (optional, for building a visual representation of the graph)
 
 ## Patches:
-We'd love to have your patch! Please send it by email, or as a pull request.
+We'd love to have your patches! Please send them by email, or as a pull request.
+
+## On the web:
+* Introductory blog post: [https://ttboj.wordpress.com/2016/01/18/next-generation-configuration-mgmt/](https://ttboj.wordpress.com/2016/01/18/next-generation-configuration-mgmt/)
+* Introductory recording from DevConf.cz 2016: [https://www.youtube.com/watch?v=GVhpPF0j-iE&html5=1](https://www.youtube.com/watch?v=GVhpPF0j-iE&html5=1)
+* Introductory recording from CfgMgmtCamp.eu 2016: [https://www.youtube.com/watch?v=fNeooSiIRnA&html5=1](https://www.youtube.com/watch?v=fNeooSiIRnA&html5=1)
+* Julian Dunn at CfgMgmtCamp.eu 2016: [https://www.youtube.com/watch?v=kfF9IATUask&t=1949&html5=1](https://www.youtube.com/watch?v=kfF9IATUask&t=1949&html5=1)
+* Walter Heck at CfgMgmtCamp.eu 2016: [http://www.slideshare.net/olindata/configuration-management-time-for-a-4th-generation/3](http://www.slideshare.net/olindata/configuration-management-time-for-a-4th-generation/3)
+* Marco Marongiu on mgmt: [http://syslog.me/2016/02/15/leap-or-die/](http://syslog.me/2016/02/15/leap-or-die/)
+* Felix Frank on puppet to mgmt "transpiling" [https://ffrank.github.io/features/2016/02/18/from-catalog-to-mgmt/](https://ffrank.github.io/features/2016/02/18/from-catalog-to-mgmt/)
+* Blog post on automatic edges and the pkg resource: [https://ttboj.wordpress.com/2016/03/14/automatic-edges-in-mgmt/](https://ttboj.wordpress.com/2016/03/14/automatic-edges-in-mgmt/)
 
 ##
 

THANKS

@@ -9,6 +9,8 @@ Chris Wright - For encouraging me to continue work on my prototype.
 
 Daniel Riek - For supporting and sheltering this project from bureaucracy.
 
+Ira Cooper - For having an algorithmic design discussion with me.
+
 Jeff Darcy - For some algorithm recommendations, and NACKing my TopoSort idea!
 
 Red Hat, inc. - For paying my salary, thus financially supporting my hacking.

TODO.md

@@ -0,0 +1,43 @@
+# TODO
+If you're looking for something to do, look here!
+Let us know if you're working on one of the items.
+
+## Package resource
+- [ ] base resource [bug](https://github.com/purpleidea/mgmt/issues/11)
+- [ ] dnf blocker [bug](https://github.com/hughsie/PackageKit/issues/110)
+- [ ] install signal blocker [bug](https://github.com/hughsie/PackageKit/issues/109)
+
+## File resource [bug](https://github.com/purpleidea/mgmt/issues/13) [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove)
+- [ ] ability to make/delete folders
+- [ ] recursive argument (can recursively watch/modify contents)
+- [ ] force argument (can cause switch from file <-> folder)
+- [ ] fanotify support [bug](https://github.com/go-fsnotify/fsnotify/issues/114)
+
+## Exec resource
+- [ ] base resource improvements
+
+## Timer resource
+- [ ] base resource [bug](https://github.com/purpleidea/mgmt/issues/15) [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove)
+- [ ] reset on recompile
+- [ ] increment algorithm (linear, exponential, etc...)
+
+## Etcd improvements
+- [ ] embedded etcd master
+- [ ] capnslog fixes [bug](https://github.com/coreos/etcd/issues/4115)
+
+## Language improvements
+- [ ] language design
+- [ ] lexer/parser
+- [ ] automatic language formatter, ala `gofmt`
+- [ ] gedit/gnome-builder/gtksourceview syntax highlighting
+- [ ] vim syntax highlighting
+- [ ] emacs syntax highlighting
+
+## Other
+- [ ] better error/retry handling
+- [ ] resource grouping
+- [ ] automatic dependency adding (eg: packagekit file dependencies)
+- [ ] mgmt systemd service file [bug](https://github.com/purpleidea/mgmt/issues/12) [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove)
+- [ ] deb package target in Makefile
+- [ ] reproducible builds
+- [ ] add your suggestions!

config.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify
@@ -19,30 +19,21 @@ package main
 
 import (
 	"errors"
+	"fmt"
 	"gopkg.in/yaml.v2"
 	"io/ioutil"
 	"log"
+	"reflect"
+	"strings"
 )
 
-type noopTypeConfig struct {
-	Name string `yaml:"name"`
-}
-
-type fileTypeConfig struct {
-	Name    string `yaml:"name"`
-	Path    string `yaml:"path"`
-	Content string `yaml:"content"`
-	State   string `yaml:"state"`
-}
-
-type serviceTypeConfig struct {
-	Name    string `yaml:"name"`
-	State   string `yaml:"state"`
-	Startup string `yaml:"startup"`
+type collectorResConfig struct {
+	Kind    string `yaml:"kind"`
+	Pattern string `yaml:"pattern"` // XXX: Not Implemented
 }
 
 type vertexConfig struct {
-	Type string `yaml:"type"`
+	Kind string `yaml:"kind"`
 	Name string `yaml:"name"`
 }
 
@@ -52,18 +43,21 @@ type edgeConfig struct {
 	To   vertexConfig `yaml:"to"`
 }
 
-type graphConfig struct {
+type GraphConfig struct {
 	Graph     string `yaml:"graph"`
-	Types struct {
-		Noop    []noopTypeConfig    `yaml:"noop"`
-		File    []fileTypeConfig    `yaml:"file"`
-		Service []serviceTypeConfig `yaml:"service"`
-	} `yaml:"types"`
+	Resources struct {
+		Noop []*NoopRes `yaml:"noop"`
+		Pkg  []*PkgRes  `yaml:"pkg"`
+		File []*FileRes `yaml:"file"`
+		Svc  []*SvcRes  `yaml:"svc"`
+		Exec []*ExecRes `yaml:"exec"`
+	} `yaml:"resources"`
+	Collector []collectorResConfig `yaml:"collect"`
 	Edges     []edgeConfig         `yaml:"edges"`
 	Comment   string               `yaml:"comment"`
 }
 
-func (c *graphConfig) Parse(data []byte) error {
+func (c *GraphConfig) Parse(data []byte) error {
 	if err := yaml.Unmarshal(data, c); err != nil {
 		return err
 	}
@@ -73,54 +67,469 @@ func (c *graphConfig) Parse(data []byte) error {
 	return nil
 }
 
-func GraphFromConfig(filename string) *Graph {
-
-	var NoopMap map[string]*Vertex = make(map[string]*Vertex)
-	var FileMap map[string]*Vertex = make(map[string]*Vertex)
-	var ServiceMap map[string]*Vertex = make(map[string]*Vertex)
-
-	var lookup map[string]map[string]*Vertex = make(map[string]map[string]*Vertex)
-	lookup["noop"] = NoopMap
-	lookup["file"] = FileMap
-	lookup["service"] = ServiceMap
-
+func ParseConfigFromFile(filename string) *GraphConfig {
 	data, err := ioutil.ReadFile(filename)
 	if err != nil {
-		log.Fatal(err)
+		log.Printf("Error: Config: ParseConfigFromFile: File: %v", err)
+		return nil
 	}
 
-	var config graphConfig
+	var config GraphConfig
 	if err := config.Parse(data); err != nil {
-		log.Fatal(err)
-	}
-	//fmt.Printf("%+v\n", config)	// debug
-
-	g := NewGraph(config.Graph)
-
-	for _, t := range config.Types.Noop {
-		NoopMap[t.Name] = NewVertex(t.Name, "noop")
-		// FIXME: duplicate of name stored twice... where should it go?
-		NoopMap[t.Name].Associate(NewNoopType(t.Name))
-		g.AddVertex(NoopMap[t.Name]) // call standalone in case not part of an edge
+		log.Printf("Error: Config: ParseConfigFromFile: Parse: %v", err)
+		return nil
 	}
 
-	for _, t := range config.Types.File {
-		FileMap[t.Name] = NewVertex(t.Name, "file")
-		// FIXME: duplicate of name stored twice... where should it go?
-		FileMap[t.Name].Associate(NewFileType(t.Name, t.Path, t.Content, t.State))
-		g.AddVertex(FileMap[t.Name]) // call standalone in case not part of an edge
+	return &config
 }
 
-	for _, t := range config.Types.Service {
-		ServiceMap[t.Name] = NewVertex(t.Name, "service")
-		// FIXME: duplicate of name stored twice... where should it go?
-		ServiceMap[t.Name].Associate(NewServiceType(t.Name, t.State, t.Startup))
-		g.AddVertex(ServiceMap[t.Name]) // call standalone in case not part of an edge
+// NewGraphFromConfig returns a new graph from existing input, such as from the
+// existing graph, and a GraphConfig struct.
+func (g *Graph) NewGraphFromConfig(config *GraphConfig, etcdO *EtcdWObject, hostname string) (*Graph, error) {
+
+	var graph *Graph // new graph to return
+	if g == nil {    // FIXME: how can we check for an empty graph?
+		graph = NewGraph("Graph") // give graph a default name
+	} else {
+		graph = g.Copy() // same vertices, since they're pointers!
+	}
+
+	var lookup = make(map[string]map[string]*Vertex)
+
+	//log.Printf("%+v", config) // debug
+
+	// TODO: if defined (somehow)...
+	graph.SetName(config.Graph) // set graph name
+
+	var keep []*Vertex // list of vertex which are the same in new graph
+
+	// use reflection to avoid duplicating code... better options welcome!
+	value := reflect.Indirect(reflect.ValueOf(config.Resources))
+	vtype := value.Type()
+	for i := 0; i < vtype.NumField(); i++ { // number of fields in struct
+		name := vtype.Field(i).Name // string of field name
+		field := value.FieldByName(name)
+		iface := field.Interface() // interface type of value
+		slice := reflect.ValueOf(iface)
+		// XXX: should we just drop these everywhere and have the kind strings be all lowercase?
+		kind := FirstToUpper(name)
+		if DEBUG {
+			log.Printf("Config: Processing: %v...", kind)
+		}
+		for j := 0; j < slice.Len(); j++ { // loop through resources of same kind
+			x := slice.Index(j).Interface()
+			obj, ok := x.(Res) // convert to Res type
+			if !ok {
+				return nil, fmt.Errorf("Error: Config: Can't convert: %v of type: %T to Res.", x, x)
+			}
+
+			if _, exists := lookup[kind]; !exists {
+				lookup[kind] = make(map[string]*Vertex)
+			}
+			// XXX: should we export based on a @@ prefix, or a metaparam
+			// like exported => true || exported => (host pattern)||(other pattern?)
+			if !strings.HasPrefix(obj.GetName(), "@@") { // exported resource
+				// XXX: we don't have a way of knowing if any of the
+				// metaparams are undefined, and as a result to set the
+				// defaults that we want! I hate the go yaml parser!!!
+				v := graph.GetVertexMatch(obj)
+				if v == nil { // no match found
+					obj.Init()
+					v = NewVertex(obj)
+					graph.AddVertex(v) // call standalone in case not part of an edge
+				}
+				lookup[kind][obj.GetName()] = v // used for constructing edges
+				keep = append(keep, v)          // append
+
+			} else {
+				// XXX: do this in a different function...
+				// add to etcd storage...
+				obj.SetName(obj.GetName()[2:]) //slice off @@
+
+				data, err := ResToB64(obj)
+				if err != nil {
+					return nil, fmt.Errorf("Config: Could not encode %v resource: %v, error: %v", kind, obj.GetName(), err)
+				}
+
+				if !etcdO.EtcdPut(hostname, obj.GetName(), kind, data) {
+					return nil, fmt.Errorf("Config: Could not export %v resource: %v", kind, obj.GetName())
+				}
+			}
+		}
+	}
+
+	// lookup from etcd graph
+	// do all the graph look ups in one single step, so that if the etcd
+	// database changes, we don't have a partial state of affairs...
+	nodes, ok := etcdO.EtcdGet()
+	if ok {
+		for _, t := range config.Collector {
+			// XXX: should we just drop these everywhere and have the kind strings be all lowercase?
+			kind := FirstToUpper(t.Kind)
+
+			// use t.Kind and optionally t.Pattern to collect from etcd storage
+			log.Printf("Collect: %v; Pattern: %v", kind, t.Pattern)
+			for _, str := range etcdO.EtcdGetProcess(nodes, kind) {
+				obj, err := B64ToRes(str)
+				if err != nil {
+					log.Printf("B64ToRes failed to decode: %v", err)
+					log.Printf("Collect: %v: not collected!", kind)
+					continue
+				}
+
+				if t.Pattern != "" { // XXX: simplistic for now
+					obj.CollectPattern(t.Pattern) // obj.Dirname = t.Pattern
+				}
+
+				log.Printf("Collect: %v[%v]: collected!", kind, obj.GetName())
+
+				// XXX: similar to other resource add code:
+				if _, exists := lookup[kind]; !exists {
+					lookup[kind] = make(map[string]*Vertex)
+				}
+				v := graph.GetVertexMatch(obj)
+				if v == nil { // no match found
+					obj.Init() // initialize go channels or things won't work!!!
+					v = NewVertex(obj)
+					graph.AddVertex(v) // call standalone in case not part of an edge
+				}
+				lookup[kind][obj.GetName()] = v // used for constructing edges
+				keep = append(keep, v)          // append
+			}
+		}
+	}
+
+	// get rid of any vertices we shouldn't "keep" (that aren't in new graph)
+	for _, v := range graph.GetVertices() {
+		if !VertexContains(v, keep) {
+			// wait for exit before starting new graph!
+			v.SendEvent(eventExit, true, false)
+			graph.DeleteVertex(v)
+		}
 	}
 
 	for _, e := range config.Edges {
-		g.AddEdge(lookup[e.From.Type][e.From.Name], lookup[e.To.Type][e.To.Name], NewEdge(e.Name))
+		if _, ok := lookup[FirstToUpper(e.From.Kind)]; !ok {
+			return nil, fmt.Errorf("Can't find 'from' resource!")
+		}
+		if _, ok := lookup[FirstToUpper(e.To.Kind)]; !ok {
+			return nil, fmt.Errorf("Can't find 'to' resource!")
+		}
+		if _, ok := lookup[FirstToUpper(e.From.Kind)][e.From.Name]; !ok {
+			return nil, fmt.Errorf("Can't find 'from' name!")
+		}
+		if _, ok := lookup[FirstToUpper(e.To.Kind)][e.To.Name]; !ok {
+			return nil, fmt.Errorf("Can't find 'to' name!")
+		}
+		graph.AddEdge(lookup[FirstToUpper(e.From.Kind)][e.From.Name], lookup[FirstToUpper(e.To.Kind)][e.To.Name], NewEdge(e.Name))
 	}
 
-	return g
+	return graph, nil
+}
+
+// add edges to the vertex in a graph based on if it matches a uuid list
+func (g *Graph) addEdgesByMatchingUUIDS(v *Vertex, uuids []ResUUID) []bool {
+	// search for edges and see what matches!
+	var result []bool
+
+	// loop through each uuid, and see if it matches any vertex
+	for _, uuid := range uuids {
+		var found = false
+		// uuid is a ResUUID object
+		for _, vv := range g.GetVertices() { // search
+			if v == vv { // skip self
+				continue
+			}
+			if DEBUG {
+				log.Printf("Compile: AutoEdge: Match: %v[%v] with UUID: %v[%v]", vv.Kind(), vv.GetName(), uuid.Kind(), uuid.GetName())
+			}
+			// we must match to an effective UUID for the resource,
+			// that is to say, the name value of a res is a helpful
+			// handle, but it is not necessarily a unique identity!
+			// remember, resources can return multiple UUID's each!
+			if UUIDExistsInUUIDs(uuid, vv.GetUUIDs()) {
+				// add edge from: vv -> v
+				if uuid.Reversed() {
+					txt := fmt.Sprintf("AutoEdge: %v[%v] -> %v[%v]", vv.Kind(), vv.GetName(), v.Kind(), v.GetName())
+					log.Printf("Compile: Adding %v", txt)
+					g.AddEdge(vv, v, NewEdge(txt))
+				} else { // edges go the "normal" way, eg: pkg resource
+					txt := fmt.Sprintf("AutoEdge: %v[%v] -> %v[%v]", v.Kind(), v.GetName(), vv.Kind(), vv.GetName())
+					log.Printf("Compile: Adding %v", txt)
+					g.AddEdge(v, vv, NewEdge(txt))
+				}
+				found = true
+				break
+			}
+		}
+		result = append(result, found)
+	}
+	return result
+}
+
+// add auto edges to graph
+func (g *Graph) AutoEdges() {
+	log.Println("Compile: Adding AutoEdges...")
+	for _, v := range g.GetVertices() { // for each vertexes autoedges
+		if !v.GetMeta().AutoEdge { // is the metaparam true?
+			continue
+		}
+		autoEdgeObj := v.AutoEdges()
+		if autoEdgeObj == nil {
+			log.Printf("%v[%v]: Config: No auto edges were found!", v.Kind(), v.GetName())
+			continue // next vertex
+		}
+
+		for { // while the autoEdgeObj has more uuids to add...
+			uuids := autoEdgeObj.Next() // get some!
+			if uuids == nil {
+				log.Printf("%v[%v]: Config: The auto edge list is empty!", v.Kind(), v.GetName())
+				break // inner loop
+			}
+			if DEBUG {
+				log.Println("Compile: AutoEdge: UUIDS:")
+				for i, u := range uuids {
+					log.Printf("Compile: AutoEdge: UUID%d: %v", i, u)
+				}
+			}
+
+			// match and add edges
+			result := g.addEdgesByMatchingUUIDS(v, uuids)
+
+			// report back, and find out if we should continue
+			if !autoEdgeObj.Test(result) {
+				break
+			}
+		}
+	}
+}
+
+// AutoGrouper is the required interface to implement for an autogroup algorithm
+type AutoGrouper interface {
+	// listed in the order these are typically called in...
+	name() string                                  // friendly identifier
+	init(*Graph) error                             // only call once
+	vertexNext() (*Vertex, *Vertex, error)         // mostly algorithmic
+	vertexCmp(*Vertex, *Vertex) error              // can we merge these ?
+	vertexMerge(*Vertex, *Vertex) (*Vertex, error) // vertex merge fn to use
+	edgeMerge(*Edge, *Edge) *Edge                  // edge merge fn to use
+	vertexTest(bool) (bool, error)                 // call until false
+}
+
+// baseGrouper is the base type for implementing the AutoGrouper interface
+type baseGrouper struct {
+	graph    *Graph    // store a pointer to the graph
+	vertices []*Vertex // cached list of vertices
+	i        int
+	j        int
+	done     bool
+}
+
+// name provides a friendly name for the logs to see
+func (ag *baseGrouper) name() string {
+	return "baseGrouper"
+}
+
+// init is called only once and before using other AutoGrouper interface methods
+// the name method is the only exception: call it any time without side effects!
+func (ag *baseGrouper) init(g *Graph) error {
+	if ag.graph != nil {
+		return fmt.Errorf("The init method has already been called!")
+	}
+	ag.graph = g                               // pointer
+	ag.vertices = ag.graph.GetVerticesSorted() // cache in deterministic order!
+	ag.i = 0
+	ag.j = 0
+	if len(ag.vertices) == 0 { // empty graph
+		ag.done = true
+		return nil
+	}
+	return nil
+}
+
+// vertexNext is a simple iterator that loops through vertex (pair) combinations
+// an intelligent algorithm would selectively offer only valid pairs of vertices
+// these should satisfy logical grouping requirements for the autogroup designs!
+// the desired algorithms can override, but keep this method as a base iterator!
+func (ag *baseGrouper) vertexNext() (v1, v2 *Vertex, err error) {
+	// this does a for v... { for w... { return v, w }} but stepwise!
+	l := len(ag.vertices)
+	if ag.i < l {
+		v1 = ag.vertices[ag.i]
+	}
+	if ag.j < l {
+		v2 = ag.vertices[ag.j]
+	}
+
+	// in case the vertex was deleted
+	if !ag.graph.HasVertex(v1) {
+		v1 = nil
+	}
+	if !ag.graph.HasVertex(v2) {
+		v2 = nil
+	}
+
+	// two nested loops...
+	if ag.j < l {
+		ag.j++
+	}
+	if ag.j == l {
+		ag.j = 0
+		if ag.i < l {
+			ag.i++
+		}
+		if ag.i == l {
+			ag.done = true
+		}
+	}
+
+	return
+}
+
+func (ag *baseGrouper) vertexCmp(v1, v2 *Vertex) error {
+	if v1 == nil || v2 == nil {
+		return fmt.Errorf("Vertex is nil!")
+	}
+	if v1 == v2 { // skip yourself
+		return fmt.Errorf("Vertices are the same!")
+	}
+	if v1.Kind() != v2.Kind() { // we must group similar kinds
+		// TODO: maybe future resources won't need this limitation?
+		return fmt.Errorf("The two resources aren't the same kind!")
+	}
+	// someone doesn't want to group!
+	if !v1.GetMeta().AutoGroup || !v2.GetMeta().AutoGroup {
+		return fmt.Errorf("One of the autogroup flags is false!")
+	}
+	if v1.Res.IsGrouped() { // already grouped!
+		return fmt.Errorf("Already grouped!")
+	}
+	if len(v2.Res.GetGroup()) > 0 { // already has children grouped!
+		return fmt.Errorf("Already has groups!")
+	}
+	if !v1.Res.GroupCmp(v2.Res) { // resource groupcmp failed!
+		return fmt.Errorf("The GroupCmp failed!")
+	}
+	return nil // success
+}
+
+func (ag *baseGrouper) vertexMerge(v1, v2 *Vertex) (v *Vertex, err error) {
+	// NOTE: it's important to use w.Res instead of w, b/c
+	// the w by itself is the *Vertex obj, not the *Res obj
+	// which is contained within it! They both satisfy the
+	// Res interface, which is why both will compile! :(
+	err = v1.Res.GroupRes(v2.Res) // GroupRes skips stupid groupings
+	return                        // success or fail, and no need to merge the actual vertices!
+}
+
+func (ag *baseGrouper) edgeMerge(e1, e2 *Edge) *Edge {
+	return e1 // noop
+}
+
+// vertexTest processes the results of the grouping for the algorithm to know
+// return an error if something went horribly wrong, and bool false to stop
+func (ag *baseGrouper) vertexTest(b bool) (bool, error) {
+	// NOTE: this particular baseGrouper version doesn't track what happens
+	// because since we iterate over every pair, we don't care which merge!
+	if ag.done {
+		return false, nil
+	}
+	return true, nil
+}
+
+// TODO: this algorithm may not be correct in all cases. replace if needed!
+type nonReachabilityGrouper struct {
+	baseGrouper // "inherit" what we want, and reimplement the rest
+}
+
+func (ag *nonReachabilityGrouper) name() string {
+	return "nonReachabilityGrouper"
+}
+
+// this algorithm relies on the observation that if there's a path from a to b,
+// then they *can't* be merged (b/c of the existing dependency) so therefore we
+// merge anything that *doesn't* satisfy this condition or that of the reverse!
+func (ag *nonReachabilityGrouper) vertexNext() (v1, v2 *Vertex, err error) {
+	for {
+		v1, v2, err = ag.baseGrouper.vertexNext() // get all iterable pairs
+		if err != nil {
+			log.Fatalf("Error running autoGroup(vertexNext): %v", err)
+		}
+
+		if v1 != v2 { // ignore self cmp early (perf optimization)
+			// if NOT reachable, they're viable...
+			out1 := ag.graph.Reachability(v1, v2)
+			out2 := ag.graph.Reachability(v2, v1)
+			if len(out1) == 0 && len(out2) == 0 {
+				return // return v1 and v2, they're viable
+			}
+		}
+
+		// if we got here, it means we're skipping over this candidate!
+		if ok, err := ag.baseGrouper.vertexTest(false); err != nil {
+			log.Fatalf("Error running autoGroup(vertexTest): %v", err)
+		} else if !ok {
+			return nil, nil, nil // done!
+		}
+
+		// the vertexTest passed, so loop and try with a new pair...
+	}
+}
+
+// autoGroup is the mechanical auto group "runner" that runs the interface spec
+func (g *Graph) autoGroup(ag AutoGrouper) chan string {
+	strch := make(chan string) // output log messages here
+	go func(strch chan string) {
+		strch <- fmt.Sprintf("Compile: Grouping: Algorithm: %v...", ag.name())
+		if err := ag.init(g); err != nil {
+			log.Fatalf("Error running autoGroup(init): %v", err)
+		}
+
+		for {
+			var v, w *Vertex
+			v, w, err := ag.vertexNext() // get pair to compare
+			if err != nil {
+				log.Fatalf("Error running autoGroup(vertexNext): %v", err)
+			}
+			merged := false
+			// save names since they change during the runs
+			vStr := fmt.Sprintf("%s", v) // valid even if it is nil
+			wStr := fmt.Sprintf("%s", w)
+
+			if err := ag.vertexCmp(v, w); err != nil { // cmp ?
+				if DEBUG {
+					strch <- fmt.Sprintf("Compile: Grouping: !GroupCmp for: %s into %s", wStr, vStr)
+				}
+
+				// remove grouped vertex and merge edges (res is safe)
+			} else if err := g.VertexMerge(v, w, ag.vertexMerge, ag.edgeMerge); err != nil { // merge...
+				strch <- fmt.Sprintf("Compile: Grouping: !VertexMerge for: %s into %s", wStr, vStr)
+
+			} else { // success!
+				strch <- fmt.Sprintf("Compile: Grouping: Success for: %s into %s", wStr, vStr)
+				merged = true // woo
+			}
+
+			// did these get used?
+			if ok, err := ag.vertexTest(merged); err != nil {
+				log.Fatalf("Error running autoGroup(vertexTest): %v", err)
+			} else if !ok {
+				break // done!
+			}
+		}
+
+		close(strch)
+		return
+	}(strch) // call function
+	return strch
+}
+
+// AutoGroup runs the auto grouping on the graph and prints out log messages
+func (g *Graph) AutoGroup() {
+	// receive log messages from channel...
+	// this allows test cases to avoid printing them when they're unwanted!
+	// TODO: this algorithm may not be correct in all cases. replace if needed!
+	for str := range g.autoGroup(&nonReachabilityGrouper{}) {
+		log.Println(str)
+	}
 }

configwatch.go

@@ -0,0 +1,155 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"gopkg.in/fsnotify.v1"
+	//"github.com/go-fsnotify/fsnotify" // git master of "gopkg.in/fsnotify.v1"
+	"log"
+	"math"
+	"path"
+	"strings"
+	"syscall"
+)
+
+// XXX: it would be great if we could reuse code between this and the file resource
+// XXX: patch this to submit it as part of go-fsnotify if they're interested...
+func ConfigWatch(file string) chan bool {
+	ch := make(chan bool)
+	go func() {
+		var safename = path.Clean(file) // no trailing slash
+
+		watcher, err := fsnotify.NewWatcher()
+		if err != nil {
+			log.Fatal(err)
+		}
+		defer watcher.Close()
+
+		patharray := PathSplit(safename) // tokenize the path
+		var index = len(patharray)       // starting index
+		var current string               // current "watcher" location
+		var deltaDepth int               // depth delta between watcher and event
+		var send = false                 // send event?
+
+		for {
+			current = strings.Join(patharray[0:index], "/")
+			if current == "" { // the empty string top is the root dir ("/")
+				current = "/"
+			}
+			log.Printf("Watching: %v", current) // attempting to watch...
+
+			// initialize in the loop so that we can reset on rm-ed handles
+			err = watcher.Add(current)
+			if err != nil {
+				if err == syscall.ENOENT {
+					index-- // usually not found, move up one dir
+				} else if err == syscall.ENOSPC {
+					// XXX: occasionally: no space left on device,
+					// XXX: probably due to lack of inotify watches
+					log.Printf("Out of inotify watches for config(%v)", file)
+					log.Fatal(err)
+				} else {
+					log.Printf("Unknown config(%v) error:", file)
+					log.Fatal(err)
+				}
+				index = int(math.Max(1, float64(index)))
+				continue
+			}
+
+			select {
+			case event := <-watcher.Events:
+				// the deeper you go, the bigger the deltaDepth is...
+				// this is the difference between what we're watching,
+				// and the event... doesn't mean we can't watch deeper
+				if current == event.Name {
+					deltaDepth = 0 // i was watching what i was looking for
+
+				} else if HasPathPrefix(event.Name, current) {
+					deltaDepth = len(PathSplit(current)) - len(PathSplit(event.Name)) // -1 or less
+
+				} else if HasPathPrefix(current, event.Name) {
+					deltaDepth = len(PathSplit(event.Name)) - len(PathSplit(current)) // +1 or more
+
+				} else {
+					// TODO different watchers get each others events!
+					// https://github.com/go-fsnotify/fsnotify/issues/95
+					// this happened with two values such as:
+					// event.Name: /tmp/mgmt/f3 and current: /tmp/mgmt/f2
+					continue
+				}
+				//log.Printf("The delta depth is: %v", deltaDepth)
+
+				// if we have what we wanted, awesome, send an event...
+				if event.Name == safename {
+					//log.Println("Event!")
+					send = true
+
+					// file removed, move the watch upwards
+					if deltaDepth >= 0 && (event.Op&fsnotify.Remove == fsnotify.Remove) {
+						//log.Println("Removal!")
+						watcher.Remove(current)
+						index--
+					}
+
+					// we must be a parent watcher, so descend in
+					if deltaDepth < 0 {
+						watcher.Remove(current)
+						index++
+					}
+
+					// if safename starts with event.Name, we're above, and no event should be sent
+				} else if HasPathPrefix(safename, event.Name) {
+					//log.Println("Above!")
+
+					if deltaDepth >= 0 && (event.Op&fsnotify.Remove == fsnotify.Remove) {
+						log.Println("Removal!")
+						watcher.Remove(current)
+						index--
+					}
+
+					if deltaDepth < 0 {
+						log.Println("Parent!")
+						if PathPrefixDelta(safename, event.Name) == 1 { // we're the parent dir
+							//send = true
+						}
+						watcher.Remove(current)
+						index++
+					}
+
+					// if event.Name startswith safename, send event, we're already deeper
+				} else if HasPathPrefix(event.Name, safename) {
+					//log.Println("Event2!")
+					//send = true
+				}
+
+			case err := <-watcher.Errors:
+				log.Printf("error: %v", err)
+				log.Fatal(err)
+
+			}
+
+			// do our event sending all together to avoid duplicate msgs
+			if send {
+				send = false
+				ch <- true
+			}
+		}
+		//close(ch)
+	}()
+	return ch
+}

etcd.go

@@ -0,0 +1,294 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"fmt"
+	etcd "github.com/coreos/etcd/client"
+	etcd_context "golang.org/x/net/context"
+	"log"
+	"math"
+	"strings"
+	"time"
+)
+
+//go:generate stringer -type=etcdMsg -output=etcdmsg_stringer.go
+type etcdMsg int
+
+const (
+	etcdStart etcdMsg = iota
+	etcdEvent
+	etcdFoo
+	etcdBar
+)
+
+//go:generate stringer -type=etcdConvergedState -output=etcdconvergedstate_stringer.go
+type etcdConvergedState int
+
+const (
+	etcdConvergedNil etcdConvergedState = iota
+	//etcdConverged
+	etcdConvergedTimeout
+)
+
+type EtcdWObject struct { // etcd wrapper object
+	seed           string
+	ctimeout       int
+	converged      chan bool
+	kapi           etcd.KeysAPI
+	convergedState etcdConvergedState
+}
+
+func (etcdO *EtcdWObject) GetConvergedState() etcdConvergedState {
+	return etcdO.convergedState
+}
+
+func (etcdO *EtcdWObject) SetConvergedState(state etcdConvergedState) {
+	etcdO.convergedState = state
+}
+
+func (etcdO *EtcdWObject) GetKAPI() etcd.KeysAPI {
+	if etcdO.kapi != nil { // memoize
+		return etcdO.kapi
+	}
+
+	cfg := etcd.Config{
+		Endpoints: []string{etcdO.seed},
+		Transport: etcd.DefaultTransport,
+		// set timeout per request to fail fast when the target endpoint is unavailable
+		HeaderTimeoutPerRequest: time.Second,
+	}
+
+	var c etcd.Client
+	var err error
+
+	c, err = etcd.New(cfg)
+	if err != nil {
+		// XXX: not sure if this ever errors
+		if cerr, ok := err.(*etcd.ClusterError); ok {
+			// XXX: not sure if this part ever matches
+			// not running or disconnected
+			if cerr == etcd.ErrClusterUnavailable {
+				log.Fatal("XXX: etcd: ErrClusterUnavailable")
+			} else {
+				log.Fatal("XXX: etcd: Unknown")
+			}
+		}
+		log.Fatal(err) // some unhandled error
+	}
+	etcdO.kapi = etcd.NewKeysAPI(c)
+	return etcdO.kapi
+}
+
+type EtcdChannelWatchResponse struct {
+	resp *etcd.Response
+	err  error
+}
+
+// wrap the etcd watcher.Next blocking function inside of a channel
+func (etcdO *EtcdWObject) EtcdChannelWatch(watcher etcd.Watcher, context etcd_context.Context) chan *EtcdChannelWatchResponse {
+	ch := make(chan *EtcdChannelWatchResponse)
+	go func() {
+		for {
+			resp, err := watcher.Next(context) // blocks here
+			ch <- &EtcdChannelWatchResponse{resp, err}
+		}
+	}()
+	return ch
+}
+
+func (etcdO *EtcdWObject) EtcdWatch() chan etcdMsg {
+	kapi := etcdO.GetKAPI()
+	ctimeout := etcdO.ctimeout
+	converged := etcdO.converged
+	// XXX: i think we need this buffered so that when we're hanging on the
+	// channel, which is inside the EtcdWatch main loop, we still want the
+	// calls to Get/Set on etcd to succeed, so blocking them here would
+	// kill the whole thing
+	ch := make(chan etcdMsg, 1) // XXX: buffer of at least 1 is required
+	go func(ch chan etcdMsg) {
+		tmin := 500   // initial (min) delay in ms
+		t := tmin     // current time
+		tmult := 2    // multiplier for exponential delay
+		tmax := 16000 // max delay
+		watcher := kapi.Watcher("/exported/", &etcd.WatcherOptions{Recursive: true})
+		etcdch := etcdO.EtcdChannelWatch(watcher, etcd_context.Background())
+		for {
+			log.Printf("Etcd: Watching...")
+			var resp *etcd.Response // = nil by default
+			var err error
+			select {
+			case out := <-etcdch:
+				etcdO.SetConvergedState(etcdConvergedNil)
+				resp, err = out.resp, out.err
+
+			case _ = <-TimeAfterOrBlock(ctimeout):
+				etcdO.SetConvergedState(etcdConvergedTimeout)
+				converged <- true
+				continue
+			}
+
+			if err != nil {
+				if err == etcd_context.Canceled {
+					// ctx is canceled by another routine
+					log.Fatal("Canceled")
+				} else if err == etcd_context.DeadlineExceeded {
+					// ctx is attached with a deadline and it exceeded
+					log.Fatal("Deadline")
+				} else if cerr, ok := err.(*etcd.ClusterError); ok {
+					// not running or disconnected
+					// TODO: is there a better way to parse errors?
+					for _, e := range cerr.Errors {
+						if strings.HasSuffix(e.Error(), "getsockopt: connection refused") {
+							t = int(math.Min(float64(t*tmult), float64(tmax)))
+							log.Printf("Etcd: Waiting %d ms for connection...", t)
+							time.Sleep(time.Duration(t) * time.Millisecond) // sleep for t ms
+
+						} else if e.Error() == "unexpected EOF" {
+							log.Printf("Etcd: Unexpected disconnect...")
+
+						} else if e.Error() == "EOF" {
+							log.Printf("Etcd: Disconnected...")
+
+						} else if strings.HasPrefix(e.Error(), "unsupported protocol scheme") {
+							// usually a bad peer endpoint value
+							log.Fatal("Bad peer endpoint value?")
+
+						} else {
+							log.Fatal("Woops: ", e.Error())
+						}
+					}
+				} else {
+					// bad cluster endpoints, which are not etcd servers
+					log.Fatal("Woops: ", err)
+				}
+			} else {
+				//log.Print(resp)
+				//log.Printf("Watcher().Node.Value(%v): %+v", key, resp.Node.Value)
+				// FIXME: we should actually reset when the server comes back, not here on msg!
+				//XXX: can we fix this with one of these patterns?: https://blog.golang.org/go-concurrency-patterns-timing-out-and
+				t = tmin // reset timer
+
+				// don't trigger event if nothing changed
+				if n, p := resp.Node, resp.PrevNode; resp.Action == "set" && p != nil {
+					if n.Key == p.Key && n.Value == p.Value {
+						continue
+					}
+				}
+
+				// FIXME: we get events on key/res/value changes for
+				// each res directory... ignore the non final ones...
+				// IOW, ignore everything except for the value or some
+				// field which gets set last... this could be the max count field thing...
+
+				log.Printf("Etcd: Value: %v", resp.Node.Value) // event
+				ch <- etcdEvent                                // event
+			}
+
+		} // end for loop
+		//close(ch)
+	}(ch) // call go routine
+	return ch
+}
+
+// helper function to store our data in etcd
+func (etcdO *EtcdWObject) EtcdPut(hostname, key, res string, data string) bool {
+	kapi := etcdO.GetKAPI()
+	path := fmt.Sprintf("/exported/%s/resources/%s/res", hostname, key)
+	_, err := kapi.Set(etcd_context.Background(), path, res, nil)
+	// XXX validate...
+
+	path = fmt.Sprintf("/exported/%s/resources/%s/value", hostname, key)
+	resp, err := kapi.Set(etcd_context.Background(), path, data, nil)
+	if err != nil {
+		if cerr, ok := err.(*etcd.ClusterError); ok {
+			// not running or disconnected
+			for _, e := range cerr.Errors {
+				if strings.HasSuffix(e.Error(), "getsockopt: connection refused") {
+				}
+				//if e == etcd.ErrClusterUnavailable
+			}
+		}
+		log.Printf("Etcd: Could not store %v key.", key)
+		return false
+	}
+	log.Print("Etcd: ", resp) // w00t... bonus
+	return true
+}
+
+// lookup /exported/ node hierarchy
+func (etcdO *EtcdWObject) EtcdGet() (etcd.Nodes, bool) {
+	kapi := etcdO.GetKAPI()
+	// key structure is /exported/<hostname>/resources/...
+	resp, err := kapi.Get(etcd_context.Background(), "/exported/", &etcd.GetOptions{Recursive: true})
+	if err != nil {
+		return nil, false // not found
+	}
+	return resp.Node.Nodes, true
+}
+
+func (etcdO *EtcdWObject) EtcdGetProcess(nodes etcd.Nodes, res string) []string {
+	//path := fmt.Sprintf("/exported/%s/resources/", h)
+	top := "/exported/"
+	log.Printf("Etcd: Get: %+v", nodes) // Get().Nodes.Nodes
+	var output []string
+
+	for _, x := range nodes { // loop through hosts
+		if !strings.HasPrefix(x.Key, top) {
+			log.Fatal("Error!")
+		}
+		host := x.Key[len(top):]
+		//log.Printf("Get().Nodes[%v]: %+v ==> %+v", -1, host, x.Nodes)
+		//log.Printf("Get().Nodes[%v]: %+v ==> %+v", i, x.Key, x.Nodes)
+		resources, ok := EtcdGetChildNodeByKey(x, "resources")
+		if !ok {
+			continue
+		}
+		for _, y := range resources.Nodes { // loop through resources
+			//key := y.Key # UUID?
+			//log.Printf("Get(%v): RES[%v]", host, y.Key)
+			t, ok := EtcdGetChildNodeByKey(y, "res")
+			if !ok {
+				continue
+			}
+			if res != "" && res != t.Value {
+				continue
+			} // filter based on res
+
+			v, ok := EtcdGetChildNodeByKey(y, "value") // B64ToObj this
+			if !ok {
+				continue
+			}
+			log.Printf("Etcd: Hostname: %v; Get: %v", host, t.Value)
+
+			output = append(output, v.Value)
+		}
+	}
+	return output
+}
+
+// TODO: wrap this somehow so it's a method of *etcd.Node
+// helper function that returns the node for a particular key under a node
+func EtcdGetChildNodeByKey(node *etcd.Node, key string) (*etcd.Node, bool) {
+	for _, x := range node.Nodes {
+		if x.Key == fmt.Sprintf("%s/%s", node.Key, key) {
+			return x, true
+		}
+	}
+	return nil, false // not found
+}

event.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify
@@ -17,20 +17,59 @@
 
 package main
 
-import (
-	"code.google.com/p/go-uuid/uuid"
+//go:generate stringer -type=eventName -output=eventname_stringer.go
+type eventName int
+
+const (
+	eventNil eventName = iota
+	eventExit
+	eventStart
+	eventPause
+	eventPoke
+	eventBackPoke
 )
 
+type Resp chan bool
+
 type Event struct {
-	uuid string
-	Name string
-	Type string
+	Name eventName
+	Resp Resp // channel to send an ack response on, nil to skip
+	//Wg   *sync.WaitGroup // receiver barrier to Wait() for everyone else on
+	Msg      string // some words for fun
+	Activity bool   // did something interesting happen?
 }
 
-func NewEvent(name, t string) *Event {
-	return &Event{
-		uuid: uuid.New(),
-		Name: name,
-		Type: t,
+// send a single acknowledgement on the channel if one was requested
+func (event *Event) ACK() {
+	if event.Resp != nil { // if they've requested an ACK
+		event.Resp <- true // send ACK
 	}
 }
+
+func (event *Event) NACK() {
+	if event.Resp != nil { // if they've requested an ACK
+		event.Resp <- false // send NACK
+	}
+}
+
+// Resp is just a helper to return the right type of response channel
+func NewResp() Resp {
+	resp := make(chan bool)
+	return resp
+}
+
+// ACKWait waits for a +ive Ack from a Resp channel
+func (resp Resp) ACKWait() {
+	for {
+		value := <-resp
+		// wait until true value
+		if value {
+			return
+		}
+	}
+}
+
+// get the activity value
+func (event *Event) GetActivity() bool {
+	return event.Activity
+}

examples/autoedges1.yaml

@@ -0,0 +1,19 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1
+    meta:
+      autoedge: true
+    path: "/tmp/foo/bar/f1"
+    content: |
+      i am f1
+    state: exists
+  - name: file2
+    meta:
+      autoedge: true
+    path: "/tmp/foo/"
+    content: |
+      i am f2
+    state: exists
+edges: []

examples/autoedges2.yaml

@@ -0,0 +1,24 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1
+    meta:
+      autoedge: true
+    path: "/etc/drbd.conf"
+    content: |
+      # this is an mgmt test
+    state: exists
+  - name: file2
+    meta:
+      autoedge: true
+    path: "/tmp/foo/"
+    content: |
+      i am f2
+    state: exists
+  pkg:
+  - name: drbd-utils
+    meta:
+      autoedge: true
+    state: installed
+edges: []

examples/autoedges3.yaml

@@ -0,0 +1,29 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: drbd-utils
+    meta:
+      autoedge: true
+    state: installed
+  file:
+  - name: file1
+    meta:
+      autoedge: true
+    path: "/etc/drbd.conf"
+    content: |
+      # this is an mgmt test
+    state: exists
+  - name: file2
+    meta:
+      autoedge: true
+    path: "/etc/drbd.d/"
+    content: |
+      i am a directory
+    state: exists
+  svc:
+  - name: drbd
+    meta:
+      autoedge: true
+    state: stopped
+edges: []

examples/autogroup1.yaml

@@ -0,0 +1,21 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: drbd-utils
+    meta:
+      autogroup: false
+    state: installed
+  - name: powertop
+    meta:
+      autogroup: true
+    state: installed
+  - name: sl
+    meta:
+      autogroup: true
+    state: installed
+  - name: cowsay
+    meta:
+      autogroup: true
+    state: installed
+edges: []

examples/etcd1a.yaml

@@ -0,0 +1,18 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1a
+    path: "/tmp/mgmtA/f1a"
+    content: |
+      i am f1
+    state: exists
+  - name: "@@file2a"
+    path: "/tmp/mgmtA/f2a"
+    content: |
+      i am f2, exported from host A
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtA/"
+edges: []

examples/etcd1b.yaml

@@ -0,0 +1,18 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1b
+    path: "/tmp/mgmtB/f1b"
+    content: |
+      i am f1
+    state: exists
+  - name: "@@file2b"
+    path: "/tmp/mgmtB/f2b"
+    content: |
+      i am f2, exported from host B
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtB/"
+edges: []

examples/etcd1c.yaml

@@ -0,0 +1,18 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1c
+    path: "/tmp/mgmtC/f1c"
+    content: |
+      i am f1
+    state: exists
+  - name: "@@file2c"
+    path: "/tmp/mgmtC/f2c"
+    content: |
+      i am f2, exported from host C
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtC/"
+edges: []

examples/exec1.yaml

@@ -0,0 +1,59 @@
+---
+graph: mygraph
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec3
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec4
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec2
+- name: e2
+  from:
+    kind: exec
+    name: exec2
+  to:
+    kind: exec
+    name: exec3

examples/exec1a.yaml

@@ -0,0 +1,32 @@
+---
+graph: mygraph
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec2

examples/exec1b.yaml

@@ -0,0 +1,32 @@
+---
+graph: mygraph
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: 'true'
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: 'true'
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec2

examples/exec1c.yaml

@@ -0,0 +1,32 @@
+---
+graph: mygraph
+resources:
+  exec:
+  - name: exec1
+    cmd: echo hello from exec1
+    shell: ''
+    timeout: 0
+    watchcmd: sleep 10s
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: echo hello from exec2
+    shell: ''
+    timeout: 0
+    watchcmd: sleep 10s
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec2

examples/exec1d.yaml

@@ -0,0 +1,15 @@
+---
+graph: mygraph
+resources:
+  exec:
+  - name: exec1
+    cmd: echo hello from exec1
+    shell: ''
+    timeout: 0
+    watchcmd: sleep 5s
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges: []

examples/exec2.yaml

@@ -0,0 +1,83 @@
+---
+graph: mygraph
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec3
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec4
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec5
+    cmd: sleep 15s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec2
+- name: e2
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec3
+- name: e3
+  from:
+    kind: exec
+    name: exec2
+  to:
+    kind: exec
+    name: exec4
+- name: e4
+  from:
+    kind: exec
+    name: exec3
+  to:
+    kind: exec
+    name: exec4

examples/file1.yaml

@@ -0,0 +1,41 @@
+---
+graph: mygraph
+resources:
+  noop:
+  - name: noop1
+  file:
+  - name: file1
+    path: "/tmp/mgmt/f1"
+    content: |
+      i am f1
+    state: exists
+  - name: file2
+    path: "/tmp/mgmt/f2"
+    content: |
+      i am f2
+    state: exists
+  - name: file3
+    path: "/tmp/mgmt/f3"
+    content: |
+      i am f3
+    state: exists
+  - name: file4
+    path: "/tmp/mgmt/f4"
+    content: |
+      i am f4 and i should not be here
+    state: absent
+edges:
+- name: e1
+  from:
+    kind: file
+    name: file1
+  to:
+    kind: file
+    name: file2
+- name: e2
+  from:
+    kind: file
+    name: file2
+  to:
+    kind: file
+    name: file3

examples/graph0.yaml

@@ -0,0 +1,20 @@
+---
+graph: mygraph
+comment: hello world example
+resources:
+  noop:
+  - name: noop1
+  file:
+  - name: file1
+    path: "/tmp/mgmt-hello-world"
+    content: |
+      hello world from @purpleidea
+    state: exists
+edges:
+- name: e1
+  from:
+    kind: noop
+    name: noop1
+  to:
+    kind: file
+    name: file1

examples/graph10.yaml

@@ -0,0 +1,128 @@
+---
+graph: mygraph
+comment: simple exec fan in to fan out example to demonstrate optimization
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec3
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec4
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec5
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec6
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec7
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec8
+    cmd: sleep 15s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec4
+- name: e2
+  from:
+    kind: exec
+    name: exec2
+  to:
+    kind: exec
+    name: exec4
+- name: e3
+  from:
+    kind: exec
+    name: exec3
+  to:
+    kind: exec
+    name: exec4
+- name: e4
+  from:
+    kind: exec
+    name: exec4
+  to:
+    kind: exec
+    name: exec5
+- name: e5
+  from:
+    kind: exec
+    name: exec4
+  to:
+    kind: exec
+    name: exec6
+- name: e6
+  from:
+    kind: exec
+    name: exec4
+  to:
+    kind: exec
+    name: exec7

examples/graph1a.yaml

@@ -0,0 +1,22 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1
+    path: "/tmp/mgmt/f1"
+    content: |
+      i am f1
+    state: exists
+  - name: file2
+    path: "/tmp/mgmt/f2"
+    content: |
+      i am f2
+    state: exists
+edges:
+- name: e1
+  from:
+    kind: file
+    name: file1
+  to:
+    kind: file
+    name: file2

examples/graph1b.yaml

@@ -0,0 +1,22 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file2
+    path: "/tmp/mgmt/f2"
+    content: |
+      i am f2
+    state: exists
+  - name: file3
+    path: "/tmp/mgmt/f3"
+    content: |
+      i am f3
+    state: exists
+edges:
+- name: e2
+  from:
+    kind: file
+    name: file2
+  to:
+    kind: file
+    name: file3

examples/graph3a.yaml

@@ -0,0 +1,28 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1a
+    path: "/tmp/mgmtA/f1a"
+    content: |
+      i am f1
+    state: exists
+  - name: file2a
+    path: "/tmp/mgmtA/f2a"
+    content: |
+      i am f2
+    state: exists
+  - name: "@@file3a"
+    path: "/tmp/mgmtA/f3a"
+    content: |
+      i am f3, exported from host A
+    state: exists
+  - name: "@@file4a"
+    path: "/tmp/mgmtA/f4a"
+    content: |
+      i am f4, exported from host A
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtA/"
+edges: []

examples/graph3b.yaml

@@ -0,0 +1,28 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1b
+    path: "/tmp/mgmtB/f1b"
+    content: |
+      i am f1
+    state: exists
+  - name: file2b
+    path: "/tmp/mgmtB/f2b"
+    content: |
+      i am f2
+    state: exists
+  - name: "@@file3b"
+    path: "/tmp/mgmtB/f3b"
+    content: |
+      i am f3, exported from host B
+    state: exists
+  - name: "@@file4b"
+    path: "/tmp/mgmtB/f4b"
+    content: |
+      i am f4, exported from host B
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtB/"
+edges: []

examples/graph3c.yaml

@@ -0,0 +1,28 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1c
+    path: "/tmp/mgmtC/f1c"
+    content: |
+      i am f1
+    state: exists
+  - name: file2c
+    path: "/tmp/mgmtC/f2c"
+    content: |
+      i am f2
+    state: exists
+  - name: "@@file3c"
+    path: "/tmp/mgmtC/f3c"
+    content: |
+      i am f3, exported from host C
+    state: exists
+  - name: "@@file4c"
+    path: "/tmp/mgmtC/f4c"
+    content: |
+      i am f4, exported from host C
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtC/"
+edges: []

examples/graph4.yaml

@@ -0,0 +1,18 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1
+    path: "/tmp/mgmt/f1"
+    content: |
+      i am f1
+    state: exists
+  - name: "@@file3"
+    path: "/tmp/mgmt/f3"
+    content: |
+      i am f3, exported from host A
+    state: exists
+collect:
+- kind: file
+  pattern: ''
+edges:

examples/graph5.yaml

@@ -0,0 +1,13 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1
+    path: "/tmp/mgmt/f1"
+    content: |
+      i am f1
+    state: exists
+collect:
+- kind: file
+  pattern: ''
+edges:

examples/graph6.yaml

@@ -0,0 +1,6 @@
+---
+graph: mygraph
+resources:
+  noop:
+  - name: noop1
+edges:

examples/graph7.yaml

@@ -0,0 +1,17 @@
+---
+graph: mygraph
+resources:
+  noop:
+  - name: noop1
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:

examples/graph9.yaml

@@ -0,0 +1,77 @@
+---
+graph: mygraph
+comment: simple exec fan in example to demonstrate optimization
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec3
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec4
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec5
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec5
+- name: e2
+  from:
+    kind: exec
+    name: exec2
+  to:
+    kind: exec
+    name: exec5
+- name: e3
+  from:
+    kind: exec
+    name: exec3
+  to:
+    kind: exec
+    name: exec5

examples/pkg1.yaml

@@ -0,0 +1,7 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: powertop
+    state: installed
+edges: []

examples/pkg2.yaml

@@ -0,0 +1,7 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: powertop
+    state: uninstalled
+edges: []

examples/purpleidea.service

@@ -0,0 +1,8 @@
+[Unit]
+Description=Fake service for testing
+
+[Service]
+ExecStart=/usr/bin/sleep 8h
+
+[Install]
+WantedBy=multi-user.target

examples/svc1.yaml

@@ -1,30 +1,30 @@
 ---
 graph: mygraph
-types:
+resources:
   noop:
   - name: noop1
   file:
   - name: file1
-    path: /tmp/mgmt/f1
+    path: "/tmp/mgmt/f1"
     content: |
       i am f1
     state: exists
-  service:
+  svc:
   - name: purpleidea
     state: running
     startup: enabled
 edges:
 - name: e1
   from:
-    type: noop
+    kind: noop
     name: noop1
   to:
-    type: file
+    kind: file
     name: file1
 - name: e2
   from:
-    type: file
+    kind: file
     name: file1
   to:
-    type: service
+    kind: svc
     name: purpleidea

exec.go

@@ -0,0 +1,425 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"bufio"
+	"bytes"
+	"encoding/gob"
+	"errors"
+	"log"
+	"os/exec"
+	"strings"
+)
+
+func init() {
+	gob.Register(&ExecRes{})
+}
+
+type ExecRes struct {
+	BaseRes    `yaml:",inline"`
+	State      string `yaml:"state"`      // state: exists/present?, absent, (undefined?)
+	Cmd        string `yaml:"cmd"`        // the command to run
+	Shell      string `yaml:"shell"`      // the (optional) shell to use to run the cmd
+	Timeout    int    `yaml:"timeout"`    // the cmd timeout in seconds
+	WatchCmd   string `yaml:"watchcmd"`   // the watch command to run
+	WatchShell string `yaml:"watchshell"` // the (optional) shell to use to run the watch cmd
+	IfCmd      string `yaml:"ifcmd"`      // the if command to run
+	IfShell    string `yaml:"ifshell"`    // the (optional) shell to use to run the if cmd
+	PollInt    int    `yaml:"pollint"`    // the poll interval for the ifcmd
+}
+
+func NewExecRes(name, cmd, shell string, timeout int, watchcmd, watchshell, ifcmd, ifshell string, pollint int, state string) *ExecRes {
+	obj := &ExecRes{
+		BaseRes: BaseRes{
+			Name: name,
+		},
+		Cmd:        cmd,
+		Shell:      shell,
+		Timeout:    timeout,
+		WatchCmd:   watchcmd,
+		WatchShell: watchshell,
+		IfCmd:      ifcmd,
+		IfShell:    ifshell,
+		PollInt:    pollint,
+		State:      state,
+	}
+	obj.Init()
+	return obj
+}
+
+func (obj *ExecRes) Init() {
+	obj.BaseRes.kind = "Exec"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+}
+
+// validate if the params passed in are valid data
+// FIXME: where should this get called ?
+func (obj *ExecRes) Validate() bool {
+	if obj.Cmd == "" { // this is the only thing that is really required
+		return false
+	}
+
+	// if we have a watch command, then we don't poll with the if command!
+	if obj.WatchCmd != "" && obj.PollInt > 0 {
+		return false
+	}
+
+	return true
+}
+
+// wraps the scanner output in a channel
+func (obj *ExecRes) BufioChanScanner(scanner *bufio.Scanner) (chan string, chan error) {
+	ch, errch := make(chan string), make(chan error)
+	go func() {
+		for scanner.Scan() {
+			ch <- scanner.Text() // blocks here ?
+			if e := scanner.Err(); e != nil {
+				errch <- e // send any misc errors we encounter
+				//break // TODO ?
+			}
+		}
+		close(ch)
+		errch <- scanner.Err() // eof or some err
+		close(errch)
+	}()
+	return ch, errch
+}
+
+// Exec watcher
+func (obj *ExecRes) Watch(processChan chan Event) {
+	if obj.IsWatching() {
+		return
+	}
+	obj.SetWatching(true)
+	defer obj.SetWatching(false)
+
+	var send = false // send event?
+	var exit = false
+	bufioch, errch := make(chan string), make(chan error)
+	//vertex := obj.GetVertex()         // stored with SetVertex
+
+	if obj.WatchCmd != "" {
+		var cmdName string
+		var cmdArgs []string
+		if obj.WatchShell == "" {
+			// call without a shell
+			// FIXME: are there still whitespace splitting issues?
+			split := strings.Fields(obj.WatchCmd)
+			cmdName = split[0]
+			//d, _ := os.Getwd() // TODO: how does this ever error ?
+			//cmdName = path.Join(d, cmdName)
+			cmdArgs = split[1:len(split)]
+		} else {
+			cmdName = obj.Shell // usually bash, or sh
+			cmdArgs = []string{"-c", obj.WatchCmd}
+		}
+		cmd := exec.Command(cmdName, cmdArgs...)
+		//cmd.Dir = "" // look for program in pwd ?
+
+		cmdReader, err := cmd.StdoutPipe()
+		if err != nil {
+			log.Printf("%v[%v]: Error creating StdoutPipe for Cmd: %v", obj.Kind(), obj.GetName(), err)
+			log.Fatal(err) // XXX: how should we handle errors?
+		}
+		scanner := bufio.NewScanner(cmdReader)
+
+		defer cmd.Wait() // XXX: is this necessary?
+		defer func() {
+			// FIXME: without wrapping this in this func it panic's
+			// when running examples/graph8d.yaml
+			cmd.Process.Kill() // TODO: is this necessary?
+		}()
+		if err := cmd.Start(); err != nil {
+			log.Printf("%v[%v]: Error starting Cmd: %v", obj.Kind(), obj.GetName(), err)
+			log.Fatal(err) // XXX: how should we handle errors?
+		}
+
+		bufioch, errch = obj.BufioChanScanner(scanner)
+	}
+
+	for {
+		obj.SetState(resStateWatching) // reset
+		select {
+		case text := <-bufioch:
+			obj.SetConvergedState(resConvergedNil)
+			// each time we get a line of output, we loop!
+			log.Printf("%v[%v]: Watch output: %s", obj.Kind(), obj.GetName(), text)
+			if text != "" {
+				send = true
+			}
+
+		case err := <-errch:
+			obj.SetConvergedState(resConvergedNil) // XXX ?
+			if err == nil {                        // EOF
+				// FIXME: add an "if watch command ends/crashes"
+				// restart or generate error option
+				log.Printf("%v[%v]: Reached EOF", obj.Kind(), obj.GetName())
+				return
+			}
+			log.Printf("%v[%v]: Error reading input?: %v", obj.Kind(), obj.GetName(), err)
+			log.Fatal(err)
+			// XXX: how should we handle errors?
+
+		case event := <-obj.events:
+			obj.SetConvergedState(resConvergedNil)
+			if exit, send = obj.ReadEvent(&event); exit {
+				return // exit
+			}
+
+		case _ = <-TimeAfterOrBlock(obj.ctimeout):
+			obj.SetConvergedState(resConvergedTimeout)
+			obj.converged <- true
+			continue
+		}
+
+		// do all our event sending all together to avoid duplicate msgs
+		if send {
+			send = false
+			// it is okay to invalidate the clean state on poke too
+			obj.isStateOK = false // something made state dirty
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
+		}
+	}
+}
+
+// TODO: expand the IfCmd to be a list of commands
+func (obj *ExecRes) CheckApply(apply bool) (stateok bool, err error) {
+	log.Printf("%v[%v]: CheckApply(%t)", obj.Kind(), obj.GetName(), apply)
+
+	// if there is a watch command, but no if command, run based on state
+	if obj.WatchCmd != "" && obj.IfCmd == "" {
+		if obj.isStateOK {
+			return true, nil
+		}
+
+		// if there is no watcher, but there is an onlyif check, run it to see
+	} else if obj.IfCmd != "" { // && obj.WatchCmd == ""
+		// there is a watcher, but there is also an if command
+		//} else if obj.IfCmd != "" && obj.WatchCmd != "" {
+
+		if obj.PollInt > 0 { // && obj.WatchCmd == ""
+			// XXX have the Watch() command output onlyif poll events...
+			// XXX we can optimize by saving those results for returning here
+			// return XXX
+		}
+
+		var cmdName string
+		var cmdArgs []string
+		if obj.IfShell == "" {
+			// call without a shell
+			// FIXME: are there still whitespace splitting issues?
+			split := strings.Fields(obj.IfCmd)
+			cmdName = split[0]
+			//d, _ := os.Getwd() // TODO: how does this ever error ?
+			//cmdName = path.Join(d, cmdName)
+			cmdArgs = split[1:len(split)]
+		} else {
+			cmdName = obj.IfShell // usually bash, or sh
+			cmdArgs = []string{"-c", obj.IfCmd}
+		}
+		err = exec.Command(cmdName, cmdArgs...).Run()
+		if err != nil {
+			// TODO: check exit value
+			return true, nil // don't run
+		}
+
+		// if there is no watcher and no onlyif check, assume we should run
+	} else { // if obj.WatchCmd == "" && obj.IfCmd == "" {
+		// just run if state is dirty
+		if obj.isStateOK {
+			return true, nil
+		}
+	}
+
+	// state is not okay, no work done, exit, but without error
+	if !apply {
+		return false, nil
+	}
+
+	// apply portion
+	log.Printf("%v[%v]: Apply", obj.Kind(), obj.GetName())
+	var cmdName string
+	var cmdArgs []string
+	if obj.Shell == "" {
+		// call without a shell
+		// FIXME: are there still whitespace splitting issues?
+		// TODO: we could make the split character user selectable...!
+		split := strings.Fields(obj.Cmd)
+		cmdName = split[0]
+		//d, _ := os.Getwd() // TODO: how does this ever error ?
+		//cmdName = path.Join(d, cmdName)
+		cmdArgs = split[1:len(split)]
+	} else {
+		cmdName = obj.Shell // usually bash, or sh
+		cmdArgs = []string{"-c", obj.Cmd}
+	}
+	cmd := exec.Command(cmdName, cmdArgs...)
+	//cmd.Dir = "" // look for program in pwd ?
+	var out bytes.Buffer
+	cmd.Stdout = &out
+
+	if err = cmd.Start(); err != nil {
+		log.Printf("%v[%v]: Error starting Cmd: %v", obj.Kind(), obj.GetName(), err)
+		return false, err
+	}
+
+	timeout := obj.Timeout
+	if timeout == 0 { // zero timeout means no timer, so disable it
+		timeout = -1
+	}
+	done := make(chan error)
+	go func() { done <- cmd.Wait() }()
+
+	select {
+	case err = <-done:
+		if err != nil {
+			log.Printf("%v[%v]: Error waiting for Cmd: %v", obj.Kind(), obj.GetName(), err)
+			return false, err
+		}
+
+	case <-TimeAfterOrBlock(timeout):
+		log.Printf("%v[%v]: Timeout waiting for Cmd", obj.Kind(), obj.GetName())
+		//cmd.Process.Kill() // TODO: is this necessary?
+		return false, errors.New("Timeout waiting for Cmd!")
+	}
+
+	// TODO: if we printed the stdout while the command is running, this
+	// would be nice, but it would require terminal log output that doesn't
+	// interleave all the parallel parts which would mix it all up...
+	if s := out.String(); s == "" {
+		log.Printf("Exec[%v]: Command output is empty!", obj.Name)
+	} else {
+		log.Printf("Exec[%v]: Command output is:", obj.Name)
+		log.Printf(out.String())
+	}
+	// XXX: return based on exit value!!
+
+	// the state tracking is for exec resources that can't "detect" their
+	// state, and assume it's invalid when the Watch() function triggers.
+	// if we apply state successfully, we should reset it here so that we
+	// know that we have applied since the state was set not ok by event!
+	obj.isStateOK = true // reset
+	return false, nil    // success
+}
+
+type ExecUUID struct {
+	BaseUUID
+	Cmd   string
+	IfCmd string
+	// TODO: add more elements here
+}
+
+// if and only if they are equivalent, return true
+// if they are not equivalent, return false
+func (obj *ExecUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*ExecUUID)
+	if !ok {
+		return false
+	}
+	if obj.Cmd != res.Cmd {
+		return false
+	}
+	// TODO: add more checks here
+	//if obj.Shell != res.Shell {
+	//	return false
+	//}
+	//if obj.Timeout != res.Timeout {
+	//	return false
+	//}
+	//if obj.WatchCmd != res.WatchCmd {
+	//	return false
+	//}
+	//if obj.WatchShell != res.WatchShell {
+	//	return false
+	//}
+	if obj.IfCmd != res.IfCmd {
+		return false
+	}
+	//if obj.PollInt != res.PollInt {
+	//	return false
+	//}
+	//if obj.State != res.State {
+	//	return false
+	//}
+	return true
+}
+
+func (obj *ExecRes) AutoEdges() AutoEdge {
+	// TODO: parse as many exec params to look for auto edges, for example
+	// the path of the binary in the Cmd variable might be from in a pkg
+	return nil
+}
+
+// include all params to make a unique identification of this object
+func (obj *ExecRes) GetUUIDs() []ResUUID {
+	x := &ExecUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		Cmd:      obj.Cmd,
+		IfCmd:    obj.IfCmd,
+		// TODO: add more params here
+	}
+	return []ResUUID{x}
+}
+
+func (obj *ExecRes) GroupCmp(r Res) bool {
+	_, ok := r.(*SvcRes)
+	if !ok {
+		return false
+	}
+	return false // not possible atm
+}
+
+func (obj *ExecRes) Compare(res Res) bool {
+	switch res.(type) {
+	case *ExecRes:
+		res := res.(*ExecRes)
+		if obj.Name != res.Name {
+			return false
+		}
+		if obj.Cmd != res.Cmd {
+			return false
+		}
+		if obj.Shell != res.Shell {
+			return false
+		}
+		if obj.Timeout != res.Timeout {
+			return false
+		}
+		if obj.WatchCmd != res.WatchCmd {
+			return false
+		}
+		if obj.WatchShell != res.WatchShell {
+			return false
+		}
+		if obj.IfCmd != res.IfCmd {
+			return false
+		}
+		if obj.PollInt != res.PollInt {
+			return false
+		}
+		if obj.State != res.State {
+			return false
+		}
+	default:
+		return false
+	}
+	return true
+}

file.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify
@@ -18,12 +18,11 @@
 package main
 
 import (
-	"code.google.com/p/go-uuid/uuid"
 	"crypto/sha256"
 	"encoding/hex"
-	"fmt"
 	"gopkg.in/fsnotify.v1"
 	//"github.com/go-fsnotify/fsnotify" // git master of "gopkg.in/fsnotify.v1"
+	"encoding/gob"
 	"io"
 	"log"
 	"math"
@@ -33,40 +32,88 @@ import (
 	"syscall"
 )
 
-type FileType struct {
-	uuid      string
-	Type      string      // always "file"
-	Name      string      // name variable
-	Events    chan string // FIXME: eventually a struct for the event?
-	Path      string      // path variable (should default to name)
-	Content   string
-	State     string // state: exists/present?, absent, (undefined?)
+func init() {
+	gob.Register(&FileRes{})
+}
+
+type FileRes struct {
+	BaseRes   `yaml:",inline"`
+	Path      string `yaml:"path"` // path variable (should default to name)
+	Dirname   string `yaml:"dirname"`
+	Basename  string `yaml:"basename"`
+	Content   string `yaml:"content"`
+	State     string `yaml:"state"` // state: exists/present?, absent, (undefined?)
 	sha256sum string
 }
 
-func NewFileType(name, path, content, state string) *FileType {
+func NewFileRes(name, path, dirname, basename, content, state string) *FileRes {
 	// FIXME if path = nil, path = name ...
-	return &FileType{
-		uuid:      uuid.New(),
-		Type:      "file",
+	obj := &FileRes{
+		BaseRes: BaseRes{
 			Name: name,
-		Events:    make(chan string, 1), // XXX: chan size?
+		},
 		Path:      path,
+		Dirname:   dirname,
+		Basename:  basename,
 		Content:   content,
 		State:     state,
 		sha256sum: "",
 	}
+	obj.Init()
+	return obj
+}
+
+func (obj *FileRes) Init() {
+	obj.BaseRes.kind = "File"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+}
+
+func (obj *FileRes) GetPath() string {
+	d := Dirname(obj.Path)
+	b := Basename(obj.Path)
+	if !obj.Validate() || (obj.Dirname == "" && obj.Basename == "") {
+		return obj.Path
+	} else if obj.Dirname == "" {
+		return d + obj.Basename
+	} else if obj.Basename == "" {
+		return obj.Dirname + b
+	} else { // if obj.dirname != "" && obj.basename != "" {
+		return obj.Dirname + obj.Basename
+	}
+}
+
+// validate if the params passed in are valid data
+func (obj *FileRes) Validate() bool {
+	if obj.Dirname != "" {
+		// must end with /
+		if obj.Dirname[len(obj.Dirname)-1:] != "/" {
+			return false
+		}
+	}
+	if obj.Basename != "" {
+		// must not start with /
+		if obj.Basename[0:1] == "/" {
+			return false
+		}
+	}
+	return true
 }
 
 // File watcher for files and directories
 // Modify with caution, probably important to write some test cases first!
-func (obj FileType) Watch(v *Vertex) {
-	// obj.Path: file or directory
-	//var recursive bool = false
-	//var isdir = (obj.Path[len(obj.Path)-1:] == "/") // dirs have trailing slashes
-	//fmt.Printf("IsDirectory: %v\n", isdir)
+// obj.GetPath(): file or directory
+func (obj *FileRes) Watch(processChan chan Event) {
+	if obj.IsWatching() {
+		return
+	}
+	obj.SetWatching(true)
+	defer obj.SetWatching(false)
 
-	var safename = path.Clean(obj.Path) // no trailing slash
+	//var recursive bool = false
+	//var isdir = (obj.GetPath()[len(obj.GetPath())-1:] == "/") // dirs have trailing slashes
+	//log.Printf("IsDirectory: %v", isdir)
+	//vertex := obj.GetVertex()         // stored with SetVertex
+	var safename = path.Clean(obj.GetPath()) // no trailing slash
 
 	watcher, err := fsnotify.NewWatcher()
 	if err != nil {
@@ -77,90 +124,84 @@ func (obj FileType) Watch(v *Vertex) {
 	patharray := PathSplit(safename) // tokenize the path
 	var index = len(patharray)       // starting index
 	var current string               // current "watcher" location
-	var delta_depth int              // depth delta between watcher and event
+	var deltaDepth int               // depth delta between watcher and event
 	var send = false                 // send event?
-	var extraCheck = false
+	var exit = false
+	var dirty = false
 
 	for {
 		current = strings.Join(patharray[0:index], "/")
 		if current == "" { // the empty string top is the root dir ("/")
 			current = "/"
 		}
-		log.Printf("Watching: %v\n", current) // attempting to watch...
-
+		if DEBUG {
+			log.Printf("File[%v]: Watching: %v", obj.GetName(), current) // attempting to watch...
+		}
 		// initialize in the loop so that we can reset on rm-ed handles
 		err = watcher.Add(current)
 		if err != nil {
+			if DEBUG {
+				log.Printf("File[%v]: watcher.Add(%v): Error: %v", obj.GetName(), current, err)
+			}
 			if err == syscall.ENOENT {
 				index-- // usually not found, move up one dir
 			} else if err == syscall.ENOSPC {
-				// XXX: i sometimes see: no space left on device
-				// XXX: why causes this to happen ?
-				log.Printf("Strange file[%v] error: %+v\n", obj.Name, err.Error) // 0x408da0
+				// XXX: occasionally: no space left on device,
+				// XXX: probably due to lack of inotify watches
+				log.Printf("%v[%v]: Out of inotify watches!", obj.Kind(), obj.GetName())
 				log.Fatal(err)
 			} else {
-				log.Printf("Unknown file[%v] error:\n", obj.Name)
+				log.Printf("Unknown file[%v] error:", obj.Name)
 				log.Fatal(err)
 			}
 			index = int(math.Max(1, float64(index)))
 			continue
 		}
 
-		// XXX: check state after inotify started
-		// SMALL RACE: after we terminate watch, till when it's started
-		// something could have gotten created/changed/etc... right?
-		if extraCheck {
-			extraCheck = false
-			// XXX
-			//if exists ... {
-			//	send signal
-			//	continue
-			//	change index? i don't think so. be thorough and check
-			//}
-		}
-
+		obj.SetState(resStateWatching) // reset
 		select {
 		case event := <-watcher.Events:
-			// the deeper you go, the bigger the delta_depth is...
+			if DEBUG {
+				log.Printf("File[%v]: Watch(%v), Event(%v): %v", obj.GetName(), current, event.Name, event.Op)
+			}
+			obj.SetConvergedState(resConvergedNil) // XXX: technically i can detect if the event is erroneous or not first
+			// the deeper you go, the bigger the deltaDepth is...
 			// this is the difference between what we're watching,
 			// and the event... doesn't mean we can't watch deeper
 			if current == event.Name {
-				delta_depth = 0 // i was watching what i was looking for
+				deltaDepth = 0 // i was watching what i was looking for
 
 			} else if HasPathPrefix(event.Name, current) {
-				delta_depth = len(PathSplit(current)) - len(PathSplit(event.Name)) // -1 or less
+				deltaDepth = len(PathSplit(current)) - len(PathSplit(event.Name)) // -1 or less
 
 			} else if HasPathPrefix(current, event.Name) {
-				delta_depth = len(PathSplit(event.Name)) - len(PathSplit(current)) // +1 or more
+				deltaDepth = len(PathSplit(event.Name)) - len(PathSplit(current)) // +1 or more
 
 			} else {
-				// XXX multiple watchers receive each others events
+				// TODO different watchers get each others events!
 				// https://github.com/go-fsnotify/fsnotify/issues/95
 				// this happened with two values such as:
 				// event.Name: /tmp/mgmt/f3 and current: /tmp/mgmt/f2
-				// are the different watchers getting each others events??
-				//log.Printf("The delta depth is NaN...\n")
-				//log.Printf("Value of event.Name is: %v\n", event.Name)
-				//log.Printf("........    current is: %v\n", current)
-				//log.Fatal("The delta depth is NaN!")
 				continue
 			}
-			//log.Printf("The delta depth is: %v\n", delta_depth)
+			//log.Printf("The delta depth is: %v", deltaDepth)
 
 			// if we have what we wanted, awesome, send an event...
 			if event.Name == safename {
 				//log.Println("Event!")
+				// FIXME: should all these below cases trigger?
 				send = true
+				dirty = true
 
 				// file removed, move the watch upwards
-				if delta_depth >= 0 && (event.Op&fsnotify.Remove == fsnotify.Remove) {
+				if deltaDepth >= 0 && (event.Op&fsnotify.Remove == fsnotify.Remove) {
 					//log.Println("Removal!")
 					watcher.Remove(current)
 					index--
 				}
 
 				// we must be a parent watcher, so descend in
-				if delta_depth < 0 {
+				if deltaDepth < 0 {
 					watcher.Remove(current)
 					index++
 				}
@@ -169,13 +210,18 @@ func (obj FileType) Watch(v *Vertex) {
 			} else if HasPathPrefix(safename, event.Name) {
 				//log.Println("Above!")
 
-				if delta_depth >= 0 && (event.Op&fsnotify.Remove == fsnotify.Remove) {
+				if deltaDepth >= 0 && (event.Op&fsnotify.Remove == fsnotify.Remove) {
 					log.Println("Removal!")
 					watcher.Remove(current)
 					index--
 				}
 
-				if delta_depth < 0 {
+				if deltaDepth < 0 {
+					log.Println("Parent!")
+					if PathPrefixDelta(safename, event.Name) == 1 { // we're the parent dir
+						send = true
+						dirty = true
+					}
 					watcher.Remove(current)
 					index++
 				}
@@ -184,37 +230,44 @@ func (obj FileType) Watch(v *Vertex) {
 			} else if HasPathPrefix(event.Name, safename) {
 				//log.Println("Event2!")
 				send = true
+				dirty = true
 			}
 
 		case err := <-watcher.Errors:
-			log.Println("error:", err)
+			obj.SetConvergedState(resConvergedNil) // XXX ?
+			log.Printf("error: %v", err)
 			log.Fatal(err)
-			v.Events <- fmt.Sprintf("file: %v", "error")
+			//obj.events <- fmt.Sprintf("file: %v", "error") // XXX: how should we handle errors?
 
-		case exit := <-obj.Events:
-			if exit == "exit" {
-				return
-			} else {
-				log.Fatal("Unknown event: %v\n", exit)
+		case event := <-obj.events:
+			obj.SetConvergedState(resConvergedNil)
+			if exit, send = obj.ReadEvent(&event); exit {
+				return // exit
 			}
+			//dirty = false // these events don't invalidate state
+
+		case _ = <-TimeAfterOrBlock(obj.ctimeout):
+			obj.SetConvergedState(resConvergedTimeout)
+			obj.converged <- true
+			continue
 		}
 
 		// do all our event sending all together to avoid duplicate msgs
 		if send {
 			send = false
-			//log.Println("Sending event!")
-			//v.Events <- fmt.Sprintf("file(%v): %v", obj.Path, event.Op)
-			v.Events <- fmt.Sprintf("file(%v): %v", obj.Path, "event!") // FIXME: use struct
+			// only invalid state on certain types of events
+			if dirty {
+				dirty = false
+				obj.isStateOK = false // something made state dirty
+			}
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
 		}
 	}
 }
 
-func (obj FileType) Exit() bool {
-	obj.Events <- "exit"
-	return true
-}
-
-func (obj FileType) HashSHA256fromContent() string {
+func (obj *FileRes) HashSHA256fromContent() string {
 	if obj.sha256sum != "" { // return if already computed
 		return obj.sha256sum
 	}
@@ -225,115 +278,225 @@ func (obj FileType) HashSHA256fromContent() string {
 	return obj.sha256sum
 }
 
-func (obj FileType) StateOK() bool {
-	if _, err := os.Stat(obj.Path); os.IsNotExist(err) {
-		// no such file or directory
-		if obj.State == "absent" {
-			return true // missing file should be missing, phew :)
-		} else {
-			// state invalid, skip expensive checksums
-			return false
+func (obj *FileRes) FileHashSHA256Check() (bool, error) {
+	if PathIsDir(obj.GetPath()) { // assert
+		log.Fatal("This should only be called on a File resource.")
 	}
-	}
-
-	// TODO: add file mode check here...
-
-	if PathIsDir(obj.Path) {
-		return obj.StateOKDir()
-	} else {
-		return obj.StateOKFile()
-	}
-}
-
-func (obj FileType) StateOKFile() bool {
-	if PathIsDir(obj.Path) {
-		log.Fatal("This should only be called on a File type.")
-	}
-
-	// run a diff, and return true if needs changing
-
+	// run a diff, and return true if it needs changing
 	hash := sha256.New()
-
-	f, err := os.Open(obj.Path)
+	f, err := os.Open(obj.GetPath())
 	if err != nil {
-		//log.Fatal(err)
-		return false
+		if e, ok := err.(*os.PathError); ok && (e.Err.(syscall.Errno) == syscall.ENOENT) {
+			return false, nil // no "error", file is just absent
+		}
+		return false, err
 	}
 	defer f.Close()
-
 	if _, err := io.Copy(hash, f); err != nil {
-		//log.Fatal(err)
-		return false
+		return false, err
 	}
-
 	sha256sum := hex.EncodeToString(hash.Sum(nil))
-	//fmt.Printf("sha256sum: %v\n", sha256sum)
-
+	//log.Printf("sha256sum: %v", sha256sum)
 	if obj.HashSHA256fromContent() == sha256sum {
-		return true
+		return true, nil
+	}
+	return false, nil
 }
 
-	return false
-}
-
-func (obj FileType) StateOKDir() bool {
-	if !PathIsDir(obj.Path) {
-		log.Fatal("This should only be called on a Dir type.")
-	}
-
-	// XXX: not implemented
-	log.Fatal("Not implemented!")
-	return false
-}
-
-func (obj FileType) Apply() bool {
-	fmt.Printf("Apply->%v[%v]\n", obj.Type, obj.Name)
-
-	if PathIsDir(obj.Path) {
-		return obj.ApplyDir()
-	} else {
-		return obj.ApplyFile()
-	}
-}
-
-func (obj FileType) ApplyFile() bool {
-
-	if PathIsDir(obj.Path) {
-		log.Fatal("This should only be called on a File type.")
+func (obj *FileRes) FileApply() error {
+	if PathIsDir(obj.GetPath()) {
+		log.Fatal("This should only be called on a File resource.")
 	}
 
 	if obj.State == "absent" {
-		log.Printf("About to remove: %v\n", obj.Path)
-		err := os.Remove(obj.Path)
-		if err != nil {
-			return false
-		}
-		return true
+		log.Printf("About to remove: %v", obj.GetPath())
+		err := os.Remove(obj.GetPath())
+		return err // either nil or not, for success or failure
 	}
 
-	//fmt.Println("writing: " + filename)
-	f, err := os.Create(obj.Path)
+	f, err := os.Create(obj.GetPath())
 	if err != nil {
-		log.Println("error:", err)
-		return false
+		return nil
 	}
 	defer f.Close()
 
 	_, err = io.WriteString(f, obj.Content)
 	if err != nil {
-		log.Println("error:", err)
+		return err
+	}
+
+	return nil // success
+}
+
+func (obj *FileRes) CheckApply(apply bool) (stateok bool, err error) {
+	log.Printf("%v[%v]: CheckApply(%t)", obj.Kind(), obj.GetName(), apply)
+
+	if obj.isStateOK { // cache the state
+		return true, nil
+	}
+
+	if _, err = os.Stat(obj.GetPath()); os.IsNotExist(err) {
+		// no such file or directory
+		if obj.State == "absent" {
+			// missing file should be missing, phew :)
+			obj.isStateOK = true
+			return true, nil
+		}
+	}
+	err = nil // reset
+
+	// FIXME: add file mode check here...
+
+	if PathIsDir(obj.GetPath()) {
+		log.Fatal("Not implemented!") // XXX
+	} else {
+		ok, err := obj.FileHashSHA256Check()
+		if err != nil {
+			return false, err
+		}
+		if ok {
+			obj.isStateOK = true
+			return true, nil
+		}
+		// if no err, but !ok, then we continue on...
+	}
+
+	// state is not okay, no work done, exit, but without error
+	if !apply {
+		return false, nil
+	}
+
+	// apply portion
+	log.Printf("%v[%v]: Apply", obj.Kind(), obj.GetName())
+	if PathIsDir(obj.GetPath()) {
+		log.Fatal("Not implemented!") // XXX
+	} else {
+		err = obj.FileApply()
+		if err != nil {
+			return false, err
+		}
+	}
+
+	obj.isStateOK = true
+	return false, nil // success
+}
+
+type FileUUID struct {
+	BaseUUID
+	path string
+}
+
+// if and only if they are equivalent, return true
+// if they are not equivalent, return false
+func (obj *FileUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*FileUUID)
+	if !ok {
 		return false
 	}
+	return obj.path == res.path
+}
 
+type FileResAutoEdges struct {
+	data    []ResUUID
+	pointer int
+	found   bool
+}
+
+func (obj *FileResAutoEdges) Next() []ResUUID {
+	if obj.found {
+		log.Fatal("Shouldn't be called anymore!")
+	}
+	if len(obj.data) == 0 { // check length for rare scenarios
+		return nil
+	}
+	value := obj.data[obj.pointer]
+	obj.pointer++
+	return []ResUUID{value} // we return one, even though api supports N
+}
+
+// get results of the earlier Next() call, return if we should continue!
+func (obj *FileResAutoEdges) Test(input []bool) bool {
+	// if there aren't any more remaining
+	if len(obj.data) <= obj.pointer {
+		return false
+	}
+	if obj.found { // already found, done!
+		return false
+	}
+	if len(input) != 1 { // in case we get given bad data
+		log.Fatal("Expecting a single value!")
+	}
+	if input[0] { // if a match is found, we're done!
+		obj.found = true // no more to find!
+		return false
+	}
+	return true // keep going
+}
+
+// generate a simple linear sequence of each parent directory from bottom up!
+func (obj *FileRes) AutoEdges() AutoEdge {
+	var data []ResUUID                             // store linear result chain here...
+	values := PathSplitFullReversed(obj.GetPath()) // build it
+	_, values = values[0], values[1:]              // get rid of first value which is me!
+	for _, x := range values {
+		var reversed = true // cheat by passing a pointer
+		data = append(data, &FileUUID{
+			BaseUUID: BaseUUID{
+				name:     obj.GetName(),
+				kind:     obj.Kind(),
+				reversed: &reversed,
+			},
+			path: x, // what matters
+		}) // build list
+	}
+	return &FileResAutoEdges{
+		data:    data,
+		pointer: 0,
+		found:   false,
+	}
+}
+
+func (obj *FileRes) GetUUIDs() []ResUUID {
+	x := &FileUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		path:     obj.GetPath(),
+	}
+	return []ResUUID{x}
+}
+
+func (obj *FileRes) GroupCmp(r Res) bool {
+	_, ok := r.(*FileRes)
+	if !ok {
+		return false
+	}
+	// TODO: we might be able to group directory children into a single
+	// recursive watcher in the future, thus saving fanotify watches
+	return false // not possible atm
+}
+
+func (obj *FileRes) Compare(res Res) bool {
+	switch res.(type) {
+	case *FileRes:
+		res := res.(*FileRes)
+		if obj.Name != res.Name {
+			return false
+		}
+		if obj.GetPath() != res.Path {
+			return false
+		}
+		if obj.Content != res.Content {
+			return false
+		}
+		if obj.State != res.State {
+			return false
+		}
+	default:
+		return false
+	}
 	return true
 }
 
-func (obj FileType) ApplyDir() bool {
-	if !PathIsDir(obj.Path) {
-		log.Fatal("This should only be called on a Dir type.")
-	}
-
-	// XXX: not implemented
-	log.Fatal("Not implemented!")
-	return true
+func (obj *FileRes) CollectPattern(pattern string) {
+	// XXX: currently the pattern for files can only override the Dirname variable :P
+	obj.Dirname = pattern // XXX: simplistic for now
 }

main.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify
@@ -18,7 +18,6 @@
 package main
 
 import (
-	"fmt"
 	"github.com/codegangsta/cli"
 	"log"
 	"os"
@@ -30,8 +29,8 @@ import (
 
 // set at compile time
 var (
-	version string
 	program string
+	version string
 )
 
 const (
@@ -48,7 +47,6 @@ func waitForSignal(exit chan bool) {
 	select {
 	case e := <-signals: // any signal will do
 		if e == os.Interrupt {
-			fmt.Println() // put ^C char from terminal on its own line
 			log.Println("Interrupted by ^C")
 		} else {
 			log.Println("Interrupted by signal")
@@ -59,54 +57,160 @@ func waitForSignal(exit chan bool) {
 }
 
 func run(c *cli.Context) {
-	var start int64 = time.Now().UnixNano()
+	var start = time.Now().UnixNano()
 	var wg sync.WaitGroup
 	exit := make(chan bool)      // exit signal
-	log.Printf("This is: %v, version: %v\n", program, version)
+	converged := make(chan bool) // converged signal
+	log.Printf("This is: %v, version: %v", program, version)
+	log.Printf("Main: Start: %v", start)
+	var G, fullGraph *Graph
 
-	// exit after `exittime` seconds for no reason at all...
-	if i := c.Int("exittime"); i > 0 {
+	// exit after `max-runtime` seconds for no reason at all...
+	if i := c.Int("max-runtime"); i > 0 {
 		go func() {
 			time.Sleep(time.Duration(i) * time.Second)
 			exit <- true
 		}()
 	}
 
-	// build the graph from a config file
-	G := GraphFromConfig(c.String("file"))
-	log.Printf("Graph: %v\n", G) // show graph
+	// initial etcd peer endpoint
+	seed := c.String("seed")
+	if seed == "" {
+		// XXX: start up etcd server, others will join me!
+		seed = "http://127.0.0.1:2379" // thus we use the local server!
+	}
+	// then, connect to `seed` as a client
 
-	log.Printf("Start: %v\n", start)
+	// FIXME: validate seed, or wait for it to fail in etcd init?
 
-	for x := range G.GetVerticesChan() { // XXX ?
-		log.Printf("Main->Starting[%v]\n", x.Name)
-
-		wg.Add(1)
-		// must pass in value to avoid races...
-		// see: https://ttboj.wordpress.com/2015/07/27/golang-parallelism-issues-causing-too-many-open-files-error/
-		go func(v *Vertex) {
-			defer wg.Done()
-			v.Start()
-			log.Printf("Main->Finish[%v]\n", v.Name)
-		}(x)
-
-		// generate a startup "poke" so that an initial check happens
-		go func(v *Vertex) {
-			v.Events <- fmt.Sprintf("Startup(%v)", v.Name)
-		}(x)
+	// etcd
+	etcdO := &EtcdWObject{
+		seed:      seed,
+		ctimeout:  c.Int("converged-timeout"),
+		converged: converged,
 	}
 
-	log.Println("Running...")
+	hostname := c.String("hostname")
+	if hostname == "" {
+		hostname, _ = os.Hostname() // etcd watch key // XXX: this is not the correct key name this is the set key name... WOOPS
+	}
+	go func() {
+		startchan := make(chan struct{}) // start signal
+		go func() { startchan <- struct{}{} }()
+		file := c.String("file")
+		configchan := make(chan bool)
+		if !c.Bool("no-watch") {
+			configchan = ConfigWatch(file)
+		}
+		log.Println("Etcd: Starting...")
+		etcdchan := etcdO.EtcdWatch()
+		first := true // first loop or not
+		for {
+			log.Println("Main: Waiting...")
+			select {
+			case _ = <-startchan: // kick the loop once at start
+				// pass
+			case msg := <-etcdchan:
+				switch msg {
+				// some types of messages we ignore...
+				case etcdFoo, etcdBar:
+					continue
+				// while others passthrough and cause a compile!
+				case etcdStart, etcdEvent:
+					// pass
+				default:
+					log.Fatal("Etcd: Unhandled message: ", msg)
+				}
+			case msg := <-configchan:
+				if c.Bool("no-watch") || !msg {
+					continue // not ready to read config
+				}
+				//case compile_event: XXX
+			}
+
+			config := ParseConfigFromFile(file)
+			if config == nil {
+				log.Printf("Config parse failure")
+				continue
+			}
+
+			// run graph vertex LOCK...
+			if !first { // TODO: we can flatten this check out I think
+				G.Pause() // sync
+			}
+
+			// build graph from yaml file on events (eg: from etcd)
+			// we need the vertices to be paused to work on them
+			if newFullgraph, err := fullGraph.NewGraphFromConfig(config, etcdO, hostname); err == nil { // keep references to all original elements
+				fullGraph = newFullgraph
+			} else {
+				log.Printf("Config: Error making new graph from config: %v", err)
+				// unpause!
+				if !first {
+					G.Start(&wg, first) // sync
+				}
+				continue
+			}
+
+			G = fullGraph.Copy() // copy to active graph
+			// XXX: do etcd transaction out here...
+			G.AutoEdges() // add autoedges; modifies the graph
+			G.AutoGroup() // run autogroup; modifies the graph
+			// TODO: do we want to do a transitive reduction?
+
+			log.Printf("Graph: %v", G) // show graph
+			err := G.ExecGraphviz(c.String("graphviz-filter"), c.String("graphviz"))
+			if err != nil {
+				log.Printf("Graphviz: %v", err)
+			} else {
+				log.Printf("Graphviz: Successfully generated graph!")
+			}
+			G.SetVertex()
+			G.SetConvergedCallback(c.Int("converged-timeout"), converged)
+			// G.Start(...) needs to be synchronous or wait,
+			// because if half of the nodes are started and
+			// some are not ready yet and the EtcdWatch
+			// loops, we'll cause G.Pause(...) before we
+			// even got going, thus causing nil pointer errors
+			G.Start(&wg, first) // sync
+			first = false
+		}
+	}()
+
+	if i := c.Int("converged-timeout"); i >= 0 {
+		go func() {
+		ConvergedLoop:
+			for {
+				<-converged // when anyone says they have converged
+
+				if etcdO.GetConvergedState() != etcdConvergedTimeout {
+					continue
+				}
+				for v := range G.GetVerticesChan() {
+					if v.Res.GetConvergedState() != resConvergedTimeout {
+						continue ConvergedLoop
+					}
+				}
+
+				// if all have converged, exit
+				log.Printf("Converged for %d seconds, exiting!", i)
+				exit <- true
+				for {
+					<-converged
+				} // unblock/drain
+				//return
+			}
+		}()
+	}
+
+	log.Println("Main: Running...")
 
 	waitForSignal(exit) // pass in exit channel to watch
 
 	G.Exit() // tell all the children to exit
 
 	if DEBUG {
-		for i := range G.GetVerticesChan() {
-			fmt.Printf("Vertex: %v\n", i)
-		}
-		fmt.Printf("Graph: %v\n", G)
+		log.Printf("Graph: %v", G)
 	}
 
 	wg.Wait() // wait for primary go routines to exit
@@ -116,6 +220,13 @@ func run(c *cli.Context) {
 }
 
 func main() {
+	//if DEBUG {
+	log.SetFlags(log.LstdFlags | log.Lshortfile)
+	//}
+	log.SetFlags(log.Flags() - log.Ldate) // remove the date for now
+	if program == "" || version == "" {
+		log.Fatal("Program was not compiled correctly. Please see Makefile.")
+	}
 	app := cli.NewApp()
 	app.Name = program
 	app.Usage = "next generation config management"
@@ -134,8 +245,44 @@ func main() {
 					Value: "",
 					Usage: "graph definition to run",
 				},
+				cli.BoolFlag{
+					Name:  "no-watch",
+					Usage: "do not update graph on watched graph definition file changes",
+				},
+				cli.StringFlag{
+					Name:  "code, c",
+					Value: "",
+					Usage: "code definition to run",
+				},
+				cli.StringFlag{
+					Name:  "graphviz, g",
+					Value: "",
+					Usage: "output file for graphviz data",
+				},
+				cli.StringFlag{
+					Name:  "graphviz-filter, gf",
+					Value: "dot", // directed graph default
+					Usage: "graphviz filter to use",
+				},
+				// useful for testing multiple instances on same machine
+				cli.StringFlag{
+					Name:  "hostname",
+					Value: "",
+					Usage: "hostname to use",
+				},
+				// if empty, it will startup a new server
+				cli.StringFlag{
+					Name:  "seed, s",
+					Value: "",
+					Usage: "default etc peer endpoint",
+				},
 				cli.IntFlag{
-					Name:  "exittime",
+					Name:  "converged-timeout, t",
+					Value: -1,
+					Usage: "exit after approximately this many seconds in a converged state",
+				},
+				cli.IntFlag{
+					Name:  "max-runtime",
 					Value: 0,
 					Usage: "exit after a maximum of approximately this many seconds",
 				},

main_test.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify

misc.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify
@@ -18,18 +18,140 @@
 package main
 
 import (
+	"github.com/godbus/dbus"
 	"path"
+	"sort"
 	"strings"
+	"time"
 )
 
+// returns the string with the first character capitalized
+func FirstToUpper(str string) string {
+	return strings.ToUpper(str[0:1]) + str[1:]
+}
+
+// return true if a string exists inside a list, otherwise false
+func StrInList(needle string, haystack []string) bool {
+	for _, x := range haystack {
+		if needle == x {
+			return true
+		}
+	}
+	return false
+}
+
+// remove any duplicate values in the list
+// possibly sub-optimal, O(n^2)? implementation
+func StrRemoveDuplicatesInList(list []string) []string {
+	unique := []string{}
+	for _, x := range list {
+		if !StrInList(x, unique) {
+			unique = append(unique, x)
+		}
+	}
+	return unique
+}
+
+// remove any of the elements in filter, if they exist in list
+func StrFilterElementsInList(filter []string, list []string) []string {
+	result := []string{}
+	for _, x := range list {
+		if !StrInList(x, filter) {
+			result = append(result, x)
+		}
+	}
+	return result
+}
+
+// remove any of the elements in filter, if they don't exist in list
+// this is an in order intersection of two lists
+func StrListIntersection(list1 []string, list2 []string) []string {
+	result := []string{}
+	for _, x := range list1 {
+		if StrInList(x, list2) {
+			result = append(result, x)
+		}
+	}
+	return result
+}
+
+// reverse a list of strings
+func ReverseStringList(in []string) []string {
+	var out []string // empty list
+	l := len(in)
+	for i := range in {
+		out = append(out, in[l-i-1])
+	}
+	return out
+}
+
+// return the sorted list of string keys in a map with string keys
+// NOTE: i thought it would be nice for this to use: map[string]interface{} but
+// it turns out that's not allowed. I know we don't have generics, but common!
+func StrMapKeys(m map[string]string) []string {
+	result := []string{}
+	for k, _ := range m {
+		result = append(result, k)
+	}
+	sort.Strings(result) // deterministic order
+	return result
+}
+
+// return the sorted list of bool values in a map with string values
+func BoolMapValues(m map[string]bool) []bool {
+	result := []bool{}
+	for _, v := range m {
+		result = append(result, v)
+	}
+	//sort.Bools(result) // TODO: deterministic order
+	return result
+}
+
+// return the sorted list of string values in a map with string values
+func StrMapValues(m map[string]string) []string {
+	result := []string{}
+	for _, v := range m {
+		result = append(result, v)
+	}
+	sort.Strings(result) // deterministic order
+	return result
+}
+
+// return true if everyone is true
+func BoolMapTrue(l []bool) bool {
+	for _, b := range l {
+		if !b {
+			return false
+		}
+	}
+	return true
+}
+
 // Similar to the GNU dirname command
 func Dirname(p string) string {
+	if p == "/" {
+		return ""
+	}
 	d, _ := path.Split(path.Clean(p))
 	return d
 }
 
+func Basename(p string) string {
+	_, b := path.Split(path.Clean(p))
+	if p == "" {
+		return ""
+	}
+	if p[len(p)-1:] == "/" { // don't loose the tail slash
+		b += "/"
+	}
+	return b
+}
+
 // Split a path into an array of tokens excluding any trailing empty tokens
 func PathSplit(p string) []string {
+	if p == "/" { // TODO: can't this all be expressed nicely in one line?
+		return []string{""}
+	}
 	return strings.Split(path.Clean(p), "/")
 }
 
@@ -52,6 +174,126 @@ func HasPathPrefix(p, prefix string) bool {
 	return true
 }
 
+func StrInPathPrefixList(needle string, haystack []string) bool {
+	for _, x := range haystack {
+		if HasPathPrefix(x, needle) {
+			return true
+		}
+	}
+	return false
+}
+
+// remove redundant file path prefixes that are under the tree of other files
+func RemoveCommonFilePrefixes(paths []string) []string {
+	var result = make([]string, len(paths))
+	for i := 0; i < len(paths); i++ { // copy, b/c append can modify the args!!
+		result[i] = paths[i]
+	}
+	// is there a string path which is common everywhere?
+	// if so, remove it, and iterate until nothing common is left
+	// return what's left over, that's the most common superset
+loop:
+	for {
+		if len(result) <= 1 {
+			return result
+		}
+		for i := 0; i < len(result); i++ {
+			var copied = make([]string, len(result))
+			for j := 0; j < len(result); j++ { // copy, b/c append can modify the args!!
+				copied[j] = result[j]
+			}
+			noi := append(copied[:i], copied[i+1:]...) // rm i
+			if StrInPathPrefixList(result[i], noi) {
+				// delete the element common to everyone
+				result = noi
+				continue loop
+			}
+		}
+		break
+	}
+	return result
+}
+
+// Delta of path prefix, tells you how many path tokens different the prefix is
+func PathPrefixDelta(p, prefix string) int {
+
+	if !HasPathPrefix(p, prefix) {
+		return -1
+	}
+	patharray := PathSplit(p)
+	prefixarray := PathSplit(prefix)
+	return len(patharray) - len(prefixarray)
+}
+
 func PathIsDir(p string) bool {
 	return p[len(p)-1:] == "/" // a dir has a trailing slash in this context
 }
+
+// return the full list of "dependency" paths for a given path in reverse order
+func PathSplitFullReversed(p string) []string {
+	var result []string
+	split := PathSplit(p)
+	count := len(split)
+	var x string
+	for i := 0; i < count; i++ {
+		x = "/" + path.Join(split[0:i+1]...)
+		if i != 0 && !(i+1 == count && !PathIsDir(p)) {
+			x += "/" // add trailing slash
+		}
+		result = append(result, x)
+	}
+	return ReverseStringList(result)
+}
+
+// add trailing slashes to any likely dirs in a package manager fileList
+// if removeDirs is true, instead, don't keep the dirs in our output
+func DirifyFileList(fileList []string, removeDirs bool) []string {
+	dirs := []string{}
+	for _, file := range fileList {
+		dir, _ := path.Split(file) // dir
+		dir = path.Clean(dir)      // clean so cmp is easier
+		if !StrInList(dir, dirs) {
+			dirs = append(dirs, dir)
+		}
+	}
+
+	result := []string{}
+	for _, file := range fileList {
+		cleanFile := path.Clean(file)
+		if !StrInList(cleanFile, dirs) { // we're not a directory!
+			result = append(result, file) // pass through
+		} else if !removeDirs {
+			result = append(result, cleanFile+"/")
+		}
+	}
+
+	return result
+}
+
+// special version of time.After that blocks when given a negative integer
+// when used in a case statement, the timer restarts on each select call to it
+func TimeAfterOrBlock(t int) <-chan time.Time {
+	if t < 0 {
+		return make(chan time.Time) // blocks forever
+	}
+	return time.After(time.Duration(t) * time.Second)
+}
+
+// making using the private bus usable, should be upstream:
+// TODO: https://github.com/godbus/dbus/issues/15
+func SystemBusPrivateUsable() (conn *dbus.Conn, err error) {
+	conn, err = dbus.SystemBusPrivate()
+	if err != nil {
+		return nil, err
+	}
+	if err = conn.Auth(nil); err != nil {
+		conn.Close()
+		conn = nil
+		return
+	}
+	if err = conn.Hello(); err != nil {
+		conn.Close()
+		conn = nil
+	}
+	return conn, nil // success
+}

misc/bashrc.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+_cli_bash_autocomplete_mgmt() {
+	local cur prev opts base
+	COMPREPLY=()
+	cur="${COMP_WORDS[COMP_CWORD]}"
+	prev="${COMP_WORDS[COMP_CWORD-1]}"
+	opts=$( ${COMP_WORDS[@]:0:$COMP_CWORD} --generate-bash-completion )
+	COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
+	return 0
+}
+
+complete -F _cli_bash_autocomplete_mgmt mgmt

misc/centos-ci.py

@@ -0,0 +1,72 @@
+#!/usr/bin/python
+
+# modified from:
+# https://github.com/kbsingh/centos-ci-scripts/blob/master/build_python_script.py
+# usage: centos-ci.py giturl [branch [commands]]
+
+import os
+import sys
+import json
+import urllib
+import subprocess
+
+# static argv to be used if running script inline
+argv = [
+	#'https://github.com/purpleidea/mgmt', # giturl
+	#'master',
+	#'make test',
+]
+argv.insert(0, '') # add a fake argv[0]
+url_base = 'http://admin.ci.centos.org:8080'
+apikey = '' # put api key here if running inline
+if apikey == '':
+	apikey = os.environ.get('DUFFY_API_KEY')
+if apikey is None or apikey == '':
+	apikey = open('duffy.key', 'r').read().strip()
+ver = '7'
+arch = 'x86_64'
+count = 1
+
+if len(argv) <= 1: argv = sys.argv # use system argv because ours is empty
+if len(argv) <= 1:
+	print 'Not enough arguments supplied!'
+	sys.exit(1)
+
+git_url = argv[1]
+branch = 'master'
+if len(argv) > 2: branch = argv[2]
+folder = os.path.basename(git_url)	# should be project name
+run = 'make vtest' # the omv vtest cmd is a good option to run from this target
+if len(argv) > 3: run = ' '.join(argv[3:])
+
+get_nodes_url = "%s/Node/get?key=%s&ver=%s&arch=%s&i_count=%s" % (url_base, apikey, ver, arch, count)
+data = json.loads(urllib.urlopen(get_nodes_url).read()) # request host(s)
+hosts = data['hosts']
+ssid = data['ssid']
+done_nodes_url = "%s/Node/done?key=%s&ssid=%s" % (url_base, apikey, ssid)
+
+host = hosts[0]
+ssh = "ssh -tt -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o SendEnv=JENKINS_URL root@%s" % host
+yum = 'yum -y install git wget tree psmisc'
+omv = 'wget https://github.com/purpleidea/oh-my-vagrant/raw/master/extras/install-omv.sh && chmod u+x install-omv.sh && ./install-omv.sh && wget https://github.com/purpleidea/mgmt/raw/master/misc/make-path.sh && chmod u+x make-path.sh && ./make-path.sh'
+cmd = "%s '%s && %s'" % (ssh, yum, omv) # setup
+print cmd
+r = subprocess.call(cmd, shell=True)
+if r != 0:
+	# NOTE: we don't clean up the host here, so that it can be inspected!
+	print "Error configuring omv on: %s" % host
+	sys.exit(r)
+
+# the second ssh call will run with the omv /etc/profile.d/ script loaded
+git = "git clone --recursive %s %s && cd %s && git checkout %s" % (git_url, folder, folder, branch)
+cmd = "%s 'export JENKINS_URL=%s && %s && %s'" % (ssh, os.getenv('JENKINS_URL', ''), git, run) # run
+print cmd
+r = subprocess.call(cmd, shell=True)
+if r != 0:
+	print "Error running job on: %s" % host
+
+output = urllib.urlopen(done_nodes_url).read() # free host(s)
+if output != 'Done':
+	print "Error freeing host: %s" % host
+
+sys.exit(r)

misc/copr-build.py

@@ -0,0 +1,66 @@
+#!/usr/bin/python
+
+# README:
+# for initial setup, browse to: https://copr.fedoraproject.org/api/
+# and it will have a ~/.config/copr config that you can download.
+# happy hacking!
+
+import os
+import sys
+import copr
+import time
+
+COPR = 'mgmt'
+
+if len(sys.argv) != 2:
+	print("Usage: %s <srpm url>" % sys.argv[0])
+	sys.exit(1)
+
+url = sys.argv[1]
+
+client = copr.CoprClient.create_from_file_config(os.path.expanduser("~/.config/copr"))
+
+result = client.create_new_build(COPR, [url])
+if result.output != 'ok':
+	print(result.error)
+	sys.exit(1)
+print(result.message)
+
+# modified from: https://python-copr.readthedocs.org/en/latest/Examples.html#work-with-builds
+for bw in result.builds_list:
+	print("Build #{}: {}".format(bw.build_id, bw.handle.get_build_details().status))
+
+# cancel all created build
+#for bw in result.builds_list:
+#	bw.handle.cancel_build()
+
+# get build status for each chroot
+#for bw in result.builds_list:
+#	print("build: {}".format(bw.build_id))
+#	for ch, status in bw.handle.get_build_details().data["chroots"].items():
+#		print("\t chroot {}:\t {}".format(ch, status))
+
+# simple build progress:
+
+watched = set(result.builds_list)
+done = set()
+state = {}
+for bw in watched:	# store initial states
+	state[bw.build_id] = bw.handle.get_build_details().status
+
+while watched != done:
+	for bw in watched:
+		if bw in done:
+			continue
+		status = bw.handle.get_build_details().status
+		if status != state.get(bw.build_id):
+			print("Build #{}: {}".format(bw.build_id, status))
+			state[bw.build_id] = status	# update status
+
+		if status in ['skipped', 'failed', 'succeeded']:
+			done.add(bw)
+
+	if watched == done: break	# avoid long while sleep
+	else: time.sleep(10)
+
+print 'Done!'

misc/example.conf

@@ -0,0 +1 @@
+# example mgmt configuration file, currently has no options at the moment!

misc/make-deps.sh

@@ -0,0 +1,51 @@
+#!/bin/bash
+# setup a simple go environment
+XPWD=`pwd`
+ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && cd .. && pwd )"	# dir!
+cd "${ROOT}" >/dev/null
+
+travis=0
+if env | grep -q '^TRAVIS=true$'; then
+	travis=1
+fi
+
+if [ $travis -eq 0 ]; then
+	YUM=`which yum 2>/dev/null`
+	APT=`which apt-get 2>/dev/null`
+	if [ -z "$YUM" -a -z "$APT" ]; then
+		echo "The package managers can't be found."
+		exit 1
+	fi
+	if [ ! -z "$YUM" ]; then
+		# some go dependencies are stored in mercurial
+		sudo $YUM install -y golang golang-googlecode-tools-stringer hg
+
+	fi
+	if [ ! -z "$APT" ]; then
+		sudo $APT update
+		sudo $APT install -y golang make gcc packagekit mercurial
+		# one of these two golang tools packages should work on debian
+		sudo $APT install -y golang-golang-x-tools || true
+		sudo $APT install -y golang-go.tools || true
+	fi
+fi
+
+# build etcd
+git clone --recursive https://github.com/coreos/etcd/ && cd etcd
+goversion=$(go version)
+# if 'go version' contains string 'devel', then use git master of etcd...
+if [ "${goversion#*devel}" == "$goversion" ]; then
+	git checkout v2.2.4	# TODO: update to newer versions as needed
+fi
+[ -x build ] && ./build
+mkdir -p ~/bin/
+cp bin/etcd ~/bin/
+cd - >/dev/null
+rm -rf etcd	# clean up to avoid failing on upstream gofmt errors
+
+go get ./...	# get all the go dependencies
+[ -e "$GOBIN/mgmt" ] && rm -f "$GOBIN/mgmt"	# the `go get` version has no -X
+go get golang.org/x/tools/cmd/vet	# add in `go vet` for travis
+go get golang.org/x/tools/cmd/stringer	# for automatic stringer-ing
+go get github.com/golang/lint/golint	# for `golint`-ing
+cd "$XPWD" >/dev/null

misc/make-path.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+# setup a few environment path values
+
+if ! env | grep -q '^GOPATH='; then
+	export GOPATH="$HOME/gopath/"
+	mkdir "$GOPATH"
+	if ! grep -q '^export GOPATH=' ~/.bashrc; then
+	        echo "export GOPATH=~/gopath/" >> ~/.bashrc
+	fi
+	echo "setting go path to: $GOPATH"
+fi
+
+echo "gopath is: $GOPATH"
+
+# some versions of golang apparently require this to run go get :(
+if ! env | grep -q '^GOBIN='; then
+	export GOBIN="${GOPATH}bin/"
+	mkdir "$GOBIN"
+	if ! grep -q '^export GOBIN=' ~/.bashrc; then
+	        echo 'export GOBIN="${GOPATH}bin/"' >> ~/.bashrc
+	fi
+	echo "setting go bin to: $GOBIN"
+fi
+
+echo "gobin is: $GOBIN"
+
+# add gobin to $PATH
+if ! env | grep '^PATH=' | grep -q "$GOBIN"; then
+	if ! grep -q '^export PATH="'"${GOBIN}"':${PATH}"' ~/.bashrc; then
+	        echo 'export PATH="'"${GOBIN}"':${PATH}"' >> ~/.bashrc
+	fi
+	export PATH="${GOBIN}:${PATH}"
+	echo "setting path to: $PATH"
+fi
+
+echo "path is: $PATH"
+
+# add ~/bin/ to $PATH
+if ! env | grep '^PATH=' | grep -q "$HOME/bin"; then
+	mkdir -p "${HOME}/bin"
+	if ! grep -q '^export PATH="'"${HOME}/bin"':${PATH}"' ~/.bashrc; then
+	        echo 'export PATH="'"${HOME}/bin"':${PATH}"' >> ~/.bashrc
+	fi
+	export PATH="${HOME}/bin:${PATH}"
+	echo "setting path to: $PATH"
+fi
+
+echo "path is: $PATH"

misc_test.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify
@@ -18,6 +18,8 @@
 package main
 
 import (
+	"reflect"
+	"sort"
 	"testing"
 )
 
@@ -31,7 +33,31 @@ func TestMiscT1(t *testing.T) {
 		t.Errorf("Result is incorrect.")
 	}
 
-	if Dirname("/") != "/" {
+	if Dirname("/foo/") != "/" {
+		t.Errorf("Result is incorrect.")
+	}
+
+	if Dirname("/") != "" { // TODO: should this equal "/" or "" ?
+		t.Errorf("Result is incorrect.")
+	}
+
+	if Basename("/foo/bar/baz") != "baz" {
+		t.Errorf("Result is incorrect.")
+	}
+
+	if Basename("/foo/bar/baz/") != "baz/" {
+		t.Errorf("Result is incorrect.")
+	}
+
+	if Basename("/foo/") != "foo/" {
+		t.Errorf("Result is incorrect.")
+	}
+
+	if Basename("/") != "/" { // TODO: should this equal "" or "/" ?
+		t.Errorf("Result is incorrect.")
+	}
+
+	if Basename("") != "" { // TODO: should this equal something different?
 		t.Errorf("Result is incorrect.")
 	}
 }
@@ -39,6 +65,13 @@ func TestMiscT1(t *testing.T) {
 func TestMiscT2(t *testing.T) {
 
 	// TODO: compare the output with the actual list
+	p0 := "/"
+	r0 := []string{""} // TODO: is this correct?
+	if len(PathSplit(p0)) != len(r0) {
+		t.Errorf("Result should be: %q.", r0)
+		t.Errorf("Result should have a length of: %v.", len(r0))
+	}
+
 	p1 := "/foo/bar/baz"
 	r1 := []string{"", "foo", "bar", "baz"}
 	if len(PathSplit(p1)) != len(r1) {
@@ -78,10 +111,49 @@ func TestMiscT3(t *testing.T) {
 	if HasPathPrefix("/foo/bar/baz/", "/foo/bar/baz/dude") != false {
 		t.Errorf("Result should be false.")
 	}
+
+	if HasPathPrefix("/foo/bar/baz/boo/", "/foo/") != true {
+		t.Errorf("Result should be true.")
+	}
 }
 
 func TestMiscT4(t *testing.T) {
 
+	if PathPrefixDelta("/foo/bar/baz", "/foo/ba") != -1 {
+		t.Errorf("Result should be -1.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz", "/foo/bar") != 1 {
+		t.Errorf("Result should be 1.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz", "/foo/bar/") != 1 {
+		t.Errorf("Result should be 1.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz/", "/foo/bar") != 1 {
+		t.Errorf("Result should be 1.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz/", "/foo/bar/") != 1 {
+		t.Errorf("Result should be 1.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz/", "/foo/bar/baz/dude") != -1 {
+		t.Errorf("Result should be -1.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz/a/b/c/", "/foo/bar/baz") != 3 {
+		t.Errorf("Result should be 3.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz/", "/foo/bar/baz") != 0 {
+		t.Errorf("Result should be 0.")
+	}
+}
+
+func TestMiscT5(t *testing.T) {
+
 	if PathIsDir("/foo/bar/baz/") != true {
 		t.Errorf("Result should be false.")
 	}
@@ -97,5 +169,576 @@ func TestMiscT4(t *testing.T) {
 	if PathIsDir("/") != true {
 		t.Errorf("Result should be true.")
 	}
+}
+
+func TestMiscT8(t *testing.T) {
+
+	r0 := []string{"/"}
+	if fullList0 := PathSplitFullReversed("/"); !reflect.DeepEqual(r0, fullList0) {
+		t.Errorf("PathSplitFullReversed expected: %v; got: %v.", r0, fullList0)
+	}
+
+	r1 := []string{"/foo/bar/baz/file", "/foo/bar/baz/", "/foo/bar/", "/foo/", "/"}
+	if fullList1 := PathSplitFullReversed("/foo/bar/baz/file"); !reflect.DeepEqual(r1, fullList1) {
+		t.Errorf("PathSplitFullReversed expected: %v; got: %v.", r1, fullList1)
+	}
+
+	r2 := []string{"/foo/bar/baz/dir/", "/foo/bar/baz/", "/foo/bar/", "/foo/", "/"}
+	if fullList2 := PathSplitFullReversed("/foo/bar/baz/dir/"); !reflect.DeepEqual(r2, fullList2) {
+		t.Errorf("PathSplitFullReversed expected: %v; got: %v.", r2, fullList2)
+	}
 
 }
+
+func TestMiscT9(t *testing.T) {
+	fileListIn := []string{ // list taken from drbd-utils package
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d",
+		"/usr/share/doc/drbd-utils",
+		"/var/lib/drbd",
+	}
+	sort.Strings(fileListIn)
+
+	fileListOut := []string{ // fixed up manually
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d/",               // added trailing slash
+		"/usr/share/doc/drbd-utils/", // added trailing slash
+		"/var/lib/drbd",              // can't be fixed :(
+	}
+	sort.Strings(fileListOut)
+
+	dirify := DirifyFileList(fileListIn, false) // TODO: test with true
+	sort.Strings(dirify)
+	equals := reflect.DeepEqual(fileListOut, dirify)
+	if a, b := len(fileListOut), len(dirify); a != b {
+		t.Errorf("DirifyFileList counts didn't match: %d != %d", a, b)
+	} else if !equals {
+		t.Error("DirifyFileList did not match expected!")
+		for i := 0; i < len(dirify); i++ {
+			if fileListOut[i] != dirify[i] {
+				t.Errorf("# %d: %v <> %v", i, fileListOut[i], dirify[i])
+			}
+		}
+	}
+}
+
+func TestMiscT10(t *testing.T) {
+	fileListIn := []string{ // fake package list
+		"/etc/drbd.conf",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d",
+		"/etc/drbd.d/foo",
+		"/var/lib/drbd",
+		"/var/somedir/",
+	}
+	sort.Strings(fileListIn)
+
+	fileListOut := []string{ // fixed up manually
+		"/etc/drbd.conf",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d/", // added trailing slash
+		"/etc/drbd.d/foo",
+		"/var/lib/drbd", // can't be fixed :(
+		"/var/somedir/", // stays the same
+	}
+	sort.Strings(fileListOut)
+
+	dirify := DirifyFileList(fileListIn, false) // TODO: test with true
+	sort.Strings(dirify)
+	equals := reflect.DeepEqual(fileListOut, dirify)
+	if a, b := len(fileListOut), len(dirify); a != b {
+		t.Errorf("DirifyFileList counts didn't match: %d != %d", a, b)
+	} else if !equals {
+		t.Error("DirifyFileList did not match expected!")
+		for i := 0; i < len(dirify); i++ {
+			if fileListOut[i] != dirify[i] {
+				t.Errorf("# %d: %v <> %v", i, fileListOut[i], dirify[i])
+			}
+		}
+	}
+}
+
+func TestMiscT11(t *testing.T) {
+	in1 := []string{"/", "/usr/", "/usr/lib/", "/usr/share/"} // input
+	ex1 := []string{"/usr/lib/", "/usr/share/"}               // expected
+	sort.Strings(ex1)
+	out1 := RemoveCommonFilePrefixes(in1)
+	sort.Strings(out1)
+	if !reflect.DeepEqual(ex1, out1) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex1, out1)
+	}
+
+	in2 := []string{"/", "/usr/"}
+	ex2 := []string{"/usr/"}
+	sort.Strings(ex2)
+	out2 := RemoveCommonFilePrefixes(in2)
+	sort.Strings(out2)
+	if !reflect.DeepEqual(ex2, out2) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex2, out2)
+	}
+
+	in3 := []string{"/"}
+	ex3 := []string{"/"}
+	out3 := RemoveCommonFilePrefixes(in3)
+	if !reflect.DeepEqual(ex3, out3) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex3, out3)
+	}
+
+	in4 := []string{"/usr/bin/foo", "/usr/bin/bar", "/usr/lib/", "/usr/share/"}
+	ex4 := []string{"/usr/bin/foo", "/usr/bin/bar", "/usr/lib/", "/usr/share/"}
+	sort.Strings(ex4)
+	out4 := RemoveCommonFilePrefixes(in4)
+	sort.Strings(out4)
+	if !reflect.DeepEqual(ex4, out4) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex4, out4)
+	}
+
+	in5 := []string{"/usr/bin/foo", "/usr/bin/bar", "/usr/lib/", "/usr/share/", "/usr/bin"}
+	ex5 := []string{"/usr/bin/foo", "/usr/bin/bar", "/usr/lib/", "/usr/share/"}
+	sort.Strings(ex5)
+	out5 := RemoveCommonFilePrefixes(in5)
+	sort.Strings(out5)
+	if !reflect.DeepEqual(ex5, out5) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex5, out5)
+	}
+
+	in6 := []string{"/etc/drbd.d/", "/lib/drbd/", "/usr/lib/drbd/", "/usr/lib/systemd/system/", "/usr/lib/tmpfiles.d/", "/usr/sbin/", "/usr/share/doc/drbd-utils/", "/usr/share/man/man5/", "/usr/share/man/man8/", "/usr/share/doc/", "/var/lib/"}
+	ex6 := []string{"/etc/drbd.d/", "/lib/drbd/", "/usr/lib/drbd/", "/usr/lib/systemd/system/", "/usr/lib/tmpfiles.d/", "/usr/sbin/", "/usr/share/doc/drbd-utils/", "/usr/share/man/man5/", "/usr/share/man/man8/", "/var/lib/"}
+	sort.Strings(ex6)
+	out6 := RemoveCommonFilePrefixes(in6)
+	sort.Strings(out6)
+	if !reflect.DeepEqual(ex6, out6) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex6, out6)
+	}
+
+	in7 := []string{"/etc/", "/lib/", "/usr/lib/", "/usr/lib/systemd/", "/usr/", "/usr/share/doc/", "/usr/share/man/", "/var/"}
+	ex7 := []string{"/etc/", "/lib/", "/usr/lib/systemd/", "/usr/share/doc/", "/usr/share/man/", "/var/"}
+	sort.Strings(ex7)
+	out7 := RemoveCommonFilePrefixes(in7)
+	sort.Strings(out7)
+	if !reflect.DeepEqual(ex7, out7) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex7, out7)
+	}
+
+	in8 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d/",
+		"/usr/share/doc/drbd-utils/",
+		"/var/lib/drbd",
+	}
+	ex8 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/var/lib/drbd",
+	}
+	sort.Strings(ex8)
+	out8 := RemoveCommonFilePrefixes(in8)
+	sort.Strings(out8)
+	if !reflect.DeepEqual(ex8, out8) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex8, out8)
+	}
+
+	in9 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/",
+		"/lib/drbd/drbd",
+		"/lib/drbd/",
+		"/lib/drbd/",
+		"/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/systemd/system/",
+		"/usr/lib/tmpfiles.d/",
+		"/usr/sbin/",
+		"/usr/sbin/",
+		"/usr/share/doc/drbd-utils/",
+		"/usr/share/doc/drbd-utils/",
+		"/usr/share/man/man5/",
+		"/usr/share/man/man5/",
+		"/usr/share/man/man8/",
+		"/usr/share/man/man8/",
+		"/usr/share/man/man8/",
+		"/etc/drbd.d/",
+		"/usr/share/doc/drbd-utils/",
+		"/var/lib/drbd",
+	}
+	ex9 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/",
+		"/lib/drbd/drbd",
+		"/usr/lib/drbd/",
+		"/usr/lib/systemd/system/",
+		"/usr/lib/tmpfiles.d/",
+		"/usr/sbin/",
+		"/usr/share/doc/drbd-utils/",
+		"/usr/share/man/man5/",
+		"/usr/share/man/man8/",
+		"/var/lib/drbd",
+	}
+	sort.Strings(ex9)
+	out9 := RemoveCommonFilePrefixes(in9)
+	sort.Strings(out9)
+	if !reflect.DeepEqual(ex9, out9) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex9, out9)
+	}
+
+	in10 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/",                   // watch me, i'm a dir
+		"/etc/drbd.d/global_common.conf", // and watch me i'm a file!
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/", // watch me, i'm a dir too
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/var/lib/drbd",
+	}
+	ex10 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/var/lib/drbd",
+	}
+	sort.Strings(ex10)
+	out10 := RemoveCommonFilePrefixes(in10)
+	sort.Strings(out10)
+	if !reflect.DeepEqual(ex10, out10) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex10, out10)
+		for i := 0; i < len(ex10); i++ {
+			if ex10[i] != out10[i] {
+				t.Errorf("# %d: %v <> %v", i, ex10[i], out10[i])
+			}
+		}
+	}
+}

noop.go

@@ -0,0 +1,143 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"encoding/gob"
+	"log"
+)
+
+func init() {
+	gob.Register(&NoopRes{})
+}
+
+type NoopRes struct {
+	BaseRes `yaml:",inline"`
+	Comment string `yaml:"comment"` // extra field for example purposes
+}
+
+func NewNoopRes(name string) *NoopRes {
+	obj := &NoopRes{
+		BaseRes: BaseRes{
+			Name: name,
+		},
+		Comment: "",
+	}
+	obj.Init()
+	return obj
+}
+
+func (obj *NoopRes) Init() {
+	obj.BaseRes.kind = "Noop"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+}
+
+// validate if the params passed in are valid data
+// FIXME: where should this get called ?
+func (obj *NoopRes) Validate() bool {
+	return true
+}
+
+func (obj *NoopRes) Watch(processChan chan Event) {
+	if obj.IsWatching() {
+		return
+	}
+	obj.SetWatching(true)
+	defer obj.SetWatching(false)
+
+	//vertex := obj.vertex // stored with SetVertex
+	var send = false // send event?
+	var exit = false
+	for {
+		obj.SetState(resStateWatching) // reset
+		select {
+		case event := <-obj.events:
+			obj.SetConvergedState(resConvergedNil)
+			// we avoid sending events on unpause
+			if exit, send = obj.ReadEvent(&event); exit {
+				return // exit
+			}
+
+		case _ = <-TimeAfterOrBlock(obj.ctimeout):
+			obj.SetConvergedState(resConvergedTimeout)
+			obj.converged <- true
+			continue
+		}
+
+		// do all our event sending all together to avoid duplicate msgs
+		if send {
+			send = false
+			// only do this on certain types of events
+			//obj.isStateOK = false // something made state dirty
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
+		}
+	}
+}
+
+// CheckApply method for Noop resource. Does nothing, returns happy!
+func (obj *NoopRes) CheckApply(apply bool) (stateok bool, err error) {
+	log.Printf("%v[%v]: CheckApply(%t)", obj.Kind(), obj.GetName(), apply)
+	return true, nil // state is always okay
+}
+
+type NoopUUID struct {
+	BaseUUID
+	name string
+}
+
+func (obj *NoopRes) AutoEdges() AutoEdge {
+	return nil
+}
+
+// include all params to make a unique identification of this object
+// most resources only return one, although some resources return multiple
+func (obj *NoopRes) GetUUIDs() []ResUUID {
+	x := &NoopUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		name:     obj.Name,
+	}
+	return []ResUUID{x}
+}
+
+func (obj *NoopRes) GroupCmp(r Res) bool {
+	_, ok := r.(*NoopRes)
+	if !ok {
+		// NOTE: technically we could group a noop into any other
+		// resource, if that resource knew how to handle it, although,
+		// since the mechanics of inter-kind resource grouping are
+		// tricky, avoid doing this until there's a good reason.
+		return false
+	}
+	return true // noop resources can always be grouped together!
+}
+
+func (obj *NoopRes) Compare(res Res) bool {
+	switch res.(type) {
+	// we can only compare NoopRes to others of the same resource
+	case *NoopRes:
+		res := res.(*NoopRes)
+		if obj.Name != res.Name {
+			return false
+		}
+	default:
+		return false
+	}
+	return true
+}

omv.yaml

@@ -0,0 +1,39 @@
+---
+:domain: example.com
+:network: 192.168.123.0/24
+:image: fedora-23
+:cpus: ''
+:memory: ''
+:disks: 0
+:disksize: 40G
+:boxurlprefix: ''
+:sync: rsync
+:syncdir: mgmt/
+:syncsrc: "../"
+:folder: ".omv"
+:extern: []
+:cd: "-"
+:puppet: false
+:classes: []
+:shell:
+- cd /vagrant/mgmt/ && make deps
+:docker: false
+:kubernetes: false
+:ansible: []
+:playbook: []
+:ansible_extras: {}
+:cachier: false
+:vms: []
+:namespace: omv
+:count: 1
+:username: ''
+:password: ''
+:poolid: true
+:repos: []
+:update: false
+:reboot: false
+:unsafe: false
+:nested: false
+:tests: []
+:comment: ''
+:reallyrm: false

packagekit.go

@@ -0,0 +1,912 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+// DOCS: https://www.freedesktop.org/software/PackageKit/gtk-doc/index.html
+
+//package packagekit // TODO
+package main
+
+import (
+	"fmt"
+	"github.com/godbus/dbus"
+	"log"
+	"runtime"
+	"strings"
+)
+
+const (
+	PK_DEBUG = false
+	PARANOID = false // enable if you see any ghosts
+)
+
+const (
+	// FIXME: if PkBufferSize is too low, install seems to drop signals
+	PkBufferSize = 1000
+	// TODO: the PkSignalTimeout value might be too low
+	PkSignalPackageTimeout = 60 // 60 seconds, arbitrary
+	PkSignalDestroyTimeout = 15 // 15 seconds, arbitrary
+	PkPath                 = "/org/freedesktop/PackageKit"
+	PkIface                = "org.freedesktop.PackageKit"
+	PkIfaceTransaction     = PkIface + ".Transaction"
+	dbusAddMatch           = "org.freedesktop.DBus.AddMatch"
+)
+
+var (
+	// GOARCH's: 386, amd64, arm, arm64, mips64, mips64le, ppc64, ppc64le
+	PkArchMap = map[string]string{ // map of PackageKit arch to GOARCH
+		// TODO: add more values
+		// noarch
+		"noarch": "ANY", // special value "ANY"
+		// fedora
+		"x86_64":  "amd64",
+		"aarch64": "arm64",
+		// debian, from: https://www.debian.org/ports/
+		"amd64": "amd64",
+		"arm64": "arm64",
+		"i386":  "386",
+		"i486":  "386",
+		"i586":  "386",
+		"i686":  "386",
+	}
+)
+
+//type enum_filter uint64
+// https://github.com/hughsie/PackageKit/blob/master/lib/packagekit-glib2/pk-enum.c
+const ( //static const PkEnumMatch enum_filter[]
+	PK_FILTER_ENUM_UNKNOWN         uint64 = 1 << iota // "unknown"
+	PK_FILTER_ENUM_NONE                               // "none"
+	PK_FILTER_ENUM_INSTALLED                          // "installed"
+	PK_FILTER_ENUM_NOT_INSTALLED                      // "~installed"
+	PK_FILTER_ENUM_DEVELOPMENT                        // "devel"
+	PK_FILTER_ENUM_NOT_DEVELOPMENT                    // "~devel"
+	PK_FILTER_ENUM_GUI                                // "gui"
+	PK_FILTER_ENUM_NOT_GUI                            // "~gui"
+	PK_FILTER_ENUM_FREE                               // "free"
+	PK_FILTER_ENUM_NOT_FREE                           // "~free"
+	PK_FILTER_ENUM_VISIBLE                            // "visible"
+	PK_FILTER_ENUM_NOT_VISIBLE                        // "~visible"
+	PK_FILTER_ENUM_SUPPORTED                          // "supported"
+	PK_FILTER_ENUM_NOT_SUPPORTED                      // "~supported"
+	PK_FILTER_ENUM_BASENAME                           // "basename"
+	PK_FILTER_ENUM_NOT_BASENAME                       // "~basename"
+	PK_FILTER_ENUM_NEWEST                             // "newest"
+	PK_FILTER_ENUM_NOT_NEWEST                         // "~newest"
+	PK_FILTER_ENUM_ARCH                               // "arch"
+	PK_FILTER_ENUM_NOT_ARCH                           // "~arch"
+	PK_FILTER_ENUM_SOURCE                             // "source"
+	PK_FILTER_ENUM_NOT_SOURCE                         // "~source"
+	PK_FILTER_ENUM_COLLECTIONS                        // "collections"
+	PK_FILTER_ENUM_NOT_COLLECTIONS                    // "~collections"
+	PK_FILTER_ENUM_APPLICATION                        // "application"
+	PK_FILTER_ENUM_NOT_APPLICATION                    // "~application"
+	PK_FILTER_ENUM_DOWNLOADED                         // "downloaded"
+	PK_FILTER_ENUM_NOT_DOWNLOADED                     // "~downloaded"
+)
+
+const ( //static const PkEnumMatch enum_transaction_flag[]
+	PK_TRANSACTION_FLAG_ENUM_NONE            uint64 = 1 << iota // "none"
+	PK_TRANSACTION_FLAG_ENUM_ONLY_TRUSTED                       // "only-trusted"
+	PK_TRANSACTION_FLAG_ENUM_SIMULATE                           // "simulate"
+	PK_TRANSACTION_FLAG_ENUM_ONLY_DOWNLOAD                      // "only-download"
+	PK_TRANSACTION_FLAG_ENUM_ALLOW_REINSTALL                    // "allow-reinstall"
+	PK_TRANSACTION_FLAG_ENUM_JUST_REINSTALL                     // "just-reinstall"
+	PK_TRANSACTION_FLAG_ENUM_ALLOW_DOWNGRADE                    // "allow-downgrade"
+)
+
+const ( //typedef enum
+	PK_INFO_ENUM_UNKNOWN uint64 = 1 << iota
+	PK_INFO_ENUM_INSTALLED
+	PK_INFO_ENUM_AVAILABLE
+	PK_INFO_ENUM_LOW
+	PK_INFO_ENUM_ENHANCEMENT
+	PK_INFO_ENUM_NORMAL
+	PK_INFO_ENUM_BUGFIX
+	PK_INFO_ENUM_IMPORTANT
+	PK_INFO_ENUM_SECURITY
+	PK_INFO_ENUM_BLOCKED
+	PK_INFO_ENUM_DOWNLOADING
+	PK_INFO_ENUM_UPDATING
+	PK_INFO_ENUM_INSTALLING
+	PK_INFO_ENUM_REMOVING
+	PK_INFO_ENUM_CLEANUP
+	PK_INFO_ENUM_OBSOLETING
+	PK_INFO_ENUM_COLLECTION_INSTALLED
+	PK_INFO_ENUM_COLLECTION_AVAILABLE
+	PK_INFO_ENUM_FINISHED
+	PK_INFO_ENUM_REINSTALLING
+	PK_INFO_ENUM_DOWNGRADING
+	PK_INFO_ENUM_PREPARING
+	PK_INFO_ENUM_DECOMPRESSING
+	PK_INFO_ENUM_UNTRUSTED
+	PK_INFO_ENUM_TRUSTED
+	PK_INFO_ENUM_UNAVAILABLE
+	PK_INFO_ENUM_LAST
+)
+
+// wrapper struct so we can pass bus connection around in the struct
+type Conn struct {
+	conn *dbus.Conn
+}
+
+// struct that is returned by PackagesToPackageIDs in the map values
+type PkPackageIDActionData struct {
+	Found     bool
+	Installed bool
+	Version   string
+	PackageID string
+	Newest    bool
+}
+
+// get a new bus connection
+func NewBus() *Conn {
+	// if we share the bus with others, we will get each others messages!!
+	bus, err := SystemBusPrivateUsable() // don't share the bus connection!
+	if err != nil {
+		return nil
+	}
+	return &Conn{
+		conn: bus,
+	}
+}
+
+// get the dbus connection object
+func (bus *Conn) GetBus() *dbus.Conn {
+	return bus.conn
+}
+
+// close the dbus connection object
+func (bus *Conn) Close() error {
+	return bus.conn.Close()
+}
+
+// internal helper to add signal matches to the bus, should only be called once
+func (bus *Conn) matchSignal(ch chan *dbus.Signal, path dbus.ObjectPath, iface string, signals []string) error {
+	if PK_DEBUG {
+		log.Printf("PackageKit: matchSignal(%v, %v, %v, %v)", ch, path, iface, signals)
+	}
+	// eg: gdbus monitor --system --dest org.freedesktop.PackageKit --object-path /org/freedesktop/PackageKit | grep <signal>
+	var call *dbus.Call
+	// TODO: if we make this call many times, we seem to receive signals
+	// that many times... Maybe this should be an object singleton?
+	obj := bus.GetBus().BusObject()
+	pathStr := fmt.Sprintf("%s", path)
+	if len(signals) == 0 {
+		call = obj.Call(dbusAddMatch, 0, "type='signal',path='"+pathStr+"',interface='"+iface+"'")
+	} else {
+		for _, signal := range signals {
+			call = obj.Call(dbusAddMatch, 0, "type='signal',path='"+pathStr+"',interface='"+iface+"',member='"+signal+"'")
+			if call.Err != nil {
+				break
+			}
+		}
+	}
+	if call.Err != nil {
+		return call.Err
+	}
+	// The caller has to make sure that ch is sufficiently buffered; if a
+	// message arrives when a write to c is not possible, it is discarded!
+	// This can be disastrous if we're waiting for a "Finished" signal!
+	bus.GetBus().Signal(ch)
+	return nil
+}
+
+// get a signal anytime an event happens
+func (bus *Conn) WatchChanges() (chan *dbus.Signal, error) {
+	ch := make(chan *dbus.Signal, PkBufferSize)
+	// NOTE: the TransactionListChanged signal fires much more frequently,
+	// but with much less specificity. If we're missing events, report the
+	// issue upstream! The UpdatesChanged signal is what hughsie suggested
+	var signal = "UpdatesChanged"
+	err := bus.matchSignal(ch, PkPath, PkIface, []string{signal})
+	if err != nil {
+		return nil, err
+	}
+	if PARANOID { // TODO: this filtering might not be necessary anymore...
+		// try to handle the filtering inside this function!
+		rch := make(chan *dbus.Signal)
+		go func() {
+		loop:
+			for {
+				select {
+				case event := <-ch:
+					// "A receive from a closed channel returns the
+					// zero value immediately": if i get nil here,
+					// it means the channel was closed by someone!!
+					if event == nil { // shared bus issue?
+						log.Println("PackageKit: Hrm, channel was closed!")
+						break loop // TODO: continue?
+					}
+					// i think this was caused by using the shared
+					// bus, but we might as well leave it in for now
+					if event.Path != PkPath || event.Name != fmt.Sprintf("%s.%s", PkIface, signal) {
+						log.Printf("PackageKit: Woops: Event: %+v", event)
+						continue
+					}
+					rch <- event // forward...
+				}
+			}
+			defer close(ch)
+		}()
+		return rch, nil
+	}
+	return ch, nil
+}
+
+// create and return a transaction path
+func (bus *Conn) CreateTransaction() (dbus.ObjectPath, error) {
+	if PK_DEBUG {
+		log.Println("PackageKit: CreateTransaction()")
+	}
+	var interfacePath dbus.ObjectPath
+	obj := bus.GetBus().Object(PkIface, PkPath)
+	call := obj.Call(fmt.Sprintf("%s.CreateTransaction", PkIface), 0).Store(&interfacePath)
+	if call != nil {
+		return "", call
+	}
+	if PK_DEBUG {
+		log.Printf("PackageKit: CreateTransaction(): %v", interfacePath)
+	}
+	return interfacePath, nil
+}
+
+func (bus *Conn) ResolvePackages(packages []string, filter uint64) ([]string, error) {
+	packageIDs := []string{}
+	ch := make(chan *dbus.Signal, PkBufferSize)   // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction() // emits Destroy on close
+	if err != nil {
+		return []string{}, err
+	}
+
+	// add signal matches for Package and Finished which will always be last
+	var signals = []string{"Package", "Finished", "Error", "Destroy"}
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+	if PK_DEBUG {
+		log.Printf("PackageKit: ResolvePackages(): Object(%v, %v)", PkIface, interfacePath)
+	}
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("Resolve"), 0, filter, packages)
+	if PK_DEBUG {
+		log.Println("PackageKit: ResolvePackages(): Call: Success!")
+	}
+	if call.Err != nil {
+		return []string{}, call.Err
+	}
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+			if PK_DEBUG {
+				log.Printf("PackageKit: ResolvePackages(): Signal: %+v", signal)
+			}
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("Package") {
+				//pkg_int, ok := signal.Body[0].(int)
+				packageID, ok := signal.Body[1].(string)
+				// format is: name;version;arch;data
+				if !ok {
+					continue loop
+				}
+				//comment, ok := signal.Body[2].(string)
+				for _, p := range packageIDs {
+					if packageID == p {
+						continue loop // duplicate!
+					}
+				}
+				packageIDs = append(packageIDs, packageID)
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				return []string{}, fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		}
+	}
+	return packageIDs, nil
+}
+
+func (bus *Conn) IsInstalledList(packages []string) ([]bool, error) {
+	var filter uint64             // initializes at the "zero" value of 0
+	filter += PK_FILTER_ENUM_ARCH // always search in our arch
+	packageIDs, e := bus.ResolvePackages(packages, filter)
+	if e != nil {
+		return nil, fmt.Errorf("ResolvePackages error: %v", e)
+	}
+
+	var m = make(map[string]int)
+	for _, packageID := range packageIDs {
+		s := strings.Split(packageID, ";")
+		//if len(s) != 4 { continue } // this would be a bug!
+		pkg := s[0]
+		flags := strings.Split(s[3], ":")
+		for _, f := range flags {
+			if f == "installed" {
+				if _, exists := m[pkg]; !exists {
+					m[pkg] = 0
+				}
+				m[pkg]++ // if we see pkg installed, increment
+				break
+			}
+		}
+	}
+
+	var r []bool
+	for _, p := range packages {
+		if value, exists := m[p]; exists {
+			r = append(r, value > 0) // at least 1 means installed
+		} else {
+			r = append(r, false)
+		}
+	}
+	return r, nil
+}
+
+// is package installed ?
+// TODO: this could be optimized by making the resolve call directly
+func (bus *Conn) IsInstalled(pkg string) (bool, error) {
+	p, e := bus.IsInstalledList([]string{pkg})
+	if len(p) != 1 {
+		return false, e
+	}
+	return p[0], nil
+}
+
+// install list of packages by packageID
+func (bus *Conn) InstallPackages(packageIDs []string, transactionFlags uint64) error {
+
+	ch := make(chan *dbus.Signal, PkBufferSize)   // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction() // emits Destroy on close
+	if err != nil {
+		return err
+	}
+
+	var signals = []string{"Package", "ErrorCode", "Finished", "Destroy"} // "ItemProgress", "Status" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("InstallPackages"), 0, transactionFlags, packageIDs)
+	if call.Err != nil {
+		return call.Err
+	}
+	timeout := -1 // disabled initially
+	finished := false
+loop:
+	for {
+		select {
+		case signal := <-ch:
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			} else if signal.Name == FmtTransactionMethod("Package") {
+				// a package was installed...
+				// only start the timer once we're here...
+				timeout = PkSignalPackageTimeout
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				finished = true
+				timeout = PkSignalDestroyTimeout // wait a bit
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				return nil // success
+			} else {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		case _ = <-TimeAfterOrBlock(timeout):
+			if finished {
+				log.Println("PackageKit: Timeout: InstallPackages: Waiting for 'Destroy'")
+				return nil // got tired of waiting for Destroy
+			}
+			return fmt.Errorf("PackageKit: Timeout: InstallPackages: %v", strings.Join(packageIDs, ", "))
+		}
+	}
+}
+
+// remove list of packages
+func (bus *Conn) RemovePackages(packageIDs []string, transactionFlags uint64) error {
+
+	var allowDeps = true                          // TODO: configurable
+	var autoremove = false                        // unsupported on GNU/Linux
+	ch := make(chan *dbus.Signal, PkBufferSize)   // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction() // emits Destroy on close
+	if err != nil {
+		return err
+	}
+
+	var signals = []string{"Package", "ErrorCode", "Finished", "Destroy"} // "ItemProgress", "Status" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("RemovePackages"), 0, transactionFlags, packageIDs, allowDeps, autoremove)
+	if call.Err != nil {
+		return call.Err
+	}
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			} else if signal.Name == FmtTransactionMethod("Package") {
+				// a package was installed...
+				continue loop
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		}
+	}
+	return nil
+}
+
+// update list of packages to versions that are specified
+func (bus *Conn) UpdatePackages(packageIDs []string, transactionFlags uint64) error {
+	ch := make(chan *dbus.Signal, PkBufferSize) // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction()
+	if err != nil {
+		return err
+	}
+
+	var signals = []string{"Package", "ErrorCode", "Finished", "Destroy"} // "ItemProgress", "Status" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("UpdatePackages"), 0, transactionFlags, packageIDs)
+	if call.Err != nil {
+		return call.Err
+	}
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			} else if signal.Name == FmtTransactionMethod("Package") {
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		}
+	}
+	return nil
+}
+
+// get the list of files that are contained inside a list of packageids
+func (bus *Conn) GetFilesByPackageID(packageIDs []string) (files map[string][]string, err error) {
+	// NOTE: the maximum number of files in an RPM is 52116 in Fedora 23
+	// https://gist.github.com/purpleidea/b98e60dcd449e1ac3b8a
+	ch := make(chan *dbus.Signal, PkBufferSize) // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction()
+	if err != nil {
+		return
+	}
+
+	var signals = []string{"Files", "ErrorCode", "Finished", "Destroy"} // "ItemProgress", "Status" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("GetFiles"), 0, packageIDs)
+	if call.Err != nil {
+		err = call.Err
+		return
+	}
+	files = make(map[string][]string)
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				err = fmt.Errorf("PackageKit: Error: %v", signal.Body)
+				return
+
+				// one signal returned per packageID found...
+			} else if signal.Name == FmtTransactionMethod("Files") {
+				if len(signal.Body) != 2 { // bad data
+					continue loop
+				}
+				var ok bool
+				var key string
+				var fileList []string
+				if key, ok = signal.Body[0].(string); !ok {
+					continue loop
+				}
+				if fileList, ok = signal.Body[1].([]string); !ok {
+					continue loop // failed conversion
+				}
+				files[key] = fileList // build up map
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				err = fmt.Errorf("PackageKit: Error: %v", signal.Body)
+				return
+			}
+		}
+	}
+	return
+}
+
+// get list of packages that are installed and which can be updated, mod filter
+func (bus *Conn) GetUpdates(filter uint64) ([]string, error) {
+	if PK_DEBUG {
+		log.Println("PackageKit: GetUpdates()")
+	}
+	packageIDs := []string{}
+	ch := make(chan *dbus.Signal, PkBufferSize) // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction()
+	if err != nil {
+		return nil, err
+	}
+
+	var signals = []string{"Package", "ErrorCode", "Finished", "Destroy"} // "ItemProgress" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("GetUpdates"), 0, filter)
+	if call.Err != nil {
+		return nil, call.Err
+	}
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				return nil, fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			} else if signal.Name == FmtTransactionMethod("Package") {
+
+				//pkg_int, ok := signal.Body[0].(int)
+				packageID, ok := signal.Body[1].(string)
+				// format is: name;version;arch;data
+				if !ok {
+					continue loop
+				}
+				//comment, ok := signal.Body[2].(string)
+				for _, p := range packageIDs { // optional?
+					if packageID == p {
+						continue loop // duplicate!
+					}
+				}
+				packageIDs = append(packageIDs, packageID)
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				return nil, fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		}
+	}
+	return packageIDs, nil
+}
+
+// this is a helper function that *might* be generally useful outside mgmtconfig
+// packageMap input has the package names as keys and requested states as values
+// these states can be installed, uninstalled, newest or a requested version str
+func (bus *Conn) PackagesToPackageIDs(packageMap map[string]string, filter uint64) (map[string]*PkPackageIDActionData, error) {
+	count := 0
+	packages := make([]string, len(packageMap))
+	for k := range packageMap { // lol, golang has no hash.keys() function!
+		packages[count] = k
+		count++
+	}
+
+	if !(filter&PK_FILTER_ENUM_ARCH == PK_FILTER_ENUM_ARCH) {
+		filter += PK_FILTER_ENUM_ARCH // always search in our arch
+	}
+
+	if PK_DEBUG {
+		log.Printf("PackageKit: PackagesToPackageIDs(): %v", strings.Join(packages, ", "))
+	}
+	resolved, e := bus.ResolvePackages(packages, filter)
+	if e != nil {
+		return nil, fmt.Errorf("Resolve error: %v", e)
+	}
+
+	found := make([]bool, count) // default false
+	installed := make([]bool, count)
+	version := make([]string, count)
+	usePackageID := make([]string, count)
+	newest := make([]bool, count) // default true
+	for i := range newest {
+		newest[i] = true // assume, for now
+	}
+	var index int
+
+	for _, packageID := range resolved {
+		index = -1
+		//log.Printf("* %v", packageID)
+		// format is: name;version;arch;data
+		s := strings.Split(packageID, ";")
+		//if len(s) != 4 { continue } // this would be a bug!
+		pkg, ver, arch, data := s[0], s[1], s[2], s[3]
+		// we might need to allow some of this, eg: i386 .deb on amd64
+		if !IsMyArch(arch) {
+			continue
+		}
+
+		for i := range packages { // find pkg if it exists
+			if pkg == packages[i] {
+				index = i
+			}
+		}
+		if index == -1 { // can't find what we're looking for
+			continue
+		}
+		state := packageMap[pkg] // lookup the requested state/version
+		if state == "" {
+			return nil, fmt.Errorf("Empty package state for %v", pkg)
+		}
+		found[index] = true
+		stateIsVersion := (state != "installed" && state != "uninstalled" && state != "newest") // must be a ver. string
+
+		if stateIsVersion {
+			if state == ver && ver != "" { // we match what we want...
+				usePackageID[index] = packageID
+			}
+		}
+
+		if FlagInData("installed", data) {
+			installed[index] = true
+			version[index] = ver
+			// state of "uninstalled" matched during CheckApply, and
+			// states of "installed" and "newest" for fileList
+			if !stateIsVersion {
+				usePackageID[index] = packageID // save for later
+			}
+		} else { // not installed...
+			if !stateIsVersion {
+				// if there is more than one result, eg: there
+				// is the old and newest version of a package,
+				// then this section can run more than once...
+				// in that case, don't worry, we'll choose the
+				// right value in the "updates" section below!
+				usePackageID[index] = packageID
+			}
+		}
+	}
+
+	// we can't determine which packages are "newest", without searching
+	// for each one individually, so instead we check if any updates need
+	// to be done, and if so, anything that needs updating isn't newest!
+	// if something isn't installed, we can't verify it with this method
+	// FIXME: https://github.com/hughsie/PackageKit/issues/116
+	updates, e := bus.GetUpdates(filter)
+	if e != nil {
+		return nil, fmt.Errorf("Updates error: %v", e)
+	}
+	for _, packageID := range updates {
+		//log.Printf("* %v", packageID)
+		// format is: name;version;arch;data
+		s := strings.Split(packageID, ";")
+		//if len(s) != 4 { continue } // this would be a bug!
+		pkg, _, _, _ := s[0], s[1], s[2], s[3]
+		for index := range packages { // find pkg if it exists
+			if pkg == packages[index] {
+				state := packageMap[pkg] // lookup
+				newest[index] = false
+				if state == "installed" || state == "newest" {
+					// fix up in case above wasn't correct!
+					usePackageID[index] = packageID
+				}
+				break
+			}
+		}
+	}
+
+	// skip if the "newest" filter was used, otherwise we might need fixing
+	// this check is for packages that need to verify their "newest" status
+	// we need to know this so we can install the correct newest packageID!
+	recursion := make(map[string]*PkPackageIDActionData)
+	if !(filter&PK_FILTER_ENUM_NEWEST == PK_FILTER_ENUM_NEWEST) {
+		checkPackages := []string{}
+		filteredPackageMap := make(map[string]string)
+		for index, pkg := range packages {
+			state := packageMap[pkg]               // lookup the requested state/version
+			if !found[index] || installed[index] { // skip these, they're okay
+				continue
+			}
+			if !(state == "newest" || state == "installed") {
+				continue
+			}
+
+			checkPackages = append(checkPackages, pkg)
+			filteredPackageMap[pkg] = packageMap[pkg] // check me!
+		}
+
+		// we _could_ do a second resolve and then parse like this...
+		//resolved, e := bus.ResolvePackages(..., filter+PK_FILTER_ENUM_NEWEST)
+		// but that's basically what recursion here could do too!
+		if len(checkPackages) > 0 {
+			if PK_DEBUG {
+				log.Printf("PackageKit: PackagesToPackageIDs(): Recurse: %v", strings.Join(checkPackages, ", "))
+			}
+			recursion, e = bus.PackagesToPackageIDs(filteredPackageMap, filter+PK_FILTER_ENUM_NEWEST)
+			if e != nil {
+				return nil, fmt.Errorf("Recursion error: %v", e)
+			}
+		}
+	}
+
+	// fix up and build result format
+	result := make(map[string]*PkPackageIDActionData)
+	for index, pkg := range packages {
+
+		if !found[index] || !installed[index] {
+			newest[index] = false // make the results more logical!
+		}
+
+		// prefer recursion results if present
+		if lookup, ok := recursion[pkg]; ok {
+			result[pkg] = lookup
+		} else {
+			result[pkg] = &PkPackageIDActionData{
+				Found:     found[index],
+				Installed: installed[index],
+				Version:   version[index],
+				PackageID: usePackageID[index],
+				Newest:    newest[index],
+			}
+		}
+	}
+
+	return result, nil
+}
+
+// returns a list of packageIDs which match the set of package names in packages
+func FilterPackageIDs(m map[string]*PkPackageIDActionData, packages []string) ([]string, error) {
+	result := []string{}
+	for _, k := range packages {
+		obj, ok := m[k] // lookup single package
+		// package doesn't exist, this is an error!
+		if !ok || !obj.Found || obj.PackageID == "" {
+			return nil, fmt.Errorf("Can't find package named '%s'.", k)
+		}
+		result = append(result, obj.PackageID)
+	}
+	return result, nil
+}
+
+func FilterState(m map[string]*PkPackageIDActionData, packages []string, state string) (result map[string]bool, err error) {
+	result = make(map[string]bool)
+	pkgs := []string{} // bad pkgs that don't have a bool state
+	for _, k := range packages {
+		obj, ok := m[k] // lookup single package
+		// package doesn't exist, this is an error!
+		if !ok || !obj.Found {
+			return nil, fmt.Errorf("Can't find package named '%s'.", k)
+		}
+		var b bool
+		if state == "installed" {
+			b = obj.Installed
+		} else if state == "uninstalled" {
+			b = !obj.Installed
+		} else if state == "newest" {
+			b = obj.Newest
+		} else {
+			// we can't filter "version" state in this function
+			pkgs = append(pkgs, k)
+			continue
+		}
+		result[k] = b // save
+	}
+	if len(pkgs) > 0 {
+		err = fmt.Errorf("Can't filter non-boolean state on: %v!", strings.Join(pkgs, ","))
+	}
+	return result, err
+}
+
+// return all packages that are in package and match the specific state
+func FilterPackageState(m map[string]*PkPackageIDActionData, packages []string, state string) (result []string, err error) {
+	result = []string{}
+	for _, k := range packages {
+		obj, ok := m[k] // lookup single package
+		// package doesn't exist, this is an error!
+		if !ok || !obj.Found {
+			return nil, fmt.Errorf("Can't find package named '%s'.", k)
+		}
+		b := false
+		if state == "installed" && obj.Installed {
+			b = true
+		} else if state == "uninstalled" && !obj.Installed {
+			b = true
+		} else if state == "newest" && obj.Newest {
+			b = true
+		} else if state == obj.Version {
+			b = true
+		}
+		if b {
+			result = append(result, k)
+		}
+	}
+	return result, err
+}
+
+// does flag exist inside data portion of packageID field?
+func FlagInData(flag, data string) bool {
+	flags := strings.Split(data, ":")
+	for _, f := range flags {
+		if f == flag {
+			return true
+		}
+	}
+	return false
+}
+
+// builds the transaction method string
+func FmtTransactionMethod(method string) string {
+	return fmt.Sprintf("%s.%s", PkIfaceTransaction, method)
+}
+
+func IsMyArch(arch string) bool {
+	goarch, ok := PkArchMap[arch]
+	if !ok {
+		// if you get this error, please update the PkArchMap const
+		log.Fatalf("PackageKit: Arch '%v', not found!", arch)
+	}
+	if goarch == "ANY" { // special value that corresponds to noarch
+		return true
+	}
+	return goarch == runtime.GOARCH
+}

pkg.go

@@ -0,0 +1,545 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	//"packagekit" // TODO
+	"encoding/gob"
+	"errors"
+	"fmt"
+	"log"
+	"path"
+	"strings"
+)
+
+func init() {
+	gob.Register(&PkgRes{})
+}
+
+type PkgRes struct {
+	BaseRes          `yaml:",inline"`
+	State            string `yaml:"state"`            // state: installed, uninstalled, newest, <version>
+	AllowUntrusted   bool   `yaml:"allowuntrusted"`   // allow untrusted packages to be installed?
+	AllowNonFree     bool   `yaml:"allownonfree"`     // allow nonfree packages to be found?
+	AllowUnsupported bool   `yaml:"allowunsupported"` // allow unsupported packages to be found?
+	//bus              *Conn    // pk bus connection
+	fileList []string // FIXME: update if pkg changes
+}
+
+// helper function for creating new pkg resources that calls Init()
+func NewPkgRes(name, state string, allowuntrusted, allownonfree, allowunsupported bool) *PkgRes {
+	obj := &PkgRes{
+		BaseRes: BaseRes{
+			Name:   name,
+			events: make(chan Event),
+			vertex: nil,
+		},
+		State:            state,
+		AllowUntrusted:   allowuntrusted,
+		AllowNonFree:     allownonfree,
+		AllowUnsupported: allowunsupported,
+	}
+	obj.Init()
+	return obj
+}
+
+func (obj *PkgRes) Init() {
+	obj.BaseRes.kind = "Pkg"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+
+	bus := NewBus()
+	if bus == nil {
+		log.Fatal("Can't connect to PackageKit bus.")
+	}
+	defer bus.Close()
+
+	result, err := obj.pkgMappingHelper(bus)
+	if err != nil {
+		// FIXME: return error?
+		log.Fatalf("The pkgMappingHelper failed with: %v.", err)
+		return
+	}
+
+	data, ok := result[obj.Name] // lookup single package (init does just one)
+	// package doesn't exist, this is an error!
+	if !ok || !data.Found {
+		// FIXME: return error?
+		log.Fatalf("Can't find package named '%s'.", obj.Name)
+		return
+	}
+
+	packageIDs := []string{data.PackageID} // just one for now
+	filesMap, err := bus.GetFilesByPackageID(packageIDs)
+	if err != nil {
+		// FIXME: return error?
+		log.Fatalf("Can't run GetFilesByPackageID: %v", err)
+		return
+	}
+	if files, ok := filesMap[data.PackageID]; ok {
+		obj.fileList = DirifyFileList(files, false)
+	}
+}
+
+func (obj *PkgRes) Validate() bool {
+
+	if obj.State == "" {
+		return false
+	}
+
+	return true
+}
+
+// use UpdatesChanged signal to watch for changes
+// TODO: https://github.com/hughsie/PackageKit/issues/109
+// TODO: https://github.com/hughsie/PackageKit/issues/110
+func (obj *PkgRes) Watch(processChan chan Event) {
+	if obj.IsWatching() {
+		return
+	}
+	obj.SetWatching(true)
+	defer obj.SetWatching(false)
+
+	bus := NewBus()
+	if bus == nil {
+		log.Fatal("Can't connect to PackageKit bus.")
+	}
+	defer bus.Close()
+
+	ch, err := bus.WatchChanges()
+	if err != nil {
+		log.Fatalf("Error adding signal match: %v", err)
+	}
+
+	var send = false // send event?
+	var exit = false
+	var dirty = false
+
+	for {
+		if DEBUG {
+			log.Printf("%v: Watching...", obj.fmtNames(obj.getNames()))
+		}
+
+		obj.SetState(resStateWatching) // reset
+		select {
+		case event := <-ch:
+			// FIXME: ask packagekit for info on what packages changed
+			if DEBUG {
+				log.Printf("%v: Event: %v", obj.fmtNames(obj.getNames()), event.Name)
+			}
+
+			// since the chan is buffered, remove any supplemental
+			// events since they would just be duplicates anyways!
+			for len(ch) > 0 { // we can detect pending count here!
+				<-ch // discard
+			}
+
+			obj.SetConvergedState(resConvergedNil)
+			send = true
+			dirty = true
+
+		case event := <-obj.events:
+			obj.SetConvergedState(resConvergedNil)
+			if exit, send = obj.ReadEvent(&event); exit {
+				return // exit
+			}
+			//dirty = false // these events don't invalidate state
+
+		case _ = <-TimeAfterOrBlock(obj.ctimeout):
+			obj.SetConvergedState(resConvergedTimeout)
+			obj.converged <- true
+			continue
+		}
+
+		// do all our event sending all together to avoid duplicate msgs
+		if send {
+			send = false
+			// only invalid state on certain types of events
+			if dirty {
+				dirty = false
+				obj.isStateOK = false // something made state dirty
+			}
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
+		}
+	}
+}
+
+// get list of names when grouped or not
+func (obj *PkgRes) getNames() []string {
+	if g := obj.GetGroup(); len(g) > 0 { // grouped elements
+		names := []string{obj.GetName()}
+		for _, x := range g {
+			pkg, ok := x.(*PkgRes) // convert from Res
+			if ok {
+				names = append(names, pkg.Name)
+			}
+		}
+		return names
+	}
+	return []string{obj.GetName()}
+}
+
+// pretty print for header values
+func (obj *PkgRes) fmtNames(names []string) string {
+	if len(obj.GetGroup()) > 0 { // grouped elements
+		return fmt.Sprintf("%v[autogroup:(%v)]", obj.Kind(), strings.Join(names, ","))
+	}
+	return fmt.Sprintf("%v[%v]", obj.Kind(), obj.GetName())
+}
+
+func (obj *PkgRes) groupMappingHelper() map[string]string {
+	var result = make(map[string]string)
+	if g := obj.GetGroup(); len(g) > 0 { // add any grouped elements
+		for _, x := range g {
+			pkg, ok := x.(*PkgRes) // convert from Res
+			if !ok {
+				log.Fatalf("Grouped member %v is not a %v", x, obj.Kind())
+			}
+			result[pkg.Name] = pkg.State
+		}
+	}
+	return result
+}
+
+func (obj *PkgRes) pkgMappingHelper(bus *Conn) (map[string]*PkPackageIDActionData, error) {
+	packageMap := obj.groupMappingHelper() // get the grouped values
+	packageMap[obj.Name] = obj.State       // key is pkg name, value is pkg state
+	var filter uint64                      // initializes at the "zero" value of 0
+	filter += PK_FILTER_ENUM_ARCH          // always search in our arch (optional!)
+	// we're requesting latest version, or to narrow down install choices!
+	if obj.State == "newest" || obj.State == "installed" {
+		// if we add this, we'll still see older packages if installed
+		// this is an optimization, and is *optional*, this logic is
+		// handled inside of PackagesToPackageIDs now automatically!
+		filter += PK_FILTER_ENUM_NEWEST // only search for newest packages
+	}
+	if !obj.AllowNonFree {
+		filter += PK_FILTER_ENUM_FREE
+	}
+	if !obj.AllowUnsupported {
+		filter += PK_FILTER_ENUM_SUPPORTED
+	}
+	result, e := bus.PackagesToPackageIDs(packageMap, filter)
+	if e != nil {
+		return nil, fmt.Errorf("Can't run PackagesToPackageIDs: %v", e)
+	}
+	return result, nil
+}
+
+func (obj *PkgRes) CheckApply(apply bool) (stateok bool, err error) {
+	log.Printf("%v: CheckApply(%t)", obj.fmtNames(obj.getNames()), apply)
+
+	if obj.State == "" { // TODO: Validate() should replace this check!
+		log.Fatalf("%v: Package state is undefined!", obj.fmtNames(obj.getNames()))
+	}
+
+	if obj.isStateOK { // cache the state
+		return true, nil
+	}
+
+	bus := NewBus()
+	if bus == nil {
+		return false, errors.New("Can't connect to PackageKit bus.")
+	}
+	defer bus.Close()
+
+	result, err := obj.pkgMappingHelper(bus)
+	if err != nil {
+		return false, fmt.Errorf("The pkgMappingHelper failed with: %v.", err)
+	}
+
+	packageMap := obj.groupMappingHelper() // map[string]string
+	packageList := []string{obj.Name}
+	packageList = append(packageList, StrMapKeys(packageMap)...)
+	//stateList := []string{obj.State}
+	//stateList = append(stateList, StrMapValues(packageMap)...)
+
+	// TODO: at the moment, all the states are the same, but
+	// eventually we might be able to drop this constraint!
+	states, err := FilterState(result, packageList, obj.State)
+	if err != nil {
+		return false, fmt.Errorf("The FilterState method failed with: %v.", err)
+	}
+	data, _ := result[obj.Name] // if above didn't error, we won't either!
+	validState := BoolMapTrue(BoolMapValues(states))
+
+	// obj.State == "installed" || "uninstalled" || "newest" || "4.2-1.fc23"
+	switch obj.State {
+	case "installed":
+		fallthrough
+	case "uninstalled":
+		fallthrough
+	case "newest":
+		if validState {
+			return true, nil // state is correct, exit!
+		}
+	default: // version string
+		if obj.State == data.Version && data.Version != "" {
+			return true, nil
+		}
+	}
+
+	// state is not okay, no work done, exit, but without error
+	if !apply {
+		return false, nil
+	}
+
+	// apply portion
+	log.Printf("%v: Apply", obj.fmtNames(obj.getNames()))
+	readyPackages, err := FilterPackageState(result, packageList, obj.State)
+	if err != nil {
+		return false, err // fail
+	}
+	// these are the packages that actually need their states applied!
+	applyPackages := StrFilterElementsInList(readyPackages, packageList)
+	packageIDs, _ := FilterPackageIDs(result, applyPackages) // would be same err as above
+
+	var transactionFlags uint64 // initializes at the "zero" value of 0
+	if !obj.AllowUntrusted {    // allow
+		transactionFlags += PK_TRANSACTION_FLAG_ENUM_ONLY_TRUSTED
+	}
+	// apply correct state!
+	log.Printf("%v: Set: %v...", obj.fmtNames(StrListIntersection(applyPackages, obj.getNames())), obj.State)
+	switch obj.State {
+	case "uninstalled": // run remove
+		// NOTE: packageID is different than when installed, because now
+		// it has the "installed" flag added to the data portion if it!!
+		err = bus.RemovePackages(packageIDs, transactionFlags)
+
+	case "newest": // TODO: isn't this the same operation as install, below?
+		err = bus.UpdatePackages(packageIDs, transactionFlags)
+
+	case "installed":
+		fallthrough // same method as for "set specific version", below
+	default: // version string
+		err = bus.InstallPackages(packageIDs, transactionFlags)
+	}
+	if err != nil {
+		return false, err // fail
+	}
+	log.Printf("%v: Set: %v success!", obj.fmtNames(StrListIntersection(applyPackages, obj.getNames())), obj.State)
+	return false, nil // success
+}
+
+type PkgUUID struct {
+	BaseUUID
+	name  string // pkg name
+	state string // pkg state or "version"
+}
+
+// if and only if they are equivalent, return true
+// if they are not equivalent, return false
+func (obj *PkgUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*PkgUUID)
+	if !ok {
+		return false
+	}
+	// FIXME: match on obj.state vs. res.state ?
+	return obj.name == res.name
+}
+
+type PkgResAutoEdges struct {
+	fileList   []string
+	svcUUIDs   []ResUUID
+	testIsNext bool   // safety
+	name       string // saved data from PkgRes obj
+	kind       string
+}
+
+func (obj *PkgResAutoEdges) Next() []ResUUID {
+	if obj.testIsNext {
+		log.Fatal("Expecting a call to Test()")
+	}
+	obj.testIsNext = true // set after all the errors paths are past
+
+	// first return any matching svcUUIDs
+	if x := obj.svcUUIDs; len(x) > 0 {
+		return x
+	}
+
+	var result []ResUUID
+	// return UUID's for whatever is in obj.fileList
+	for _, x := range obj.fileList {
+		var reversed = false // cheat by passing a pointer
+		result = append(result, &FileUUID{
+			BaseUUID: BaseUUID{
+				name:     obj.name,
+				kind:     obj.kind,
+				reversed: &reversed,
+			},
+			path: x, // what matters
+		}) // build list
+	}
+	return result
+}
+
+func (obj *PkgResAutoEdges) Test(input []bool) bool {
+	if !obj.testIsNext {
+		log.Fatal("Expecting a call to Next()")
+	}
+
+	// ack the svcUUID's...
+	if x := obj.svcUUIDs; len(x) > 0 {
+		if y := len(x); y != len(input) {
+			log.Fatalf("Expecting %d value(s)!", y)
+		}
+		obj.svcUUIDs = []ResUUID{} // empty
+		obj.testIsNext = false
+		return true
+	}
+
+	count := len(obj.fileList)
+	if count != len(input) {
+		log.Fatalf("Expecting %d value(s)!", count)
+	}
+	obj.testIsNext = false // set after all the errors paths are past
+
+	// while i do believe this algorithm generates the *correct* result, i
+	// don't know if it does so in the optimal way. improvements welcome!
+	// the basic logic is:
+	// 0) Next() returns whatever is in fileList
+	// 1) Test() computes the dirname of each file, and removes duplicates
+	// and dirname's that have been in the path of an ack from input results
+	// 2) It then simplifies the list by removing the common path prefixes
+	// 3) Lastly, the remaining set of files (dirs) is used as new fileList
+	// 4) We then iterate in (0) until the fileList is empty!
+	var dirs = make([]string, count)
+	done := []string{}
+	for i := 0; i < count; i++ {
+		dir := Dirname(obj.fileList[i]) // dirname of /foo/ should be /
+		dirs[i] = dir
+		if input[i] {
+			done = append(done, dir)
+		}
+	}
+	nodupes := StrRemoveDuplicatesInList(dirs)                // remove duplicates
+	nodones := StrFilterElementsInList(done, nodupes)         // filter out done
+	noempty := StrFilterElementsInList([]string{""}, nodones) // remove the "" from /
+	obj.fileList = RemoveCommonFilePrefixes(noempty)          // magic
+
+	if len(obj.fileList) == 0 { // nothing more, don't continue
+		return false
+	}
+	return true // continue, there are more files!
+}
+
+// produce an object which generates a minimal pkg file optimization sequence
+func (obj *PkgRes) AutoEdges() AutoEdge {
+	// in contrast with the FileRes AutoEdges() function which contains
+	// more of the mechanics, most of the AutoEdge mechanics for the PkgRes
+	// is contained in the Test() method! This design is completely okay!
+
+	// add matches for any svc resources found in pkg definition!
+	var svcUUIDs []ResUUID
+	for _, x := range ReturnSvcInFileList(obj.fileList) {
+		var reversed = false
+		svcUUIDs = append(svcUUIDs, &SvcUUID{
+			BaseUUID: BaseUUID{
+				name:     obj.GetName(),
+				kind:     obj.Kind(),
+				reversed: &reversed,
+			},
+			name: x, // the svc name itself in the SvcUUID object!
+		}) // build list
+	}
+
+	return &PkgResAutoEdges{
+		fileList:   RemoveCommonFilePrefixes(obj.fileList), // clean start!
+		svcUUIDs:   svcUUIDs,
+		testIsNext: false,         // start with Next() call
+		name:       obj.GetName(), // save data for PkgResAutoEdges obj
+		kind:       obj.Kind(),
+	}
+}
+
+// include all params to make a unique identification of this object
+func (obj *PkgRes) GetUUIDs() []ResUUID {
+	x := &PkgUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		name:     obj.Name,
+		state:    obj.State,
+	}
+	result := []ResUUID{x}
+	return result
+}
+
+// can these two resources be merged ?
+// (aka does this resource support doing so?)
+// will resource allow itself to be grouped _into_ this obj?
+func (obj *PkgRes) GroupCmp(r Res) bool {
+	res, ok := r.(*PkgRes)
+	if !ok {
+		return false
+	}
+	objStateIsVersion := (obj.State != "installed" && obj.State != "uninstalled" && obj.State != "newest") // must be a ver. string
+	resStateIsVersion := (res.State != "installed" && res.State != "uninstalled" && res.State != "newest") // must be a ver. string
+	if objStateIsVersion || resStateIsVersion {
+		// can't merge specific version checks atm
+		return false
+	}
+	// FIXME: keep it simple for now, only merge same states
+	if obj.State != res.State {
+		return false
+	}
+	return true
+}
+
+func (obj *PkgRes) Compare(res Res) bool {
+	switch res.(type) {
+	case *PkgRes:
+		res := res.(*PkgRes)
+		if obj.Name != res.Name {
+			return false
+		}
+		if obj.State != res.State {
+			return false
+		}
+		if obj.AllowUntrusted != res.AllowUntrusted {
+			return false
+		}
+		if obj.AllowNonFree != res.AllowNonFree {
+			return false
+		}
+		if obj.AllowUnsupported != res.AllowUnsupported {
+			return false
+		}
+	default:
+		return false
+	}
+	return true
+}
+
+// return a list of svc names for matches like /usr/lib/systemd/system/*.service
+func ReturnSvcInFileList(fileList []string) []string {
+	result := []string{}
+	for _, x := range fileList {
+		dirname, basename := path.Split(path.Clean(x))
+		// TODO: do we also want to look for /etc/systemd/system/ ?
+		if dirname != "/usr/lib/systemd/system/" {
+			continue
+		}
+		if !strings.HasSuffix(basename, ".service") {
+			continue
+		}
+		if s := strings.TrimSuffix(basename, ".service"); !StrInList(s, result) {
+			result = append(result, s)
+		}
+	}
+	return result
+}

resources.go

@@ -0,0 +1,353 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"bytes"
+	"encoding/base64"
+	"encoding/gob"
+	"fmt"
+	"log"
+)
+
+//go:generate stringer -type=resState -output=resstate_stringer.go
+type resState int
+
+const (
+	resStateNil resState = iota
+	resStateWatching
+	resStateEvent // an event has happened, but we haven't poked yet
+	resStateCheckApply
+	resStatePoking
+)
+
+//go:generate stringer -type=resConvergedState -output=resconvergedstate_stringer.go
+type resConvergedState int
+
+const (
+	resConvergedNil resConvergedState = iota
+	//resConverged
+	resConvergedTimeout
+)
+
+// a unique identifier for a resource, namely it's name, and the kind ("type")
+type ResUUID interface {
+	GetName() string
+	Kind() string
+	IFF(ResUUID) bool
+
+	Reversed() bool // true means this resource happens before the generator
+}
+
+type BaseUUID struct {
+	name string // name and kind are the values of where this is coming from
+	kind string
+
+	reversed *bool // piggyback edge information here
+}
+
+type AutoEdge interface {
+	Next() []ResUUID  // call to get list of edges to add
+	Test([]bool) bool // call until false
+}
+
+type MetaParams struct {
+	AutoEdge  bool `yaml:"autoedge"`  // metaparam, should we generate auto edges? // XXX should default to true
+	AutoGroup bool `yaml:"autogroup"` // metaparam, should we auto group? // XXX should default to true
+}
+
+// this interface is everything that is common to all resources
+// everything here only needs to be implemented once, in the BaseRes
+type Base interface {
+	GetName() string // can't be named "Name()" because of struct field
+	SetName(string)
+	Kind() string
+	GetMeta() MetaParams
+	SetVertex(*Vertex)
+	SetConvergedCallback(ctimeout int, converged chan bool)
+	IsWatching() bool
+	SetWatching(bool)
+	GetConvergedState() resConvergedState
+	SetConvergedState(resConvergedState)
+	GetState() resState
+	SetState(resState)
+	SendEvent(eventName, bool, bool) bool
+	ReadEvent(*Event) (bool, bool) // TODO: optional here?
+	GroupCmp(Res) bool             // TODO: is there a better name for this?
+	GroupRes(Res) error            // group resource (arg) into self
+	IsGrouped() bool               // am I grouped?
+	SetGrouped(bool)               // set grouped bool
+	GetGroup() []Res               // return everyone grouped inside me
+	SetGroup([]Res)
+}
+
+// this is the minimum interface you need to implement to make a new resource
+type Res interface {
+	Base // include everything from the Base interface
+	Init()
+	//Validate() bool    // TODO: this might one day be added
+	GetUUIDs() []ResUUID // most resources only return one
+	Watch(chan Event)    // send on channel to signal process() events
+	CheckApply(bool) (bool, error)
+	AutoEdges() AutoEdge
+	Compare(Res) bool
+	CollectPattern(string) // XXX: temporary until Res collection is more advanced
+}
+
+type BaseRes struct {
+	Name           string     `yaml:"name"`
+	Meta           MetaParams `yaml:"meta"` // struct of all the metaparams
+	kind           string
+	events         chan Event
+	vertex         *Vertex
+	state          resState
+	convergedState resConvergedState
+	watching       bool // is Watch() loop running ?
+	ctimeout       int  // converged timeout
+	converged      chan bool
+	isStateOK      bool  // whether the state is okay based on events or not
+	isGrouped      bool  // am i contained within a group?
+	grouped        []Res // list of any grouped resources
+}
+
+// wraps the IFF method when used with a list of UUID's
+func UUIDExistsInUUIDs(uuid ResUUID, uuids []ResUUID) bool {
+	for _, u := range uuids {
+		if uuid.IFF(u) {
+			return true
+		}
+	}
+	return false
+}
+
+func (obj *BaseUUID) GetName() string {
+	return obj.name
+}
+
+func (obj *BaseUUID) Kind() string {
+	return obj.kind
+}
+
+// if and only if they are equivalent, return true
+// if they are not equivalent, return false
+// most resource will want to override this method, since it does the important
+// work of actually discerning if two resources are identical in function
+func (obj *BaseUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*BaseUUID)
+	if !ok {
+		return false
+	}
+	return obj.name == res.name
+}
+
+func (obj *BaseUUID) Reversed() bool {
+	if obj.reversed == nil {
+		log.Fatal("Programming error!")
+	}
+	return *obj.reversed
+}
+
+// initialize structures like channels if created without New constructor
+func (obj *BaseRes) Init() {
+	obj.events = make(chan Event) // unbuffered chan size to avoid stale events
+}
+
+// this method gets used by all the resources
+func (obj *BaseRes) GetName() string {
+	return obj.Name
+}
+
+func (obj *BaseRes) SetName(name string) {
+	obj.Name = name
+}
+
+// return the kind of resource this is
+func (obj *BaseRes) Kind() string {
+	return obj.kind
+}
+
+func (obj *BaseRes) GetMeta() MetaParams {
+	return obj.Meta
+}
+
+func (obj *BaseRes) GetVertex() *Vertex {
+	return obj.vertex
+}
+
+func (obj *BaseRes) SetVertex(v *Vertex) {
+	obj.vertex = v
+}
+
+func (obj *BaseRes) SetConvergedCallback(ctimeout int, converged chan bool) {
+	obj.ctimeout = ctimeout
+	obj.converged = converged
+}
+
+// is the Watch() function running?
+func (obj *BaseRes) IsWatching() bool {
+	return obj.watching
+}
+
+// store status of if the Watch() function is running
+func (obj *BaseRes) SetWatching(b bool) {
+	obj.watching = b
+}
+
+func (obj *BaseRes) GetConvergedState() resConvergedState {
+	return obj.convergedState
+}
+
+func (obj *BaseRes) SetConvergedState(state resConvergedState) {
+	obj.convergedState = state
+}
+
+func (obj *BaseRes) GetState() resState {
+	return obj.state
+}
+
+func (obj *BaseRes) SetState(state resState) {
+	if DEBUG {
+		log.Printf("%v[%v]: State: %v -> %v", obj.Kind(), obj.GetName(), obj.GetState(), state)
+	}
+	obj.state = state
+}
+
+// push an event into the message queue for a particular vertex
+func (obj *BaseRes) SendEvent(event eventName, sync bool, activity bool) bool {
+	// TODO: isn't this race-y ?
+	if !obj.IsWatching() { // element has already exited
+		return false // if we don't return, we'll block on the send
+	}
+	if !sync {
+		obj.events <- Event{event, nil, "", activity}
+		return true
+	}
+
+	resp := make(chan bool)
+	obj.events <- Event{event, resp, "", activity}
+	for {
+		value := <-resp
+		// wait until true value
+		if value {
+			return true
+		}
+	}
+}
+
+// process events when a select gets one, this handles the pause code too!
+// the return values specify if we should exit and poke respectively
+func (obj *BaseRes) ReadEvent(event *Event) (exit, poke bool) {
+	event.ACK()
+	switch event.Name {
+	case eventStart:
+		return false, true
+
+	case eventPoke:
+		return false, true
+
+	case eventBackPoke:
+		return false, true // forward poking in response to a back poke!
+
+	case eventExit:
+		return true, false
+
+	case eventPause:
+		// wait for next event to continue
+		select {
+		case e := <-obj.events:
+			e.ACK()
+			if e.Name == eventExit {
+				return true, false
+			} else if e.Name == eventStart { // eventContinue
+				return false, false // don't poke on unpause!
+			} else {
+				// if we get a poke event here, it's a bug!
+				log.Fatalf("%v[%v]: Unknown event: %v, while paused!", obj.Kind(), obj.GetName(), e)
+			}
+		}
+
+	default:
+		log.Fatal("Unknown event: ", event)
+	}
+	return true, false // required to keep the stupid go compiler happy
+}
+
+func (obj *BaseRes) GroupRes(res Res) error {
+	if l := len(res.GetGroup()); l > 0 {
+		return fmt.Errorf("Res: %v already contains %d grouped resources!", res, l)
+	}
+	if res.IsGrouped() {
+		return fmt.Errorf("Res: %v is already grouped!", res)
+	}
+
+	obj.grouped = append(obj.grouped, res)
+	res.SetGrouped(true) // i am contained _in_ a group
+	return nil
+}
+
+func (obj *BaseRes) IsGrouped() bool { // am I grouped?
+	return obj.isGrouped
+}
+
+func (obj *BaseRes) SetGrouped(b bool) {
+	obj.isGrouped = b
+}
+
+func (obj *BaseRes) GetGroup() []Res { // return everyone grouped inside me
+	return obj.grouped
+}
+
+func (obj *BaseRes) SetGroup(g []Res) {
+	obj.grouped = g
+}
+
+func (obj *BaseRes) CollectPattern(pattern string) {
+	// XXX: default method is empty
+}
+
+// ResToB64 encodes a resource to a base64 encoded string (after serialization)
+func ResToB64(res Res) (string, error) {
+	b := bytes.Buffer{}
+	e := gob.NewEncoder(&b)
+	err := e.Encode(&res) // pass with &
+	if err != nil {
+		return "", fmt.Errorf("Gob failed to encode: %v", err)
+	}
+	return base64.StdEncoding.EncodeToString(b.Bytes()), nil
+}
+
+// B64ToRes decodes a resource from a base64 encoded string (after deserialization)
+func B64ToRes(str string) (Res, error) {
+	var output interface{}
+	bb, err := base64.StdEncoding.DecodeString(str)
+	if err != nil {
+		return nil, fmt.Errorf("Base64 failed to decode: %v", err)
+	}
+	b := bytes.NewBuffer(bb)
+	d := gob.NewDecoder(b)
+	err = d.Decode(&output) // pass with &
+	if err != nil {
+		return nil, fmt.Errorf("Gob failed to decode: %v", err)
+	}
+	res, ok := output.(Res)
+	if !ok {
+		return nil, fmt.Errorf("Output %v is not a Res", res)
+
+	}
+	return res, nil
+}

resources_test.go

@@ -0,0 +1,105 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"bytes"
+	"encoding/base64"
+	"encoding/gob"
+	"testing"
+)
+
+func TestMiscEncodeDecode1(t *testing.T) {
+	var err error
+	//gob.Register( &NoopRes{} ) // happens in noop.go : init()
+	//gob.Register( &FileRes{} ) // happens in file.go : init()
+	// ...
+
+	// encode
+	var input interface{} = &FileRes{}
+	b1 := bytes.Buffer{}
+	e := gob.NewEncoder(&b1)
+	err = e.Encode(&input) // pass with &
+	if err != nil {
+		t.Errorf("Gob failed to Encode: %v", err)
+	}
+	str := base64.StdEncoding.EncodeToString(b1.Bytes())
+
+	// decode
+	var output interface{}
+	bb, err := base64.StdEncoding.DecodeString(str)
+	if err != nil {
+		t.Errorf("Base64 failed to Decode: %v", err)
+	}
+	b2 := bytes.NewBuffer(bb)
+	d := gob.NewDecoder(b2)
+	err = d.Decode(&output) // pass with &
+	if err != nil {
+		t.Errorf("Gob failed to Decode: %v", err)
+	}
+
+	res1, ok := input.(Res)
+	if !ok {
+		t.Errorf("Input %v is not a Res", res1)
+		return
+	}
+	res2, ok := output.(Res)
+	if !ok {
+		t.Errorf("Output %v is not a Res", res2)
+		return
+	}
+	if !res1.Compare(res2) {
+		t.Error("The input and output Res values do not match!")
+	}
+}
+
+func TestMiscEncodeDecode2(t *testing.T) {
+	var err error
+	//gob.Register( &NoopRes{} ) // happens in noop.go : init()
+	//gob.Register( &FileRes{} ) // happens in file.go : init()
+	// ...
+
+	// encode
+	var input Res = &FileRes{}
+
+	b64, err := ResToB64(input)
+	if err != nil {
+		t.Errorf("Can't encode: %v", err)
+		return
+	}
+
+	output, err := B64ToRes(b64)
+	if err != nil {
+		t.Errorf("Can't decode: %v", err)
+		return
+	}
+
+	res1, ok := input.(Res)
+	if !ok {
+		t.Errorf("Input %v is not a Res", res1)
+		return
+	}
+	res2, ok := output.(Res)
+	if !ok {
+		t.Errorf("Output %v is not a Res", res2)
+		return
+	}
+	if !res1.Compare(res2) {
+		t.Error("The input and output Res values do not match!")
+	}
+}

service.go

@@ -1,294 +0,0 @@
-// Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
-// Written by James Shubin <james@shubin.ca> and the project contributors
-//
-// This program is free software: you can redistribute it and/or modify
-// it under the terms of the GNU Affero General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-//
-// This program is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU Affero General Public License for more details.
-//
-// You should have received a copy of the GNU Affero General Public License
-// along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-// NOTE: docs are found at: https://godoc.org/github.com/coreos/go-systemd/dbus
-
-package main
-
-import (
-	"code.google.com/p/go-uuid/uuid"
-	"fmt"
-	systemd "github.com/coreos/go-systemd/dbus" // change namespace
-	"github.com/coreos/go-systemd/util"
-	"github.com/godbus/dbus" // namespace collides with systemd wrapper
-	"log"
-)
-
-type ServiceType struct {
-	uuid    string
-	Type    string      // always "service"
-	Name    string      // name variable
-	Events  chan string // FIXME: eventually a struct for the event?
-	State   string      // state: running, stopped
-	Startup string      // enabled, disabled, undefined
-}
-
-func NewServiceType(name, state, startup string) *ServiceType {
-	return &ServiceType{
-		uuid:    uuid.New(),
-		Type:    "service",
-		Name:    name,
-		Events:  make(chan string, 1), // XXX: chan size?
-		State:   state,
-		Startup: startup,
-	}
-}
-
-// Service watcher
-func (obj ServiceType) Watch(v *Vertex) {
-	// obj.Name: service name
-
-	if !util.IsRunningSystemd() {
-		log.Fatal("Systemd is not running.")
-	}
-
-	conn, err := systemd.NewSystemdConnection() // needs root access
-	if err != nil {
-		log.Fatal("Failed to connect to systemd: ", err)
-	}
-	defer conn.Close()
-
-	bus, err := dbus.SystemBus()
-	if err != nil {
-		log.Fatal("Failed to connect to bus: %v\n", err)
-	}
-
-	// XXX: will this detect new units?
-	bus.BusObject().Call("org.freedesktop.DBus.AddMatch", 0,
-		"type='signal',interface='org.freedesktop.systemd1.Manager',member='Reloading'")
-	buschan := make(chan *dbus.Signal, 10)
-	bus.Signal(buschan)
-
-	var service = fmt.Sprintf("%v.service", obj.Name) // systemd name
-	var send = false                                  // send event?
-	var invalid = false                               // does the service exist or not?
-	var previous bool                                 // previous invalid value
-	set := conn.NewSubscriptionSet()                  // no error should be returned
-	subChannel, subErrors := set.Subscribe()
-	var activeSet = false
-
-	for {
-		// XXX: watch for an event for new units...
-		// XXX: detect if startup enabled/disabled value changes...
-
-		previous = invalid
-		invalid = false
-
-		// firstly, does service even exist or not?
-		loadstate, err := conn.GetUnitProperty(service, "LoadState")
-		if err != nil {
-			log.Printf("Failed to get property: %v\n", err)
-			invalid = true
-		}
-
-		if !invalid {
-			var notFound = (loadstate.Value == dbus.MakeVariant("not-found"))
-			if notFound { // XXX: in the loop we'll handle changes better...
-				log.Printf("Failed to find service: %v\n", service)
-				invalid = true // XXX ?
-			}
-		}
-
-		if previous != invalid { // if invalid changed, send signal
-			send = true
-		}
-
-		if invalid {
-			log.Printf("Waiting for: %v\n", service) // waiting for service to appear...
-			if activeSet {
-				activeSet = false
-				set.Remove(service) // no return value should ever occur
-			}
-
-			select {
-			case _ = <-buschan: // XXX wait for new units event to unstick
-				// loop so that we can see the changed invalid signal
-				log.Printf("Service[%v]->DaemonReload()\n", service)
-
-			case exit := <-obj.Events:
-				if exit == "exit" {
-					return
-				} else {
-					log.Fatal("Unknown event: %v\n", exit)
-				}
-			}
-		} else {
-			if !activeSet {
-				activeSet = true
-				set.Add(service) // no return value should ever occur
-			}
-
-			log.Printf("Watching: %v\n", service) // attempting to watch...
-			select {
-			case event := <-subChannel:
-
-				log.Printf("Service event: %+v\n", event)
-				// NOTE: the value returned is a map for some reason...
-				if event[service] != nil {
-					// event[service].ActiveState is not nil
-					if event[service].ActiveState == "active" {
-						log.Printf("Service[%v]->Started()\n", service)
-					} else if event[service].ActiveState == "inactive" {
-						log.Printf("Service[%v]->Stopped!()\n", service)
-					} else {
-						log.Fatal("Unknown service state: ", event[service].ActiveState)
-					}
-				} else {
-					// service stopped (and ActiveState is nil...)
-					log.Printf("Service[%v]->Stopped\n", service)
-				}
-				send = true
-
-			case err := <-subErrors:
-				log.Println("error:", err)
-				log.Fatal(err)
-				v.Events <- fmt.Sprintf("service: %v", "error")
-
-			case exit := <-obj.Events:
-				if exit == "exit" {
-					return
-				} else {
-					log.Fatal("Unknown event: %v\n", exit)
-				}
-			}
-		}
-
-		if send {
-			send = false
-			//log.Println("Sending event!")
-			v.Events <- fmt.Sprintf("service(%v): %v", obj.Name, "event!") // FIXME: use struct
-		}
-	}
-}
-
-func (obj ServiceType) Exit() bool {
-	obj.Events <- "exit"
-	return true
-}
-
-func (obj ServiceType) StateOK() bool {
-
-	if !util.IsRunningSystemd() {
-		log.Fatal("Systemd is not running.")
-	}
-
-	conn, err := systemd.NewSystemdConnection() // needs root access
-	if err != nil {
-		log.Fatal("Failed to connect to systemd: ", err)
-	}
-	defer conn.Close()
-
-	var service = fmt.Sprintf("%v.service", obj.Name) // systemd name
-
-	loadstate, err := conn.GetUnitProperty(service, "LoadState")
-	if err != nil {
-		log.Printf("Failed to get load state: %v\n", err)
-		return false
-	}
-
-	// NOTE: we have to compare variants with other variants, they are really strings...
-	var notFound = (loadstate.Value == dbus.MakeVariant("not-found"))
-	if notFound {
-		log.Printf("Failed to find service: %v\n", service)
-		return false
-	}
-
-	// XXX: check service "enabled at boot" or not status...
-
-	//conn.GetUnitProperties(service)
-	activestate, err := conn.GetUnitProperty(service, "ActiveState")
-	if err != nil {
-		log.Fatal("Failed to get active state: ", err)
-	}
-
-	var running = (activestate.Value == dbus.MakeVariant("active"))
-
-	if obj.State == "running" {
-		if !running {
-			return false // we are in the wrong state
-		}
-	} else if obj.State == "stopped" {
-		if running {
-			return false
-		}
-	} else {
-		log.Fatal("Unknown state: ", obj.State)
-	}
-
-	return true // all is good, no state change needed
-}
-
-func (obj ServiceType) Apply() bool {
-	fmt.Printf("Apply->%v[%v]\n", obj.Type, obj.Name)
-
-	if !util.IsRunningSystemd() {
-		log.Fatal("Systemd is not running.")
-	}
-
-	conn, err := systemd.NewSystemdConnection() // needs root access
-	if err != nil {
-		log.Fatal("Failed to connect to systemd: ", err)
-	}
-	defer conn.Close()
-
-	var service = fmt.Sprintf("%v.service", obj.Name) // systemd name
-	var files = []string{service}                     // the service represented in a list
-	if obj.Startup == "enabled" {
-		_, _, err = conn.EnableUnitFiles(files, false, true)
-
-	} else if obj.Startup == "disabled" {
-		_, err = conn.DisableUnitFiles(files, false)
-	} else {
-		err = nil
-	}
-	if err != nil {
-		log.Printf("Unable to change startup status: %v\n", err)
-		return false
-	}
-
-	result := make(chan string, 1) // catch result information
-
-	if obj.State == "running" {
-		_, err := conn.StartUnit(service, "fail", result)
-		if err != nil {
-			log.Fatal("Failed to start unit: ", err)
-			return false
-		}
-	} else if obj.State == "stopped" {
-		_, err = conn.StopUnit(service, "fail", result)
-		if err != nil {
-			log.Fatal("Failed to stop unit: ", err)
-			return false
-		}
-	} else {
-		log.Fatal("Unknown state: ", obj.State)
-	}
-
-	status := <-result
-	if &status == nil {
-		log.Fatal("Result is nil")
-		return false
-	}
-	if status != "done" {
-		log.Fatal("Unknown return string: ", status)
-		return false
-	}
-
-	// XXX: also set enabled on boot
-
-	return true
-}

spec.in

@@ -0,0 +1,63 @@
+%global project_version __VERSION__
+%define debug_package %{nil}
+
+Name:		__PROGRAM__
+Version:	__VERSION__
+Release:	__RELEASE__
+Summary:	A next generation config management prototype!
+License:	AGPLv3+
+URL:		https://github.com/purpleidea/mgmt
+Source0:	https://dl.fedoraproject.org/pub/alt/purpleidea/__PROGRAM__/SOURCES/__PROGRAM__-%{project_version}.tar.bz2
+
+# graphviz should really be a "suggests", since technically it's optional
+Requires: graphviz
+
+BuildRequires:	golang
+BuildRequires:	golang-googlecode-tools-stringer
+BuildRequires:	git-core
+BuildRequires:	mercurial
+
+%description
+A next generation config management prototype!
+
+%prep
+%setup
+
+%build
+# FIXME: in the future, these could be vendor-ed in
+mkdir -p vendor/
+export GOPATH=`pwd`/vendor/
+go get github.com/coreos/etcd/client
+go get gopkg.in/yaml.v2
+go get gopkg.in/fsnotify.v1
+go get github.com/codegangsta/cli
+go get github.com/coreos/go-systemd/dbus
+go get github.com/coreos/go-systemd/util
+make build
+
+%install
+rm -rf %{buildroot}
+# _datadir is typically /usr/share/
+install -d -m 0755 %{buildroot}/%{_datadir}/__PROGRAM__/
+cp -a AUTHORS COPYING COPYRIGHT DOCUMENTATION.md README.md THANKS examples/ %{buildroot}/%{_datadir}/__PROGRAM__/
+
+# install the binary
+mkdir -p %{buildroot}/%{_bindir}
+install -m 0755 __PROGRAM__ %{buildroot}/%{_bindir}/__PROGRAM__
+
+# profile.d bash completion
+mkdir -p %{buildroot}%{_sysconfdir}/profile.d
+install misc/bashrc.sh -m 0755 %{buildroot}%{_sysconfdir}/profile.d/__PROGRAM__.sh
+
+# etc dir
+mkdir -p %{buildroot}%{_sysconfdir}/__PROGRAM__/
+install -m 0644 misc/example.conf %{buildroot}%{_sysconfdir}/__PROGRAM__/__PROGRAM__.conf
+
+%files
+%attr(0755, root, root) %{_sysconfdir}/profile.d/__PROGRAM__.sh
+%{_datadir}/__PROGRAM__/*
+%{_bindir}/__PROGRAM__
+%{_sysconfdir}/__PROGRAM__/*
+
+# this changelog is auto-generated by git log
+%changelog

svc.go

@@ -0,0 +1,436 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+// DOCS: https://godoc.org/github.com/coreos/go-systemd/dbus
+
+package main
+
+import (
+	"encoding/gob"
+	"errors"
+	"fmt"
+	systemd "github.com/coreos/go-systemd/dbus" // change namespace
+	systemdUtil "github.com/coreos/go-systemd/util"
+	"github.com/godbus/dbus" // namespace collides with systemd wrapper
+	"log"
+)
+
+func init() {
+	gob.Register(&SvcRes{})
+}
+
+type SvcRes struct {
+	BaseRes `yaml:",inline"`
+	State   string `yaml:"state"`   // state: running, stopped, undefined
+	Startup string `yaml:"startup"` // enabled, disabled, undefined
+}
+
+func NewSvcRes(name, state, startup string) *SvcRes {
+	obj := &SvcRes{
+		BaseRes: BaseRes{
+			Name: name,
+		},
+		State:   state,
+		Startup: startup,
+	}
+	obj.Init()
+	return obj
+}
+
+func (obj *SvcRes) Init() {
+	obj.BaseRes.kind = "Svc"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+}
+
+func (obj *SvcRes) Validate() bool {
+	if obj.State != "running" && obj.State != "stopped" && obj.State != "" {
+		return false
+	}
+	if obj.Startup != "enabled" && obj.Startup != "disabled" && obj.Startup != "" {
+		return false
+	}
+	return true
+}
+
+// Service watcher
+func (obj *SvcRes) Watch(processChan chan Event) {
+	if obj.IsWatching() {
+		return
+	}
+	obj.SetWatching(true)
+	defer obj.SetWatching(false)
+
+	// obj.Name: svc name
+	//vertex := obj.GetVertex()         // stored with SetVertex
+	if !systemdUtil.IsRunningSystemd() {
+		log.Fatal("Systemd is not running.")
+	}
+
+	conn, err := systemd.NewSystemdConnection() // needs root access
+	if err != nil {
+		log.Fatal("Failed to connect to systemd: ", err)
+	}
+	defer conn.Close()
+
+	// if we share the bus with others, we will get each others messages!!
+	bus, err := SystemBusPrivateUsable() // don't share the bus connection!
+	if err != nil {
+		log.Fatal("Failed to connect to bus: ", err)
+	}
+
+	// XXX: will this detect new units?
+	bus.BusObject().Call("org.freedesktop.DBus.AddMatch", 0,
+		"type='signal',interface='org.freedesktop.systemd1.Manager',member='Reloading'")
+	buschan := make(chan *dbus.Signal, 10)
+	bus.Signal(buschan)
+
+	var svc = fmt.Sprintf("%v.service", obj.Name) // systemd name
+	var send = false                              // send event?
+	var exit = false
+	var dirty = false
+	var invalid = false              // does the svc exist or not?
+	var previous bool                // previous invalid value
+	set := conn.NewSubscriptionSet() // no error should be returned
+	subChannel, subErrors := set.Subscribe()
+	var activeSet = false
+
+	for {
+		// XXX: watch for an event for new units...
+		// XXX: detect if startup enabled/disabled value changes...
+
+		previous = invalid
+		invalid = false
+
+		// firstly, does svc even exist or not?
+		loadstate, err := conn.GetUnitProperty(svc, "LoadState")
+		if err != nil {
+			log.Printf("Failed to get property: %v", err)
+			invalid = true
+		}
+
+		if !invalid {
+			var notFound = (loadstate.Value == dbus.MakeVariant("not-found"))
+			if notFound { // XXX: in the loop we'll handle changes better...
+				log.Printf("Failed to find svc: %v", svc)
+				invalid = true // XXX ?
+			}
+		}
+
+		if previous != invalid { // if invalid changed, send signal
+			send = true
+			dirty = true
+		}
+
+		if invalid {
+			log.Printf("Waiting for: %v", svc) // waiting for svc to appear...
+			if activeSet {
+				activeSet = false
+				set.Remove(svc) // no return value should ever occur
+			}
+
+			obj.SetState(resStateWatching) // reset
+			select {
+			case _ = <-buschan: // XXX wait for new units event to unstick
+				obj.SetConvergedState(resConvergedNil)
+				// loop so that we can see the changed invalid signal
+				log.Printf("Svc[%v]->DaemonReload()", svc)
+
+			case event := <-obj.events:
+				obj.SetConvergedState(resConvergedNil)
+				if exit, send = obj.ReadEvent(&event); exit {
+					return // exit
+				}
+				if event.GetActivity() {
+					dirty = true
+				}
+
+			case _ = <-TimeAfterOrBlock(obj.ctimeout):
+				obj.SetConvergedState(resConvergedTimeout)
+				obj.converged <- true
+				continue
+			}
+		} else {
+			if !activeSet {
+				activeSet = true
+				set.Add(svc) // no return value should ever occur
+			}
+
+			log.Printf("Watching: %v", svc) // attempting to watch...
+			obj.SetState(resStateWatching)  // reset
+			select {
+			case event := <-subChannel:
+
+				log.Printf("Svc event: %+v", event)
+				// NOTE: the value returned is a map for some reason...
+				if event[svc] != nil {
+					// event[svc].ActiveState is not nil
+					if event[svc].ActiveState == "active" {
+						log.Printf("Svc[%v]->Started()", svc)
+					} else if event[svc].ActiveState == "inactive" {
+						log.Printf("Svc[%v]->Stopped!()", svc)
+					} else {
+						log.Fatal("Unknown svc state: ", event[svc].ActiveState)
+					}
+				} else {
+					// svc stopped (and ActiveState is nil...)
+					log.Printf("Svc[%v]->Stopped", svc)
+				}
+				send = true
+				dirty = true
+
+			case err := <-subErrors:
+				obj.SetConvergedState(resConvergedNil) // XXX ?
+				log.Printf("error: %v", err)
+				log.Fatal(err)
+				//vertex.events <- fmt.Sprintf("svc: %v", "error") // XXX: how should we handle errors?
+
+			case event := <-obj.events:
+				obj.SetConvergedState(resConvergedNil)
+				if exit, send = obj.ReadEvent(&event); exit {
+					return // exit
+				}
+				if event.GetActivity() {
+					dirty = true
+				}
+			}
+		}
+
+		if send {
+			send = false
+			if dirty {
+				dirty = false
+				obj.isStateOK = false // something made state dirty
+			}
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
+		}
+
+	}
+}
+
+func (obj *SvcRes) CheckApply(apply bool) (stateok bool, err error) {
+	log.Printf("%v[%v]: CheckApply(%t)", obj.Kind(), obj.GetName(), apply)
+
+	if obj.isStateOK { // cache the state
+		return true, nil
+	}
+
+	if !systemdUtil.IsRunningSystemd() {
+		return false, errors.New("Systemd is not running.")
+	}
+
+	conn, err := systemd.NewSystemdConnection() // needs root access
+	if err != nil {
+		return false, fmt.Errorf("Failed to connect to systemd: %v", err)
+	}
+	defer conn.Close()
+
+	var svc = fmt.Sprintf("%v.service", obj.Name) // systemd name
+
+	loadstate, err := conn.GetUnitProperty(svc, "LoadState")
+	if err != nil {
+		return false, fmt.Errorf("Failed to get load state: %v", err)
+	}
+
+	// NOTE: we have to compare variants with other variants, they are really strings...
+	var notFound = (loadstate.Value == dbus.MakeVariant("not-found"))
+	if notFound {
+		return false, fmt.Errorf("Failed to find svc: %v", svc)
+	}
+
+	// XXX: check svc "enabled at boot" or not status...
+
+	//conn.GetUnitProperties(svc)
+	activestate, err := conn.GetUnitProperty(svc, "ActiveState")
+	if err != nil {
+		return false, fmt.Errorf("Failed to get active state: %v", err)
+	}
+
+	var running = (activestate.Value == dbus.MakeVariant("active"))
+	var stateOK = ((obj.State == "") || (obj.State == "running" && running) || (obj.State == "stopped" && !running))
+	var startupOK = true // XXX DETECT AND SET
+
+	if stateOK && startupOK {
+		return true, nil // we are in the correct state
+	}
+
+	// state is not okay, no work done, exit, but without error
+	if !apply {
+		return false, nil
+	}
+
+	// apply portion
+	log.Printf("%v[%v]: Apply", obj.Kind(), obj.GetName())
+	var files = []string{svc} // the svc represented in a list
+	if obj.Startup == "enabled" {
+		_, _, err = conn.EnableUnitFiles(files, false, true)
+
+	} else if obj.Startup == "disabled" {
+		_, err = conn.DisableUnitFiles(files, false)
+	}
+
+	if err != nil {
+		return false, fmt.Errorf("Unable to change startup status: %v", err)
+	}
+
+	// XXX: do we need to use a buffered channel here?
+	result := make(chan string, 1) // catch result information
+
+	if obj.State == "running" {
+		_, err = conn.StartUnit(svc, "fail", result)
+		if err != nil {
+			return false, fmt.Errorf("Failed to start unit: %v", err)
+		}
+	} else if obj.State == "stopped" {
+		_, err = conn.StopUnit(svc, "fail", result)
+		if err != nil {
+			return false, fmt.Errorf("Failed to stop unit: %v", err)
+		}
+	}
+
+	status := <-result
+	if &status == nil {
+		return false, errors.New("Systemd service action result is nil")
+	}
+	if status != "done" {
+		return false, fmt.Errorf("Unknown systemd return string: %v", status)
+	}
+
+	// XXX: also set enabled on boot
+
+	return false, nil // success
+}
+
+type SvcUUID struct {
+	// NOTE: there is also a name variable in the BaseUUID struct, this is
+	// information about where this UUID came from, and is unrelated to the
+	// information about the resource we're matching. That data which is
+	// used in the IFF function, is what you see in the struct fields here.
+	BaseUUID
+	name string // the svc name
+}
+
+// if and only if they are equivalent, return true
+// if they are not equivalent, return false
+func (obj *SvcUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*SvcUUID)
+	if !ok {
+		return false
+	}
+	return obj.name == res.name
+}
+
+type SvcResAutoEdges struct {
+	data    []ResUUID
+	pointer int
+	found   bool
+}
+
+func (obj *SvcResAutoEdges) Next() []ResUUID {
+	if obj.found {
+		log.Fatal("Shouldn't be called anymore!")
+	}
+	if len(obj.data) == 0 { // check length for rare scenarios
+		return nil
+	}
+	value := obj.data[obj.pointer]
+	obj.pointer++
+	return []ResUUID{value} // we return one, even though api supports N
+}
+
+// get results of the earlier Next() call, return if we should continue!
+func (obj *SvcResAutoEdges) Test(input []bool) bool {
+	// if there aren't any more remaining
+	if len(obj.data) <= obj.pointer {
+		return false
+	}
+	if obj.found { // already found, done!
+		return false
+	}
+	if len(input) != 1 { // in case we get given bad data
+		log.Fatal("Expecting a single value!")
+	}
+	if input[0] { // if a match is found, we're done!
+		obj.found = true // no more to find!
+		return false
+	}
+	return true // keep going
+}
+
+func (obj *SvcRes) AutoEdges() AutoEdge {
+	var data []ResUUID
+	svcFiles := []string{
+		fmt.Sprintf("/etc/systemd/system/%s.service", obj.Name),     // takes precedence
+		fmt.Sprintf("/usr/lib/systemd/system/%s.service", obj.Name), // pkg default
+	}
+	for _, x := range svcFiles {
+		var reversed = true
+		data = append(data, &FileUUID{
+			BaseUUID: BaseUUID{
+				name:     obj.GetName(),
+				kind:     obj.Kind(),
+				reversed: &reversed,
+			},
+			path: x, // what matters
+		})
+	}
+	return &FileResAutoEdges{
+		data:    data,
+		pointer: 0,
+		found:   false,
+	}
+}
+
+// include all params to make a unique identification of this object
+func (obj *SvcRes) GetUUIDs() []ResUUID {
+	x := &SvcUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		name:     obj.Name, // svc name
+	}
+	return []ResUUID{x}
+}
+
+func (obj *SvcRes) GroupCmp(r Res) bool {
+	_, ok := r.(*SvcRes)
+	if !ok {
+		return false
+	}
+	// TODO: depending on if the systemd service api allows batching, we
+	// might be able to build this, although not sure how useful it is...
+	// it might just eliminate parallelism be bunching up the graph
+	return false // not possible atm
+}
+
+func (obj *SvcRes) Compare(res Res) bool {
+	switch res.(type) {
+	case *SvcRes:
+		res := res.(*SvcRes)
+		if obj.Name != res.Name {
+			return false
+		}
+		if obj.State != res.State {
+			return false
+		}
+		if obj.Startup != res.Startup {
+			return false
+		}
+	default:
+		return false
+	}
+	return true
+}

tag.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+# TODO: don't run if current HEAD is already tagged (ensure this is idempotent)
+# take current HEAD with new version
+v=`git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --abbrev=0`
+t=`echo "${v%.*}.$((${v##*.}+1))"`	# increment version
+echo "Version $t is now tagged!"
+echo "Pushing $t to origin..."
+echo "Press ^C within 3s to abort."
+sleep 3s
+git tag $t
+git push origin $t

test.sh

@@ -1,7 +1,8 @@
 #!/bin/bash -e
-
 # test suite...
 echo running test.sh
+echo "ENV:"
+env
 
 # ensure there is no trailing whitespace or other whitespace errors
 git diff-tree --check $(git hash-object -t tree /dev/null) HEAD
@@ -9,3 +10,25 @@ git diff-tree --check $(git hash-object -t tree /dev/null) HEAD
 # ensure entries to authors file are sorted
 start=$(($(grep -n '^[[:space:]]*$' AUTHORS | awk -F ':' '{print $1}' | head -1) + 1))
 diff <(tail -n +$start AUTHORS | sort) <(tail -n +$start AUTHORS)
+
+./test/test-gofmt.sh
+./test/test-yamlfmt.sh
+./test/test-bashfmt.sh
+./test/test-headerfmt.sh
+go test
+./test/test-govet.sh
+
+# do these longer tests only when running on ci
+if env | grep -q -e '^TRAVIS=true$' -e '^JENKINS_URL=' -e '^BUILD_TAG=jenkins'; then
+	go test -race
+	./test/test-shell.sh
+else
+	# FIXME: this fails on travis for some reason
+	./test/test-reproducible.sh
+fi
+
+# run omv tests on jenkins physical hosts only
+if env | grep -q -e '^JENKINS_URL=' -e '^BUILD_TAG=jenkins'; then
+	./test/test-omv.sh
+fi
+./test/test-golint.sh	# test last, because this test is somewhat arbitrary

test/omv/helloworld.yaml

@@ -0,0 +1,52 @@
+---
+:domain: example.com
+:network: 192.168.123.0/24
+:image: centos-7.1
+:cpus: ''
+:memory: ''
+:disks: 0
+:disksize: 40G
+:boxurlprefix: ''
+:sync: rsync
+:syncdir: ''
+:syncsrc: ''
+:folder: ".omv"
+:extern:
+- type: git
+  repository: https://github.com/purpleidea/mgmt
+  directory: mgmt
+:cd: ''
+:puppet: false
+:classes: []
+:shell:
+- mkdir /tmp/mgmt/
+:docker: false
+:kubernetes: false
+:ansible: []
+:playbook: []
+:ansible_extras: {}
+:cachier: false
+:vms:
+- :name: etcd
+  :shell:
+  - iptables -F
+  - cd /vagrant/mgmt/ && make path
+  - cd /vagrant/mgmt/ && make deps && make build && cp mgmt ~/bin/
+  - etcd -bind-addr "`hostname --ip-address`:2379" &
+  - cd && mgmt --help
+:namespace: omv
+:count: 0
+:username: ''
+:password: ''
+:poolid: true
+:repos: []
+:update: false
+:reboot: false
+:unsafe: false
+:nested: false
+:tests:
+- omv up etcd
+- vssh root@etcd -c pidof etcd
+- omv destroy
+:comment: simple hello world test case for mgmt
+:reallyrm: false

test/omv/pkg1a.yaml

@@ -0,0 +1,52 @@
+---
+:domain: example.com
+:network: 192.168.123.0/24
+:image: centos-7.1
+:cpus: ''
+:memory: ''
+:disks: 0
+:disksize: 40G
+:boxurlprefix: ''
+:sync: rsync
+:syncdir: ''
+:syncsrc: ''
+:folder: ".omv"
+:extern:
+- type: git
+  repository: https://github.com/purpleidea/mgmt
+  directory: mgmt
+:cd: ''
+:puppet: false
+:classes: []
+:shell:
+- mkdir /tmp/mgmt/
+:docker: false
+:kubernetes: false
+:ansible: []
+:playbook: []
+:ansible_extras: {}
+:cachier: false
+:vms:
+- :name: mgmt1
+  :shell:
+  - iptables -F
+  - cd /vagrant/mgmt/ && make path
+  - cd /vagrant/mgmt/ && make deps && make build && cp mgmt ~/bin/
+  - etcd -bind-addr "`hostname --ip-address`:2379" &
+  - cd && mgmt run --file /vagrant/mgmt/examples/pkg1.yaml --converged-timeout=5
+:namespace: omv
+:count: 0
+:username: ''
+:password: ''
+:poolid: true
+:repos: []
+:update: false
+:reboot: false
+:unsafe: false
+:nested: false
+:tests:
+- omv up
+- vssh root@mgmt1 -c which powertop
+- omv destroy
+:comment: simple package install test case
+:reallyrm: false

test/omv/pkg1b.yaml

@@ -0,0 +1,53 @@
+---
+:domain: example.com
+:network: 192.168.123.0/24
+:image: debian-8
+:cpus: ''
+:memory: ''
+:disks: 0
+:disksize: 40G
+:boxurlprefix: ''
+:sync: rsync
+:syncdir: ''
+:syncsrc: ''
+:folder: ".omv"
+:extern:
+- type: git
+  repository: https://github.com/purpleidea/mgmt
+  directory: mgmt
+:cd: ''
+:puppet: false
+:classes: []
+:shell:
+- mkdir /tmp/mgmt/
+:docker: false
+:kubernetes: false
+:ansible: []
+:playbook: []
+:ansible_extras: {}
+:cachier: false
+:vms:
+- :name: mgmt1
+  :shell:
+  - apt-get install -y make
+  - iptables -F
+  - cd /vagrant/mgmt/ && make path
+  - cd /vagrant/mgmt/ && make deps && make build && cp mgmt ~/bin/
+  - etcd -bind-addr "`hostname --ip-address`:2379" &
+  - cd && mgmt run --file /vagrant/mgmt/examples/pkg1.yaml --converged-timeout=5
+:namespace: omv
+:count: 0
+:username: ''
+:password: ''
+:poolid: true
+:repos: []
+:update: false
+:reboot: false
+:unsafe: false
+:nested: false
+:tests:
+- omv up
+- vssh root@mgmt1 -c which powertop
+- omv destroy
+:comment: simple package install test case
+:reallyrm: false

test/shell/.gitignore

@@ -0,0 +1 @@
+mgmt

test/shell/etcd.sh

@@ -0,0 +1,23 @@
+# NOTE: boiler plate to run etcd; source with: . etcd.sh; should NOT be +x
+cleanup ()
+{
+	echo "cleanup: $1"
+	killall etcd || killall -9 etcd || true	# kill etcd
+	rm -rf /tmp/etcd/
+}
+
+trap_with_arg() {
+	func="$1"
+	shift
+	for sig in "$@"
+	do
+		trap "$func $sig" "$sig"
+	done
+}
+
+trap_with_arg cleanup INT QUIT TERM EXIT	#	ERR
+mkdir -p /tmp/etcd/
+cd /tmp/etcd/ >/dev/null	# shush the cd operation
+etcd &	# start etcd as job # 1
+sleep 1s	# let etcd startup
+cd - >/dev/null

test/shell/t1.sh

@@ -0,0 +1,14 @@
+#!/bin/bash -e
+# NOTES:
+#	* this is a simple shell based `mgmt` test case
+#	* it is recommended that you run mgmt wrapped in the timeout command
+#	* it is recommended that you run mgmt with --no-watch
+#	* it is recommended that you run mgmt --converged-timeout=<seconds>
+#	* you can run mgmt with --max-runtime=<seconds> in special scenarios
+#	* you can get a separate etcd going by sourcing etcd.sh: . etcd.sh
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+timeout --kill-after=3s 1s ./mgmt --help # hello world!

test/shell/t2.sh

@@ -0,0 +1,19 @@
+#!/bin/bash -e
+
+if env | grep -q -e '^TRAVIS=true$'; then
+	# inotify doesn't seem to work properly on travis
+	echo "Travis and Jenkins give wonky results here, skipping test!"
+	exit
+fi
+
+. etcd.sh	# start etcd as job # 1
+
+# run till completion
+timeout --kill-after=15s 10s ./mgmt run --file t2.yaml --converged-timeout=5 --no-watch &
+
+. wait.sh	# wait for everything except etcd
+
+test -e /tmp/mgmt/f1
+test -e /tmp/mgmt/f2
+test -e /tmp/mgmt/f3
+test ! -e /tmp/mgmt/f4

test/shell/t2.yaml

@@ -1,41 +1,41 @@
 ---
 graph: mygraph
-types:
+resources:
   noop:
   - name: noop1
   file:
   - name: file1
-    path: /tmp/mgmt/f1
+    path: "/tmp/mgmt/f1"
     content: |
       i am f1
     state: exists
   - name: file2
-    path: /tmp/mgmt/f2
+    path: "/tmp/mgmt/f2"
     content: |
       i am f2
     state: exists
   - name: file3
-    path: /tmp/mgmt/f3
+    path: "/tmp/mgmt/f3"
     content: |
       i am f3
     state: exists
   - name: file4
-    path: /tmp/mgmt/f4
+    path: "/tmp/mgmt/f4"
     content: |
       i am f4 and i should not be here
     state: absent
 edges:
 - name: e1
   from:
-    type: file
+    res: file
     name: file1
   to:
-    type: file
+    res: file
     name: file2
 - name: e2
   from:
-    type: file
+    res: file
     name: file2
   to:
-    type: file
+    res: file
     name: file3

test/shell/t3-a.yaml

@@ -0,0 +1,28 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1a
+    path: "/tmp/mgmt/mgmtA/f1a"
+    content: |
+      i am f1
+    state: exists
+  - name: file2a
+    path: "/tmp/mgmt/mgmtA/f2a"
+    content: |
+      i am f2
+    state: exists
+  - name: "@@file3a"
+    path: "/tmp/mgmt/mgmtA/f3a"
+    content: |
+      i am f3, exported from host A
+    state: exists
+  - name: "@@file4a"
+    path: "/tmp/mgmt/mgmtA/f4a"
+    content: |
+      i am f4, exported from host A
+    state: exists
+collect:
+- res: file
+  pattern: "/tmp/mgmt/mgmtA/"
+edges: []

test/shell/t3-b.yaml

@@ -0,0 +1,28 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1b
+    path: "/tmp/mgmt/mgmtB/f1b"
+    content: |
+      i am f1
+    state: exists
+  - name: file2b
+    path: "/tmp/mgmt/mgmtB/f2b"
+    content: |
+      i am f2
+    state: exists
+  - name: "@@file3b"
+    path: "/tmp/mgmt/mgmtB/f3b"
+    content: |
+      i am f3, exported from host B
+    state: exists
+  - name: "@@file4b"
+    path: "/tmp/mgmt/mgmtB/f4b"
+    content: |
+      i am f4, exported from host B
+    state: exists
+collect:
+- res: file
+  pattern: "/tmp/mgmt/mgmtB/"
+edges: []

test/shell/t3-c.yaml

@@ -0,0 +1,28 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1c
+    path: "/tmp/mgmt/mgmtC/f1c"
+    content: |
+      i am f1
+    state: exists
+  - name: file2c
+    path: "/tmp/mgmt/mgmtC/f2c"
+    content: |
+      i am f2
+    state: exists
+  - name: "@@file3c"
+    path: "/tmp/mgmt/mgmtC/f3c"
+    content: |
+      i am f3, exported from host C
+    state: exists
+  - name: "@@file4c"
+    path: "/tmp/mgmt/mgmtC/f4c"
+    content: |
+      i am f4, exported from host C
+    state: exists
+collect:
+- res: file
+  pattern: "/tmp/mgmt/mgmtC/"
+edges: []

test/shell/t3.sh

@@ -0,0 +1,73 @@
+#!/bin/bash -e
+
+if env | grep -q -e '^TRAVIS=true$'; then
+	# inotify doesn't seem to work properly on travis
+	echo "Travis and Jenkins give wonky results here, skipping test!"
+	exit
+fi
+
+. etcd.sh	# start etcd as job # 1
+
+# setup
+mkdir -p "${MGMT_TMPDIR}"mgmt{A..C}
+
+# run till completion
+timeout --kill-after=15s 10s ./mgmt run --file t3-a.yaml --converged-timeout=5 --no-watch &
+timeout --kill-after=15s 10s ./mgmt run --file t3-b.yaml --converged-timeout=5 --no-watch &
+timeout --kill-after=15s 10s ./mgmt run --file t3-c.yaml --converged-timeout=5 --no-watch &
+
+. wait.sh	# wait for everything except etcd
+
+# A: collected
+test -e "${MGMT_TMPDIR}"mgmtA/f3b
+test -e "${MGMT_TMPDIR}"mgmtA/f3c
+test -e "${MGMT_TMPDIR}"mgmtA/f4b
+test -e "${MGMT_TMPDIR}"mgmtA/f4c
+
+# A: local
+test -e "${MGMT_TMPDIR}"mgmtA/f1a
+test -e "${MGMT_TMPDIR}"mgmtA/f2a
+test -e "${MGMT_TMPDIR}"mgmtA/f3a
+test -e "${MGMT_TMPDIR}"mgmtA/f4a
+
+# A: nope!
+test ! -e "${MGMT_TMPDIR}"mgmtA/f1b
+test ! -e "${MGMT_TMPDIR}"mgmtA/f2b
+test ! -e "${MGMT_TMPDIR}"mgmtA/f1c
+test ! -e "${MGMT_TMPDIR}"mgmtA/f2c
+
+# B: collected
+test -e "${MGMT_TMPDIR}"mgmtB/f3a
+test -e "${MGMT_TMPDIR}"mgmtB/f3c
+test -e "${MGMT_TMPDIR}"mgmtB/f4a
+test -e "${MGMT_TMPDIR}"mgmtB/f4c
+
+# B: local
+test -e "${MGMT_TMPDIR}"mgmtB/f1b
+test -e "${MGMT_TMPDIR}"mgmtB/f2b
+test -e "${MGMT_TMPDIR}"mgmtB/f3b
+test -e "${MGMT_TMPDIR}"mgmtB/f4b
+
+# B: nope!
+test ! -e "${MGMT_TMPDIR}"mgmtB/f1a
+test ! -e "${MGMT_TMPDIR}"mgmtB/f2a
+test ! -e "${MGMT_TMPDIR}"mgmtB/f1c
+test ! -e "${MGMT_TMPDIR}"mgmtB/f2c
+
+# C: collected
+test -e "${MGMT_TMPDIR}"mgmtC/f3a
+test -e "${MGMT_TMPDIR}"mgmtC/f3b
+test -e "${MGMT_TMPDIR}"mgmtC/f4a
+test -e "${MGMT_TMPDIR}"mgmtC/f4b
+
+# C: local
+test -e "${MGMT_TMPDIR}"mgmtC/f1c
+test -e "${MGMT_TMPDIR}"mgmtC/f2c
+test -e "${MGMT_TMPDIR}"mgmtC/f3c
+test -e "${MGMT_TMPDIR}"mgmtC/f4c
+
+# C: nope!
+test ! -e "${MGMT_TMPDIR}"mgmtC/f1a
+test ! -e "${MGMT_TMPDIR}"mgmtC/f2a
+test ! -e "${MGMT_TMPDIR}"mgmtC/f1b
+test ! -e "${MGMT_TMPDIR}"mgmtC/f2b

test/shell/t4.sh

@@ -0,0 +1,10 @@
+#!/bin/bash -e
+
+. etcd.sh	# start etcd as job # 1
+
+# should take slightly more than 25s, but fail if we take 35s)
+timeout --kill-after=35s 30s ./mgmt run --file t4.yaml --converged-timeout=5 --no-watch &
+
+#jobs	# etcd is 1
+#wait -n 2	# wait for mgmt to exit
+. wait.sh	# wait for everything except etcd

test/shell/t4.yaml

@@ -0,0 +1,77 @@
+---
+graph: mygraph
+comment: simple exec fan in example to demonstrate optimization)
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec3
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec4
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec5
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    res: exec
+    name: exec1
+  to:
+    res: exec
+    name: exec5
+- name: e2
+  from:
+    res: exec
+    name: exec2
+  to:
+    res: exec
+    name: exec5
+- name: e3
+  from:
+    res: exec
+    name: exec3
+  to:
+    res: exec
+    name: exec5

test/shell/t5.sh

@@ -0,0 +1,10 @@
+#!/bin/bash -e
+
+. etcd.sh	# start etcd as job # 1
+
+# should take slightly more than 35s, but fail if we take 45s)
+timeout --kill-after=45s 40s ./mgmt run --file t5.yaml --converged-timeout=5 --no-watch &
+
+#jobs	# etcd is 1
+#wait -n 2	# wait for mgmt to exit
+. wait.sh	# wait for everything except etcd

test/shell/t5.yaml

@@ -0,0 +1,128 @@
+---
+graph: mygraph
+comment: simple exec fan in to fan out example to demonstrate optimization
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec3
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec4
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec5
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec6
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec7
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec8
+    cmd: sleep 15s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    res: exec
+    name: exec1
+  to:
+    res: exec
+    name: exec4
+- name: e2
+  from:
+    res: exec
+    name: exec2
+  to:
+    res: exec
+    name: exec4
+- name: e3
+  from:
+    res: exec
+    name: exec3
+  to:
+    res: exec
+    name: exec4
+- name: e4
+  from:
+    res: exec
+    name: exec4
+  to:
+    res: exec
+    name: exec5
+- name: e5
+  from:
+    res: exec
+    name: exec4
+  to:
+    res: exec
+    name: exec6
+- name: e6
+  from:
+    res: exec
+    name: exec4
+  to:
+    res: exec
+    name: exec7

test/shell/t6.sh

@@ -0,0 +1,33 @@
+#!/bin/bash -e
+
+if env | grep -q -e '^TRAVIS=true$'; then
+	# inotify doesn't seem to work properly on travis
+	echo "Travis and Jenkins give wonky results here, skipping test!"
+	exit
+fi
+
+. etcd.sh	# start etcd as job # 1
+
+# run till completion
+timeout --kill-after=20s 15s ./mgmt run --file t6.yaml --no-watch &
+sleep 1s	# let it converge
+test -e /tmp/mgmt/f1
+test -e /tmp/mgmt/f2
+test -e /tmp/mgmt/f3
+test ! -e /tmp/mgmt/f4
+rm -f /tmp/mgmt/f2
+sleep 0.1s	# let it converge or tests will fail
+test -e /tmp/mgmt/f2
+rm -f /tmp/mgmt/f2
+sleep 0.1s
+test -e /tmp/mgmt/f2
+echo foo > /tmp/mgmt/f2
+sleep 0.1s
+test "`cat /tmp/mgmt/f2`" = "i am f2"
+rm -f /tmp/mgmt/f2
+sleep 0.1s
+test -e /tmp/mgmt/f2
+
+killall -SIGINT mgmt	# send ^C to exit mgmt
+
+. wait.sh	# wait for everything except etcd

test/shell/t6.yaml

@@ -0,0 +1,41 @@
+---
+graph: mygraph
+resources:
+  noop:
+  - name: noop1
+  file:
+  - name: file1
+    path: "/tmp/mgmt/f1"
+    content: |
+      i am f1
+    state: exists
+  - name: file2
+    path: "/tmp/mgmt/f2"
+    content: |
+      i am f2
+    state: exists
+  - name: file3
+    path: "/tmp/mgmt/f3"
+    content: |
+      i am f3
+    state: exists
+  - name: file4
+    path: "/tmp/mgmt/f4"
+    content: |
+      i am f4 and i should not be here
+    state: absent
+edges:
+- name: e1
+  from:
+    res: file
+    name: file1
+  to:
+    res: file
+    name: file2
+- name: e2
+  from:
+    res: file
+    name: file2
+  to:
+    res: file
+    name: file3

test/shell/wait.sh

@@ -0,0 +1,9 @@
+# NOTE: boiler plate to wait on mgmt; source with: . wait.sh; should NOT be +x
+while test "`jobs -p`" != "" && test "`jobs -p`" != "`pidof etcd`"
+do
+	for j in `jobs -p`
+	do
+		[ "$j" = "`pidof etcd`" ] && continue	# don't wait for etcd
+		wait $j || continue	# wait for mgmt job $j
+	done
+done

test/test-bashfmt.sh

@@ -0,0 +1,31 @@
+#!/bin/bash
+# check for any bash files that aren't properly formatted
+# TODO: this is hardly exhaustive
+echo running test-bashfmt.sh
+set -o errexit
+set -o nounset
+set -o pipefail
+
+ROOT=$(dirname "${BASH_SOURCE}")/..
+
+cd "${ROOT}"
+
+find_files() {
+	git ls-files | grep -e '\.sh$' -e '\.bash$'
+}
+
+bad_files=$(
+	for i in $(find_files); do
+		# search for more than one leading space, to ensure we use tabs
+		if grep -q '^  ' "$i"; then
+			echo "$i"
+		fi
+	done
+)
+
+if [[ -n "${bad_files}" ]]; then
+	echo 'FAIL'
+	echo 'The following bash files are not properly formatted:'
+	echo "${bad_files}"
+	exit 1
+fi

test/test-gofmt.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 # original version of this script from kubernetes project, under ALv2 license
-
+echo running test-gofmt.sh
 set -o errexit
 set -o nounset
 set -o pipefail
@@ -9,7 +9,7 @@ ROOT=$(dirname "${BASH_SOURCE}")/..
 
 GO_VERSION=($(go version))
 
-if [[ -z $(echo "${GO_VERSION[2]}" | grep -E 'go1.2|go1.3|go1.4') ]]; then
+if [[ -z $(echo "${GO_VERSION[2]}" | grep -E 'go1.2|go1.3|go1.4|go1.5|go1.6') ]]; then
 	echo "Unknown go version '${GO_VERSION}', skipping gofmt."
 	exit 0
 fi
@@ -17,19 +17,14 @@ fi
 cd "${ROOT}"
 
 find_files() {
-	find . -not \( \
-		\( \
-			-wholename './old' \
-			-o -wholename './tmp' \
-		\) -prune \
-	\) -name '*.go'
+	git ls-files | grep '\.go$'
 }
 
 GOFMT="gofmt"	# we prefer to not use the -s flag, which is pretty annoying...
 bad_files=$(find_files | xargs $GOFMT -l)
 if [[ -n "${bad_files}" ]]; then
 	echo 'FAIL'
-	echo 'The following files are not properly formatted:'
+	echo 'The following golang files are not properly formatted:'
 	echo "${bad_files}"
 	exit 1
 fi

test/test-golint.sh

@@ -0,0 +1,67 @@
+#!/bin/bash
+# check that go lint passes or doesn't get worse by some threshold
+echo running test-golint.sh
+# TODO: output a diff of what has changed in the golint output
+# FIXME: test a range of commits, since only the last patch is checked here
+PREVIOUS='HEAD^'
+CURRENT='HEAD'
+THRESHOLD=15	# percent problems per new LOC
+XPWD=`pwd`
+ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && cd .. && pwd )"	# dir!
+cd "${ROOT}" >/dev/null
+
+# if this branch has more than one commit as compared to master, diff to that
+# note: this is a cheap way to avoid doing a fancy succession of golint's...
+HACK=''
+COMMITS="`git rev-list --count $CURRENT ^master`"	# commit delta to master
+# avoid: bad revision '^master' on travis for unknown reason :(
+if [ "$COMMITS" != "" ] && [ "$COMMITS" -gt "1" ]; then
+	PREVIOUS='master'
+	HACK="yes"
+fi
+
+LINT=`golint`	# current golint output
+COUNT=`echo -e "$LINT" | wc -l`	# number of golint problems in current branch
+[ "$LINT" = "" ] && echo PASS && exit	# everything is "perfect"
+
+T=`mktemp --tmpdir -d tmp.XXX`
+[ "$T" = "" ] && exit 1
+cd $T || exit 1
+git clone --recursive "${ROOT}" 2>/dev/null	# make a copy
+cd "`basename ${ROOT}`" >/dev/null || exit 1
+if [ "$HACK" != "" ]; then
+	# ensure master branch really exists when cloning from a branched repo!
+	git checkout master &>/dev/null && git checkout - &>/dev/null
+fi
+
+DIFF1=0
+NUMSTAT1=`git diff "$PREVIOUS" "$CURRENT" --numstat`	# numstat diff since previous commit
+while read -r line; do
+	add=`echo "$line" | cut -f1`
+	# TODO: should we only count added lines, or count the difference?
+	sum="$add"
+	#del=`echo "$line" | cut -f2`
+	#sum=`expr $add - $del`
+	DIFF1=`expr $DIFF1 + $sum`
+done <<< "$NUMSTAT1"	# three < is the secret to putting a variable into read
+
+git checkout "$PREVIOUS" &>/dev/null	# previous commit
+LINT1=`golint`
+COUNT1=`echo -e "$LINT1" | wc -l`	# number of golint problems in older branch
+
+# clean up
+cd "$XPWD" >/dev/null
+rm -rf "$T"
+
+DELTA=$(printf "%.0f\n" `echo - | awk "{ print (($COUNT - $COUNT1) / $DIFF1) * 100 }"`)
+
+echo "Lines of code: $DIFF1"
+echo "Prev. # of issues: $COUNT1"
+echo "Curr. # of issues: $COUNT"
+echo "Issue count delta is: $DELTA %"
+if [ "$DELTA" -gt "$THRESHOLD" ]; then
+	echo "Maximum threshold is: $THRESHOLD %"
+	echo '`golint` FAIL'
+	exit 1
+fi
+echo PASS

test/test-govet.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+# check that go vet passes
+echo running test-govet.sh
+ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && cd .. && pwd )"	# dir!
+cd "${ROOT}"
+
+go vet && echo PASS || exit 1	# since it doesn't output an ok message on pass
+grep 'log.' *.go | grep '\\n' && exit 1 || echo PASS	# no \n needed in log.Printf()

test/test-headerfmt.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+# check that headers are properly formatted
+echo running test-headerfmt.sh
+ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && cd .. && pwd )"	# dir!
+FILE="${ROOT}/main.go"	# file headers should match main.go
+COUNT=0
+while IFS='' read -r line; do	# find what header should look like
+	echo "$line" | grep -q '^//' || break
+	COUNT=`expr $COUNT + 1`
+done < "$FILE"
+cd "${ROOT}"
+
+find_files() {
+	git ls-files | grep '\.go$'
+}
+
+bad_files=$(
+	for i in $(find_files); do
+		if ! diff -q <( head -n $COUNT "$i" ) <( head -n $COUNT "$FILE" ) &>/dev/null; then
+			echo "$i"
+		fi
+	done
+)
+
+if [[ -n "${bad_files}" ]]; then
+	echo 'FAIL'
+	echo 'The following file headers are not properly formatted:'
+	echo "${bad_files}"
+	exit 1
+fi

test/test-omv.sh

@@ -0,0 +1,21 @@
+#!/bin/bash -i
+# simple test harness for testing mgmt via omv
+echo running test-omv.sh
+CWD=`pwd`
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"	# dir!
+cd "$DIR" >/dev/null	# work from test directory
+
+# vtest+ tests
+RET=0
+for i in omv/*.yaml; do
+	echo "running: vtest+ $i"
+	vtest+ "$i"
+	if [ $? -ne 0 ]; then
+		RET=1
+		break	# remove this if we should run all tests even if one fails
+	fi
+done
+
+# return to original dir
+cd "$CWD" >/dev/null
+exit $RET

test/test-reproducible.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+# simple test for reproducibility, probably needs major improvements
+echo running test-reproducible.sh
+set -o errexit
+set -o pipefail
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && cd .. && pwd )"	# dir!
+cd "$DIR" >/dev/null	# work from main mgmt directory
+make build
+T=`mktemp --tmpdir -d tmp.XXX`
+cp -a ./mgmt "$T"/mgmt.1
+make clean
+make build
+cp -a ./mgmt "$T"/mgmt.2
+
+# size comparison test
+[ `stat -c '%s' "$T"/mgmt.1` -eq `stat -c '%s' "$T"/mgmt.2` ] || failures="Size of binary was not reproducible"
+
+# sha1sum test
+sha1sum "$T"/mgmt.1 > "$T"/mgmt.SHA1SUMS.1
+sha1sum "$T"/mgmt.2 > "$T"/mgmt.SHA1SUMS.2
+cat "$T"/mgmt.SHA1SUMS.1 | sed 's/mgmt\.1/mgmt\.X/' > "$T"/mgmt.SHA1SUMS.1X
+cat "$T"/mgmt.SHA1SUMS.2 | sed 's/mgmt\.2/mgmt\.X/' > "$T"/mgmt.SHA1SUMS.2X
+diff -q "$T"/mgmt.SHA1SUMS.1X "$T"/mgmt.SHA1SUMS.2X || failures=$( [ -n "${failures}" ] && echo "$failures" ; echo "SHA1SUM of binary was not reproducible" )
+
+# clean up
+if [ "$T" != '' ]; then
+	rm -rf "$T"
+fi
+make clean
+
+# display errors
+if [[ -n "${failures}" ]]; then
+	echo 'FAIL'
+	echo 'The following tests failed:'
+	echo "${failures}"
+	exit 1
+fi
+echo PASS

test/test-shell.sh

@@ -0,0 +1,74 @@
+#!/bin/bash
+# simple test harness for testing mgmt
+# NOTE: this will rm -rf /tmp/mgmt/
+echo running test-shell.sh
+set -o errexit
+set -o pipefail
+
+LINE=$(printf '=%.0s' `seq -s ' ' $(tput cols)`)	# a terminal width string
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && cd .. && pwd )"	# dir!
+cd "$DIR" >/dev/null	# work from main mgmt directory
+make build
+MGMT="$DIR/test/shell/mgmt"
+cp -a "$DIR/mgmt" "$MGMT"	# put a copy there
+failures=""
+count=0
+
+# loop through tests
+for i in $DIR/test/shell/*.sh; do
+	[ -x "$i" ] || continue	# file must be executable
+	ii=`basename "$i"`	# short name
+	# if ARGV has test names, only execute those!
+	if [ "$1" != '' ]; then
+		[ "$ii" != "$1" ] && continue
+	fi
+	cd $DIR/test/shell/ >/dev/null	# shush the cd operation
+	mkdir -p '/tmp/mgmt/'	# directory for mgmt to put files in
+	#echo "Running: $ii"
+	export MGMT_TMPDIR='/tmp/mgmt/'	# we can add to env like this
+	count=`expr $count + 1`
+	set +o errexit	# don't kill script on test failure
+	out=$($i 2>&1)	# run and capture stdout & stderr
+	e=$?	# save exit code
+	set -o errexit	# re-enable killing on script failure
+	cd - >/dev/null
+	rm -rf '/tmp/mgmt/'	# clean up after test
+	if [ $e -ne 0 ]; then
+		echo -e "FAIL\t$ii"	# fail
+		# store failures...
+		failures=$(
+			# prepend previous failures if any
+			[ -n "${failures}" ] && echo "$failures" && echo "$LINE"
+			echo "Script: $ii"
+			# if we see 124, it might be the exit value of timeout!
+			[ $e -eq 124 ] && echo "Exited: $e (timeout?)" || echo "Exited: $e"
+			if [ "$out" = "" ]; then
+				echo "Output: (empty!)"
+			else
+				echo "Output:"
+				echo "$out"
+			fi
+		)
+	else
+		echo -e "ok\t$ii"	# pass
+	fi
+done
+
+# clean up
+rm -f "$MGMT"
+make clean
+
+if [ "$count" = '0' ]; then
+	echo 'FAIL'
+	echo 'No tests were run!'
+	exit 1
+fi
+
+# display errors
+if [[ -n "${failures}" ]]; then
+	echo 'FAIL'
+	echo 'The following tests failed:'
+	echo "${failures}"
+	exit 1
+fi
+echo PASS

test/test-yamlfmt.sh

@@ -1,25 +1,37 @@
 #!/bin/bash
 # check for any yaml files that aren't properly formatted
-
+echo running test-yamlfmt.sh
 set -o errexit
 set -o nounset
 set -o pipefail
 
+if env | grep -q -e '^TRAVIS=true$' -e '^JENKINS_URL=' -e '^BUILD_TAG=jenkins'; then
+	echo "Travis and Jenkins give wonky results here, skipping test!"
+	exit 0
+fi
+
 ROOT=$(dirname "${BASH_SOURCE}")/..
+
+RUBY=`which ruby 2>/dev/null`
+if [ -z $RUBY ]; then
+	echo "The 'ruby' utility can't be found."
+	exit 1
+fi
+
+$RUBY -e "require 'yaml'" 2>/dev/null || (
+	echo "The ruby 'yaml' library can't be found."
+	exit 1
+)
+
 cd "${ROOT}"
 
 find_files() {
-	find . -not \( \
-		\( \
-			-wholename './old' \
-			-o -wholename './tmp' \
-		\) -prune \
-	\) -name '*.yaml'
+	git ls-files | grep '\.yaml$'
 }
 
 bad_files=$(
 	for i in $(find_files); do
-		if ! diff -q <( ruby -e "require 'yaml'; puts YAML.load_file('$i').to_yaml" 2>/dev/null ) <( cat "$i" ) &>/dev/null; then
+		if ! diff -q <( ruby -e "require 'yaml'; puts YAML.load_file('$i').to_yaml.each_line.map(&:rstrip).join(10.chr)+10.chr" 2>/dev/null ) <( cat "$i" ) &>/dev/null; then
 			echo "$i"
 		fi
 	done
@@ -27,7 +39,7 @@ bad_files=$(
 
 if [[ -n "${bad_files}" ]]; then
 	echo 'FAIL'
-	echo 'The following files are not properly formatted:'
+	echo 'The following yaml files are not properly formatted:'
 	echo "${bad_files}"
 	exit 1
 fi

types.go

@@ -1,73 +0,0 @@
-// Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
-// Written by James Shubin <james@shubin.ca> and the project contributors
-//
-// This program is free software: you can redistribute it and/or modify
-// it under the terms of the GNU Affero General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-//
-// This program is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU Affero General Public License for more details.
-//
-// You should have received a copy of the GNU Affero General Public License
-// along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-package main
-
-import (
-	"code.google.com/p/go-uuid/uuid"
-	"fmt"
-	"log"
-)
-
-type Type interface {
-	//Name()  string
-	Watch(*Vertex)
-	StateOK() bool // TODO: can we rename this to something better?
-	Apply() bool
-	Exit() bool
-}
-
-type NoopType struct {
-	uuid   string
-	Type   string      // always "noop"
-	Name   string      // name variable
-	Events chan string // FIXME: eventually a struct for the event?
-}
-
-func NewNoopType(name string) *NoopType {
-	return &NoopType{
-		uuid:   uuid.New(),
-		Type:   "noop",
-		Name:   name,
-		Events: make(chan string, 1), // XXX: chan size?
-	}
-}
-
-func (obj NoopType) Watch(v *Vertex) {
-	select {
-	case exit := <-obj.Events:
-		if exit == "exit" {
-			return
-		} else {
-			log.Fatal("Unknown event: %v\n", exit)
-		}
-	}
-}
-
-func (obj NoopType) Exit() bool {
-	obj.Events <- "exit"
-	return true
-}
-
-func (obj NoopType) StateOK() bool {
-	return true // never needs updating
-}
-
-func (obj NoopType) Apply() bool {
-	fmt.Printf("Apply->%v[%v]\n", obj.Type, obj.Name)
-	return true
-}