thanks: More people really needed to be on the list

Felix wins the points for first asking the question I knew would
eventually come but didn't document earlier.

.dockerignore

@@ -0,0 +1 @@
+docker

.editorconfig

@@ -0,0 +1,19 @@
+; This file is for unifying the coding style for different editors and IDEs.
+; Plugins are available for notepad++, emacs, vim, gedit,
+; textmate, visual studio, and more.
+;
+; See http://editorconfig.org for details.
+
+# Top-most EditorConfig file.
+root = true
+
+[*]
+end_of_line = lf
+insert_final_newline = true
+trim_trailing_whitespace = true
+
+[*.go]
+indent_style = tab
+
+[Makefile]
+indent_style = tab

.gitignore

@@ -1,3 +1,10 @@
+.omv/
+.ssh/
+.vagrant/
 mgmt-documentation.pdf
 old/
 tmp/
+*_stringer.go
+mgmt
+mgmt.static
+rpmbuild/

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "vendor/github.com/coreos/etcd"]
+	path = vendor/github.com/coreos/etcd
+	url = https://github.com/coreos/etcd/

.travis.yml

@@ -0,0 +1,29 @@
+language: go
+go:
+  - 1.4.3
+  - 1.5.3
+  - 1.6
+  - tip
+sudo: false
+before_install: 'git fetch --unshallow'
+install: 'make deps'
+script: 'make test'
+matrix:
+  fast_finish: true
+  allow_failures:
+    - go: tip
+    - go: 1.4.3
+    - go: 1.6
+notifications:
+  irc:
+    channels:
+      - "irc.freenode.net#mgmtconfig"
+    template:
+      - "%{repository} (%{commit}: %{author}): %{message}"
+      - "More info : %{build_url}"
+    on_success: always
+    on_failure: always
+    use_notice: false
+    skip_join: false
+  email:
+    - travis-ci@shubin.ca

AUTHORS

@@ -5,3 +5,4 @@ For a more exhaustive list please run: git log --format='%aN' | sort -u
 This list is sorted alphabetically by first name.
 
 James Shubin
+Paul Morgan

COPYRIGHT

@@ -1,5 +1,5 @@
 Mgmt
-Copyright (C) 2013-2015+ James Shubin and the project contributors
+Copyright (C) 2013-2016+ James Shubin and the project contributors
 Written by James Shubin <james@shubin.ca> and the project contributors
 
 This program is free software: you can redistribute it and/or modify

DOCUMENTATION.md

@@ -2,7 +2,7 @@
 
 <!--
 Mgmt
-Copyright (C) 2013-2015+ James Shubin and the project contributors
+Copyright (C) 2013-2016+ James Shubin and the project contributors
 Written by James Shubin <james@shubin.ca> and the project contributors
 
 This program is free software: you can redistribute it and/or modify
@@ -30,18 +30,23 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 1. [Overview](#overview)
 2. [Project description - What the project does](#project-description)
 3. [Setup - Getting started with mgmt](#setup)
-4. [Usage/FAQ - Notes on usage and frequently asked questions](#usage-and-frequently-asked-questions)
-5. [Reference - Detailed reference](#reference)
-	* [graph.yaml](#graph.yaml)
+4. [Features - All things mgmt can do](#features)
+	* [Autoedges - Automatic resource relationships](#autoedges)
+	* [Autogrouping - Automatic resource grouping](#autogrouping)
+	* [Automatic clustering - Automatic cluster management](#automatic-clustering)
+	* [Remote mode - Remote "agent-less" execution](#remote-agent-less-mode)
+5. [Usage/FAQ - Notes on usage and frequently asked questions](#usage-and-frequently-asked-questions)
+6. [Reference - Detailed reference](#reference)
+	* [Graph definition file](#graph-definition-file)
 	* [Command line](#command-line)
-6. [Examples - Example configurations](#examples)
-7. [Development - Background on module development and reporting bugs](#development)
-8. [Authors - Authors and contact information](#authors)
+7. [Examples - Example configurations](#examples)
+8. [Development - Background on module development and reporting bugs](#development)
+9. [Authors - Authors and contact information](#authors)
 
 ##Overview
 
-The `mgmt` tool is a research prototype to demonstrate next generation config
-management techniques. Hopefully it will evolve into a useful, robust tool.
+The `mgmt` tool is a next generation config management prototype. It's not yet
+ready for production, but we hope to get there soon. Get involved today!
 
 ##Project Description
 
@@ -49,6 +54,16 @@ The mgmt tool is a distributed, event driven, config management tool, that
 supports parallel execution, and librarification to be used as the management
 foundation in and for, new and existing software.
 
+For more information, you may like to read some blog posts from the author:
+
+* [Next generation config mgmt](https://ttboj.wordpress.com/2016/01/18/next-generation-configuration-mgmt/)
+* [Automatic edges in mgmt](https://ttboj.wordpress.com/2016/03/14/automatic-edges-in-mgmt/)
+* [Automatic grouping in mgmt](https://ttboj.wordpress.com/2016/03/30/automatic-grouping-in-mgmt/)
+* [Automatic clustering in mgmt](https://ttboj.wordpress.com/2016/06/20/automatic-clustering-in-mgmt/)
+
+There is also an [introductory video](http://meetings-archive.debian.net/pub/debian-meetings/2016/debconf16/Next_Generation_Config_Mgmt.webm) available.
+Older videos and other material [is available](https://github.com/purpleidea/mgmt/#on-the-web).
+
 ##Setup
 
 During this prototype phase, the tool can be run out of the source directory.
@@ -57,6 +72,103 @@ get started. Beware that this _can_ cause data loss. Understand what you're
 doing first, or perform these actions in a virtual environment such as the one
 provided by [Oh-My-Vagrant](https://github.com/purpleidea/oh-my-vagrant).
 
+##Features
+
+This section details the numerous features of mgmt and some caveats you might
+need to be aware of.
+
+###Autoedges
+
+Automatic edges, or AutoEdges, is the mechanism in mgmt by which it will
+automatically create dependencies for you between resources. For example,
+since mgmt can discover which files are installed by a package it will
+automatically ensure that any file resource you declare that matches a
+file installed by your package resource will only be processed after the
+package is installed.
+
+####Controlling autoedges
+
+Though autoedges is likely to be very helpful and avoid you having to declare
+all dependencies explicitly, there are cases where this behaviour is
+undesirable.
+
+Some distributions allow package installations to automatically start the
+service they ship. This can be problematic in the case of packages like MySQL
+as there are configuration options that need to be set before MySQL is ever
+started for the first time (or you'll need to wipe the data directory). In
+order to handle this situation you can disable autoedges per resource and
+explicitly declare that you want `my.cnf` to be written to disk before the
+installation of the `mysql-server` package.
+
+You can disable autoedges for a resource by setting the `autoedge` key on
+the meta attributes of that resource to `false`.
+
+####Blog post
+
+You can read the introductory blog post about this topic here:
+[https://ttboj.wordpress.com/2016/03/14/automatic-edges-in-mgmt/](https://ttboj.wordpress.com/2016/03/14/automatic-edges-in-mgmt/)
+
+###Autogrouping
+
+Automatic grouping or AutoGroup is the mechanism in mgmt by which it will
+automatically group multiple resource vertices into a single one. This is
+particularly useful for grouping multiple package resources into a single
+resource, since the multiple installations can happen together in a single
+transaction, which saves a lot of time because package resources typically have
+a large fixed cost to running (downloading and verifying the package repo) and
+if they are grouped they share this fixed cost. This grouping feature can be
+used for other use cases too.
+
+You can disable autogrouping for a resource by setting the `autogroup` key on
+the meta attributes of that resource to `false`.
+
+####Blog post
+
+You can read the introductory blog post about this topic here:
+[https://ttboj.wordpress.com/2016/03/30/automatic-grouping-in-mgmt/](https://ttboj.wordpress.com/2016/03/30/automatic-grouping-in-mgmt/)
+
+###Automatic clustering
+
+Automatic clustering is a feature by which mgmt automatically builds, scales,
+and manages the embedded etcd cluster which is compiled into mgmt itself. It is
+quite helpful for rapidly bootstrapping clusters and avoiding the extra work to
+setup etcd.
+
+If you prefer to avoid this feature. you can always opt to use an existing etcd
+cluster that is managed separately from mgmt by pointing your mgmt agents at it
+with the `--seeds` variable.
+
+####Blog post
+
+You can read the introductory blog post about this topic here:
+[https://ttboj.wordpress.com/2016/06/20/automatic-clustering-in-mgmt/](https://ttboj.wordpress.com/2016/06/20/automatic-clustering-in-mgmt/)
+
+###Remote ("agent-less") mode
+
+Remote mode is a special mode that lets you kick off mgmt runs on one or more
+remote machines which are only accessible via SSH. In this mode the initiating
+host connects over SSH, copies over the `mgmt` binary, opens an SSH tunnel, and
+runs the remote program while simultaneously passing the etcd traffic back
+through the tunnel so that the initiators etcd cluster can be used to exchange
+resource data.
+
+The interesting benefit of this architecture is that multiple hosts which can't
+connect directly use the initiator to pass the important traffic through to each
+other. Once the cluster has converged all the remote programs can shutdown
+leaving no residual agent.
+
+This mode can also be useful for bootstrapping a new host where you'd like to
+have the service run continuously and as part of an mgmt cluster normally.
+
+In particular, when combined with the `--converged-timeout` parameter, the
+entire set of running mgmt agents will need to all simultaneously converge for
+the group to exit. This is particularly useful for bootstrapping new clusters
+which need to exchange information that is only available at run time.
+
+####Blog post
+
+An introductory blog post about this topic will follow soon.
+
 ##Usage and frequently asked questions
 (Send your questions as a patch to this FAQ! I'll review it, merge it, and
 respond by commit with the answer.)
@@ -67,6 +179,66 @@ I wanted a next generation config management solution that didn't have all of
 the design flaws or limitations that the current generation of tools do, and no
 tool existed!
 
+###Why did you use etcd? What about consul?
+
+Etcd and consul are both written in golang, which made them the top two
+contenders for my prototype. Ultimately a choice had to be made, and etcd was
+chosen, but it was also somewhat arbitrary. If there is available interest,
+good reasoning, *and* patches, then we would consider either switching or
+supporting both, but this is not a high priority at this time.
+
+###Can I use an existing etcd cluster instead of the automatic embedded servers?
+
+Yes, it's possible to use an existing etcd cluster instead of the automatic,
+elastic embedded etcd servers. To do so, simply point to the cluster with the
+`--seeds` variable, the same way you would if you were seeding a new member to
+an existing mgmt cluster.
+
+The downside to this approach is that you won't benefit from the automatic
+elastic nature of the embedded etcd servers, and that you're responsible if you
+accidentally break your etcd cluster, or if you use an unsupported version.
+
+###What does the error message about an inconsistent dataDir mean?
+
+If you get an error message similar to:
+
+```
+Etcd: Connect: CtxError...
+Etcd: CtxError: Reason: CtxDelayErr(5s): No endpoints available yet!
+Etcd: Connect: Endpoints: []
+Etcd: The dataDir (/var/lib/mgmt/etcd) might be inconsistent or corrupt.
+```
+
+This happens when there are a series of fatal connect errors in a row. This can
+happen when you start `mgmt` using a dataDir that doesn't correspond to the
+current cluster view. As a result, the embedded etcd server never finishes
+starting up, and as a result, a default endpoint never gets added. The solution
+is to either reconcile the mistake, and if there is no important data saved, you
+can remove the etcd dataDir. This is typically `/var/lib/mgmt/etcd/member/`.
+
+###Why do resources have both a `Compare` method and an `IFF` (on the UUID) method?
+
+The `Compare()` methods are for determining if two resources are effectively the
+same, which is used to make graph change delta's efficient. This is when we want
+to change from the current running graph to a new graph, but preserve the common
+vertices. Since we want to make this process efficient, we only update the parts
+that are different, and leave everything else alone. This `Compare()` method can
+tell us if two resources are the same.
+
+The `IFF()` method is part of the whole UUID system, which is for discerning if
+a resource meets the requirements another expects for an automatic edge. This is
+because the automatic edge system assumes a unified UUID pattern to test for
+equality. In the future it might be helpful or sane to merge the two similar
+comparison functions although for now they are separate because they are
+actually answer different questions.
+
+###Did you know that there is a band named `MGMT`?
+
+I didn't realize this when naming the project, and it is accidental. After much
+anguishing, I chose the name because it was short and I thought it was
+appropriately descriptive. If you need a less ambiguous search term or phrase,
+you can try using `mgmtconfig` or `mgmt config`.
+
 ###You didn't answer my question, or I have a question!
 
 It's best to ask on [IRC](https://webchat.freenode.net/?channels=#mgmtconfig)
@@ -84,11 +256,11 @@ information on these options, please view the source at:
 If you feel that a well used option needs documenting here, please patch it!
 
 ###Overview of reference
-* [graph.yaml](#graph.yaml): Main graph definition file.
+* [Graph definition file](#graph-definition-file): Main graph definition file.
 * [Command line](#command-line): Command line parameters.
 
-###graph.yaml
-This is the compiled graph definition file. The format is currently
+###Graph definition file
+graph.yaml is the compiled graph definition file. The format is currently
 undocumented, but by looking through the [examples/](https://github.com/purpleidea/mgmt/tree/master/examples)
 you can probably figure out most of it, as it's fairly intuitive.
 
@@ -99,12 +271,86 @@ documentation, please run `mgmt --help`.
 ####`--file <graph.yaml>`
 Point to a graph file to run.
 
+####`--converged-timeout <seconds>`
+Exit if the machine has converged for approximately this many seconds.
+
+####`--max-runtime <seconds>`
+Exit when the agent has run for approximately this many seconds. This is not
+generally recommended, but may be useful for users who know what they're doing.
+
+####`--noop`
+Globally force all resources into no-op mode. This also disables the export to
+etcd functionality, but does not disable resource collection, however all
+resources that are collected will have their individual noop settings set.
+
+####`--remote <graph.yaml>`
+Point to a graph file to run on the remote host specified within. This parameter
+can be used multiple times if you'd like to remotely run on multiple hosts in
+parallel.
+
+####`--allow-interactive`
+Allow interactive prompting for SSH passwords if there is no authentication
+method that works.
+
+####`--ssh-priv-id-rsa`
+Specify the path for finding SSH keys. This defaults to `~/.ssh/id_rsa`. To
+never use this method of authentication, set this to the empty string.
+
+####`--cconns`
+The maximum number of concurrent remote ssh connections to run. This defaults
+to `0`, which means unlimited.
+
+####`--no-caching`
+Don't allow remote caching of the remote execution binary. This will require
+the binary to be copied over for every remote execution, but it limits the
+likelihood that there is leftover information from the configuration process.
+
+####`--prefix <path>`
+Specify a path to a custom working directory prefix. This directory will get
+created if it does not exist. This usually defaults to `/var/lib/mgmt/`. This
+can't be combined with the `--tmp-prefix` option. It can be combined with the
+`--allow-tmp-prefix` option.
+
+####`--tmp-prefix`
+If this option is specified, a temporary prefix will be used instead of the
+default prefix. This can't be combined with the `--prefix` option.
+
+####`--allow-tmp-prefix`
+If this option is specified, we will attempt to fall back to a temporary prefix
+if the primary prefix couldn't be created. This is useful for avoiding failures
+in environments where the primary prefix may or may not be available, but you'd
+like to try. The canonical example is when running `mgmt` with `--remote` there
+might be a cached copy of the binary in the primary prefix, but in case there's
+no binary available continue working in a temporary directory to avoid failure.
+
 ##Examples
 For example configurations, please consult the [examples/](https://github.com/purpleidea/mgmt/tree/master/examples) directory in the git
 source repository. It is available from:
 
 [https://github.com/purpleidea/mgmt/tree/master/examples](https://github.com/purpleidea/mgmt/tree/master/examples)
 
+### Systemd:
+See [`misc/mgmt.service`](misc/mgmt.service) for a sample systemd unit file.
+This unit file is part of the RPM.
+
+To specify your custom options for `mgmt` on a systemd distro:
+
+```bash
+sudo mkdir -p /etc/systemd/system/mgmt.service.d/
+
+cat > /etc/systemd/system/mgmt.service.d/env.conf <<EOF
+# Environment variables:
+MGMT_SEEDS=http://127.0.0.1:2379
+MGMT_CONVERGED_TIMEOUT=-1
+MGMT_MAX_RUNTIME=0
+
+# Other CLI options if necessary.
+#OPTS="--max-runtime=0"
+EOF
+
+sudo systemctl daemon-reload
+```
+
 ##Development
 
 This is a project that I started in my free time in 2013. Development is driven
@@ -117,7 +363,7 @@ To report any bugs, please file a ticket at: [https://github.com/purpleidea/mgmt
 
 ##Authors
 
-Copyright (C) 2013-2015+ James Shubin and the project contributors
+Copyright (C) 2013-2016+ James Shubin and the project contributors
 
 Please see the
 [AUTHORS](https://github.com/purpleidea/mgmt/tree/master/AUTHORS) file

Makefile

@@ -1,45 +1,238 @@
+# Mgmt
+# Copyright (C) 2013-2016+ James Shubin and the project contributors
+# Written by James Shubin <james@shubin.ca> and the project contributors
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 SHELL = /bin/bash
-.PHONY: all version run race build clean test format docs
+.PHONY: all version program path deps run race generate build clean test gofmt yamlfmt format docs rpmbuild mkdirs rpm srpm spec tar upload upload-sources upload-srpms upload-rpms copr
 .SILENT: clean
 
-VERSION := $(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --dirty)
-PROGRAM := $(notdir $(CURDIR))
+SVERSION := $(or $(SVERSION),$(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --dirty --always))
+VERSION := $(or $(VERSION),$(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --abbrev=0))
+PROGRAM := $(shell echo $(notdir $(CURDIR)) | cut -f1 -d"-")
+OLDGOLANG := $(shell go version | grep -E 'go1.3|go1.4')
+ifeq ($(VERSION),$(SVERSION))
+	RELEASE = 1
+else
+	RELEASE = untagged
+endif
+ARCH = $(uname -m)
+SPEC = rpmbuild/SPECS/$(PROGRAM).spec
+SOURCE = rpmbuild/SOURCES/$(PROGRAM)-$(VERSION).tar.bz2
+SRPM = rpmbuild/SRPMS/$(PROGRAM)-$(VERSION)-$(RELEASE).src.rpm
+SRPM_BASE = $(PROGRAM)-$(VERSION)-$(RELEASE).src.rpm
+RPM = rpmbuild/RPMS/$(PROGRAM)-$(VERSION)-$(RELEASE).$(ARCH).rpm
+USERNAME := $(shell cat ~/.config/copr 2>/dev/null | grep username | awk -F '=' '{print $$2}' | tr -d ' ')
+SERVER = 'dl.fedoraproject.org'
+REMOTE_PATH = 'pub/alt/$(USERNAME)/$(PROGRAM)'
 
-all: docs
+all: docs $(PROGRAM).static
 
 # show the current version
 version:
 	@echo $(VERSION)
 
+program:
+	@echo $(PROGRAM)
+
+path:
+	./misc/make-path.sh
+
+deps:
+	./misc/make-deps.sh
+
 run:
-	find -maxdepth 1 -type f -name '*.go' -not -name '*_test.go' | xargs go run -ldflags "-X main.version $(VERSION) -X main.program $(PROGRAM)"
+	find . -maxdepth 1 -type f -name '*.go' -not -name '*_test.go' | xargs go run -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION)"
 
-# include race test
+# include race flag
 race:
-	find -maxdepth 1 -type f -name '*.go' -not -name '*_test.go' | xargs go run -race -ldflags "-X main.version $(VERSION) -X main.program $(PROGRAM)"
+	find . -maxdepth 1 -type f -name '*.go' -not -name '*_test.go' | xargs go run -race -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION)"
 
-build: mgmt
+generate:
+	go generate
 
-mgmt: main.go
-	go build -ldflags "-X main.version $(VERSION) -X main.program $(PROGRAM)"
+build: $(PROGRAM)
+
+$(PROGRAM): main.go
+	@echo "Building: $(PROGRAM), version: $(SVERSION)..."
+ifneq ($(OLDGOLANG),)
+	@# avoid equals sign in old golang versions eg in: -X foo=bar
+	time go build -ldflags "-X main.program $(PROGRAM) -X main.version $(SVERSION)" -o $(PROGRAM);
+else
+	time go build -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION)" -o $(PROGRAM);
+endif
+
+$(PROGRAM).static: main.go
+	@echo "Building: $(PROGRAM).static, version: $(SVERSION)..."
+	go generate
+ifneq ($(OLDGOLANG),)
+	@# avoid equals sign in old golang versions eg in: -X foo=bar
+	go build -a -installsuffix cgo -tags netgo -ldflags '-extldflags "-static" -X main.program $(PROGRAM) -X main.version $(SVERSION)' -o $(PROGRAM).static;
+else
+	go build -a -installsuffix cgo -tags netgo -ldflags '-extldflags "-static" -X main.program=$(PROGRAM) -X main.version=$(SVERSION)' -o $(PROGRAM).static;
+endif
 
 clean:
-	[ ! -e mgmt ] || rm mgmt
+	[ ! -e $(PROGRAM) ] || rm $(PROGRAM)
+	#rm -f *_stringer.go	# generated by `go generate`
 
 test:
 	./test.sh
-	./test/test-gofmt.sh
-	./test/test-yamlfmt.sh
-	go test
-	#go test ./pgraph
-	go test -race
-	#go test -race ./pgraph
 
-format:
-	find -type f -name '*.go' -not -path './old/*' -not -path './tmp/*' -exec gofmt -w {} \;
-	find -type f -name '*.yaml' -not -path './old/*' -not -path './tmp/*' -exec ruby -e "require 'yaml'; x=YAML.load_file('{}').to_yaml; File.open('{}', 'w').write x" \;
+gofmt:
+	find . -maxdepth 3 -type f -name '*.go' -not -path './old/*' -not -path './tmp/*' -exec gofmt -w {} \;
 
-docs: mgmt-documentation.pdf
+yamlfmt:
+	find . -type f -name '*.yaml' -not -path './old/*' -not -path './tmp/*' -not -path './omv.yaml' -exec ruby -e "require 'yaml'; x=YAML.load_file('{}').to_yaml.each_line.map(&:rstrip).join(10.chr)+10.chr; File.open('{}', 'w').write x" \;
 
-mgmt-documentation.pdf: DOCUMENTATION.md
-	pandoc DOCUMENTATION.md -o 'mgmt-documentation.pdf'
+format: gofmt yamlfmt
+
+docs: $(PROGRAM)-documentation.pdf
+
+$(PROGRAM)-documentation.pdf: DOCUMENTATION.md
+	pandoc DOCUMENTATION.md -o '$(PROGRAM)-documentation.pdf'
+
+#
+#	build aliases
+#
+# TODO: does making an rpm depend on making a .srpm first ?
+rpm: $(SRPM) $(RPM)
+	# do nothing
+
+srpm: $(SRPM)
+	# do nothing
+
+spec: $(SPEC)
+	# do nothing
+
+tar: $(SOURCE)
+	# do nothing
+
+rpmbuild/SOURCES/: tar
+rpmbuild/SRPMS/: srpm
+rpmbuild/RPMS/: rpm
+
+upload: upload-sources upload-srpms upload-rpms
+	# do nothing
+
+#
+#	rpmbuild
+#
+$(RPM): $(SPEC) $(SOURCE)
+	@echo Running rpmbuild -bb...
+	rpmbuild --define '_topdir $(shell pwd)/rpmbuild' -bb $(SPEC) && \
+	mv rpmbuild/RPMS/$(ARCH)/$(PROGRAM)-$(VERSION)-$(RELEASE).*.rpm $(RPM)
+
+$(SRPM): $(SPEC) $(SOURCE)
+	@echo Running rpmbuild -bs...
+	rpmbuild --define '_topdir $(shell pwd)/rpmbuild' -bs $(SPEC)
+	# renaming is not needed because we aren't using the dist variable
+	#mv rpmbuild/SRPMS/$(PROGRAM)-$(VERSION)-$(RELEASE).*.src.rpm $(SRPM)
+
+#
+#	spec
+#
+$(SPEC): rpmbuild/ spec.in
+	@echo Running templater...
+	#cat spec.in > $(SPEC)
+	sed -e s/__PROGRAM__/$(PROGRAM)/ -e s/__VERSION__/$(VERSION)/ -e s/__RELEASE__/$(RELEASE)/ < spec.in > $(SPEC)
+	# append a changelog to the .spec file
+	git log --format="* %cd %aN <%aE>%n- (%h) %s%d%n" --date=local | sed -r 's/[0-9]+:[0-9]+:[0-9]+ //' >> $(SPEC)
+
+#
+#	archive
+#
+$(SOURCE): rpmbuild/
+	@echo Running git archive...
+	# use HEAD if tag doesn't exist yet, so that development is easier...
+	git archive --prefix=$(PROGRAM)-$(VERSION)/ -o $(SOURCE) $(VERSION) 2> /dev/null || (echo 'Warning: $(VERSION) does not exist. Using HEAD instead.' && git archive --prefix=$(PROGRAM)-$(VERSION)/ -o $(SOURCE) HEAD)
+	# TODO: if git archive had a --submodules flag this would easier!
+	@echo Running git archive submodules...
+	# i thought i would need --ignore-zeros, but it doesn't seem necessary!
+	p=`pwd` && (echo .; git submodule foreach) | while read entering path; do \
+		temp="$${path%\'}"; \
+		temp="$${temp#\'}"; \
+		path=$$temp; \
+		[ "$$path" = "" ] && continue; \
+		(cd $$path && git archive --prefix=$(PROGRAM)-$(VERSION)/$$path/ HEAD > $$p/rpmbuild/tmp.tar && tar --concatenate --file=$$p/$(SOURCE) $$p/rpmbuild/tmp.tar && rm $$p/rpmbuild/tmp.tar); \
+	done
+
+# TODO: ensure that each sub directory exists
+rpmbuild/:
+	mkdir -p rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
+
+mkdirs:
+	mkdir -p rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
+
+#
+#	sha256sum
+#
+rpmbuild/SOURCES/SHA256SUMS: rpmbuild/SOURCES/ $(SOURCE)
+	@echo Running SOURCES sha256sum...
+	cd rpmbuild/SOURCES/ && sha256sum *.tar.bz2 > SHA256SUMS; cd -
+
+rpmbuild/SRPMS/SHA256SUMS: rpmbuild/SRPMS/ $(SRPM)
+	@echo Running SRPMS sha256sum...
+	cd rpmbuild/SRPMS/ && sha256sum *src.rpm > SHA256SUMS; cd -
+
+rpmbuild/RPMS/SHA256SUMS: rpmbuild/RPMS/ $(RPM)
+	@echo Running RPMS sha256sum...
+	cd rpmbuild/RPMS/ && sha256sum *.rpm > SHA256SUMS; cd -
+
+#
+#	gpg
+#
+rpmbuild/SOURCES/SHA256SUMS.asc: rpmbuild/SOURCES/SHA256SUMS
+	@echo Running SOURCES gpg...
+	# the --yes forces an overwrite of the SHA256SUMS.asc if necessary
+	gpg2 --yes --clearsign rpmbuild/SOURCES/SHA256SUMS
+
+rpmbuild/SRPMS/SHA256SUMS.asc: rpmbuild/SRPMS/SHA256SUMS
+	@echo Running SRPMS gpg...
+	gpg2 --yes --clearsign rpmbuild/SRPMS/SHA256SUMS
+
+rpmbuild/RPMS/SHA256SUMS.asc: rpmbuild/RPMS/SHA256SUMS
+	@echo Running RPMS gpg...
+	gpg2 --yes --clearsign rpmbuild/RPMS/SHA256SUMS
+
+#
+#	upload
+#
+# upload to public server
+upload-sources: rpmbuild/SOURCES/ rpmbuild/SOURCES/SHA256SUMS rpmbuild/SOURCES/SHA256SUMS.asc
+	if [ "`cat rpmbuild/SOURCES/SHA256SUMS`" != "`ssh $(SERVER) 'cd $(REMOTE_PATH)/SOURCES/ && cat SHA256SUMS'`" ]; then \
+		echo Running SOURCES upload...; \
+		rsync -avz rpmbuild/SOURCES/ $(SERVER):$(REMOTE_PATH)/SOURCES/; \
+	fi
+
+upload-srpms: rpmbuild/SRPMS/ rpmbuild/SRPMS/SHA256SUMS rpmbuild/SRPMS/SHA256SUMS.asc
+	if [ "`cat rpmbuild/SRPMS/SHA256SUMS`" != "`ssh $(SERVER) 'cd $(REMOTE_PATH)/SRPMS/ && cat SHA256SUMS'`" ]; then \
+		echo Running SRPMS upload...; \
+		rsync -avz rpmbuild/SRPMS/ $(SERVER):$(REMOTE_PATH)/SRPMS/; \
+	fi
+
+upload-rpms: rpmbuild/RPMS/ rpmbuild/RPMS/SHA256SUMS rpmbuild/RPMS/SHA256SUMS.asc
+	if [ "`cat rpmbuild/RPMS/SHA256SUMS`" != "`ssh $(SERVER) 'cd $(REMOTE_PATH)/RPMS/ && cat SHA256SUMS'`" ]; then \
+		echo Running RPMS upload...; \
+		rsync -avz --prune-empty-dirs rpmbuild/RPMS/ $(SERVER):$(REMOTE_PATH)/RPMS/; \
+	fi
+
+#
+#	copr build
+#
+copr: upload-srpms
+	./misc/copr-build.py https://$(SERVER)/$(REMOTE_PATH)/SRPMS/$(SRPM_BASE)
+
+# vim: ts=8

README.md

@@ -1,35 +1,84 @@
 # *mgmt*: This is: mgmt!
 
-[![Build Status](https://secure.travis-ci.org/purpleidea/mgmt.png)](http://travis-ci.org/purpleidea/mgmt)
+[![Go Report Card](https://goreportcard.com/badge/github.com/purpleidea/mgmt)](https://goreportcard.com/report/github.com/purpleidea/mgmt)
+[![Build Status](https://secure.travis-ci.org/purpleidea/mgmt.png?branch=master)](http://travis-ci.org/purpleidea/mgmt)
+[![Documentation](https://img.shields.io/docs/markdown.png)](DOCUMENTATION.md)
+[![IRC](https://img.shields.io/irc/%23mgmtconfig.png)](https://webchat.freenode.net/?channels=#mgmtconfig)
+[![Jenkins](https://img.shields.io/jenkins/status.png)](https://ci.centos.org/job/purpleidea-mgmt/)
+[![COPR](https://img.shields.io/copr/builds.png)](https://copr.fedoraproject.org/coprs/purpleidea/mgmt/)
+
+## Community:
+Come join us on IRC in [#mgmtconfig](https://webchat.freenode.net/?channels=#mgmtconfig) on Freenode!
+You may like the [#mgmtconfig](https://twitter.com/hashtag/mgmtconfig) hashtag if you're on [Twitter](https://twitter.com/#!/purpleidea).
+
+## Questions:
+Please join the [#mgmtconfig](https://webchat.freenode.net/?channels=#mgmtconfig) IRC community!
+If you have a well phrased question that might benefit others, consider asking it by sending a patch to the documentation [FAQ](https://github.com/purpleidea/mgmt/blob/master/DOCUMENTATION.md#usage-and-frequently-asked-questions) section. I'll merge your question, and a patch with the answer!
+
+## Quick start:
+* Clone the repository recursively, eg: `git clone --recursive https://github.com/purpleidea/mgmt/`.
+* Get the remaining golang dependencies on your own, or run `make deps` if you're comfortable with how we install them.
+* Run `make build` to get a freshly built `mgmt` binary.
+* Run `time ./mgmt run --file examples/graph0.yaml --converged-timeout=1` to try out a very simple example!
+* To run continuously in the default mode of operation, omit the `--converged-timeout` option.
+* Have fun hacking on our future technology!
+
+## Examples:
+Please look in the [examples/](examples/) folder for more examples!
 
 ## Documentation:
 Please see: [DOCUMENTATION.md](DOCUMENTATION.md) or [PDF](https://pdfdoc-purpleidea.rhcloud.com/pdf/https://github.com/purpleidea/mgmt/blob/master/DOCUMENTATION.md).
 
-## Questions:
-Come join us in [#mgmtconfig](https://webchat.freenode.net/?channels=#mgmtconfig) on Freenode!
+## Roadmap:
+Please see: [TODO.md](TODO.md) for a list of upcoming work and TODO items.
+Please get involved by working on one of these items or by suggesting something else!
+Feel free to grab one of the straightforward [#mgmtlove](https://github.com/purpleidea/mgmt/labels/mgmtlove) issues if you're a first time contributor to the project or if you're unsure about what to hack on!
 
-## Examples:
-Please look in the [examples/](examples/) folder for usage. If none exist, please contribute one!
-
-## Notes:
-* This is currently a research project into next generation config management technologies!
-* This is my first complex project in golang, please notify me of any issues.
-* I have some well thought out designs for the future of this project, which I'll try and write up clearly and publish as soon as possible.
-* Please don't expect stable interfaces, code, or any data safety.
-* This design is the result of ideas I've had from hacking on advanced config management projects.
-* I first started hacking on this in ~2013, even though I had very little time for it.
-* I couldn't think of a good name for the project, so it's now being called `mgmt` until someone contributes a better one!
-* I've published a number of articles about this tool:
-  * TODO
-* There are some screencasts available:
-  * TODO
+## Bugs:
+Please set the `DEBUG` constant in [main.go](https://github.com/purpleidea/mgmt/blob/master/main.go) to `true`, and post the logs when you report the [issue](https://github.com/purpleidea/mgmt/issues).
+Bonus points if you provide a [shell](https://github.com/purpleidea/mgmt/tree/master/test/shell) or [OMV](https://github.com/purpleidea/mgmt/tree/master/test/omv) reproducible test case.
+Feel free to read my article on [debugging golang programs](https://ttboj.wordpress.com/2016/02/15/debugging-golang-programs/).
 
 ## Dependencies:
-* golang (available in most distros)
-* pandoc (for building a pdf of the documentation)
+* golang 1.4 or higher (required, available in most distros)
+* golang libraries (required, available with `go get`)
+
+        go get github.com/coreos/etcd/client
+        go get gopkg.in/yaml.v2
+        go get gopkg.in/fsnotify.v1
+        go get github.com/codegangsta/cli
+        go get github.com/coreos/go-systemd/dbus
+        go get github.com/coreos/go-systemd/util
+        go get github.com/coreos/pkg/capnslog
+
+* stringer (required for building), available as a package on some platforms, otherwise via `go get`
+
+        go get golang.org/x/tools/cmd/stringer
+
+* pandoc (optional, for building a pdf of the documentation)
+* graphviz (optional, for building a visual representation of the graph)
 
 ## Patches:
-We'd love to have your patch! Please send it by email, or as a pull request.
+We'd love to have your patches! Please send them by email, or as a pull request.
+
+## On the web:
+* James Shubin; blog: [Next generation configuration mgmt](https://ttboj.wordpress.com/2016/01/18/next-generation-configuration-mgmt/)
+* James Shubin; video: [Introductory recording from DevConf.cz 2016](https://www.youtube.com/watch?v=GVhpPF0j-iE&html5=1)
+* James Shubin; video: [Introductory recording from CfgMgmtCamp.eu 2016](https://www.youtube.com/watch?v=fNeooSiIRnA&html5=1)
+* Julian Dunn; video: [On mgmt at CfgMgmtCamp.eu 2016](https://www.youtube.com/watch?v=kfF9IATUask&t=1949&html5=1)
+* Walter Heck; slides: [On mgmt at CfgMgmtCamp.eu 2016](http://www.slideshare.net/olindata/configuration-management-time-for-a-4th-generation/3)
+* Marco Marongiu; blog: [On mgmt](http://syslog.me/2016/02/15/leap-or-die/)
+* Felix Frank; blog: [From Catalog To Mgmt (on puppet to mgmt "transpiling")](https://ffrank.github.io/features/2016/02/18/from-catalog-to-mgmt/)
+* James Shubin; blog: [Automatic edges in mgmt (...and the pkg resource)](https://ttboj.wordpress.com/2016/03/14/automatic-edges-in-mgmt/)
+* James Shubin; blog: [Automatic grouping in mgmt](https://ttboj.wordpress.com/2016/03/30/automatic-grouping-in-mgmt/)
+* John Arundel; tweet: [“Puppet’s days are numbered.”](https://twitter.com/bitfield/status/732157519142002688)
+* Felix Frank; blog: [Puppet, Meet Mgmt (on puppet to mgmt internals)](https://ffrank.github.io/features/2016/06/12/puppet,-meet-mgmt/)
+* Felix Frank; blog: [Puppet Powered Mgmt (puppet to mgmt tl;dr)](https://ffrank.github.io/features/2016/06/19/puppet-powered-mgmt/)
+* James Shubin; blog: [Automatic clustering in mgmt](https://ttboj.wordpress.com/2016/06/20/automatic-clustering-in-mgmt/)
+* James Shubin; video: [Recording from CoreOSFest 2016](https://www.youtube.com/watch?v=KVmDCUA42wc&html5=1)
+* James Shubin; video: [Recording from DebConf16](http://meetings-archive.debian.net/pub/debian-meetings/2016/debconf16/Next_Generation_Config_Mgmt.webm) ([Slides](https://annex.debconf.org//debconf-share/debconf16/slides/15-next-generation-config-mgmt.pdf))
+* Felix Frank; blog: [Edging It All In (puppet and mgmt edges)](https://ffrank.github.io/features/2016/07/12/edging-it-all-in/)
+* Felix Frank; blog: [Translating All The Things (puppet to mgmt translation warnings)](https://ffrank.github.io/features/2016/08/19/translating-all-the-things/)
 
 ##
 

THANKS

@@ -9,8 +9,16 @@ Chris Wright - For encouraging me to continue work on my prototype.
 
 Daniel Riek - For supporting and sheltering this project from bureaucracy.
 
+Diego Ongaro - For good chats, particularly around distributed systems.
+
+Felix Frank - For taking a difficult problem and building an inspiring solution.
+
+Ira Cooper - For having an algorithmic design discussion with me.
+
 Jeff Darcy - For some algorithm recommendations, and NACKing my TopoSort idea!
 
 Red Hat, inc. - For paying my salary, thus financially supporting my hacking.
 
+Samuel Gélineau - For help with programming language theory and design.
+
 And many others...

TODO.md

@@ -0,0 +1,43 @@
+# TODO
+If you're looking for something to do, look here!
+Let us know if you're working on one of the items.
+
+## Package resource
+- [ ] getfiles support on debian [bug](https://github.com/hughsie/PackageKit/issues/118)
+- [ ] directory info on fedora [bug](https://github.com/hughsie/PackageKit/issues/117)
+- [ ] dnf blocker [bug](https://github.com/hughsie/PackageKit/issues/110)
+- [ ] install signal blocker [bug](https://github.com/hughsie/PackageKit/issues/109)
+
+## File resource [bug](https://github.com/purpleidea/mgmt/issues/13) [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove)
+- [ ] ability to make/delete folders
+- [ ] recursive argument (can recursively watch/modify contents)
+- [ ] force argument (can cause switch from file <-> folder)
+- [ ] fanotify support [bug](https://github.com/go-fsnotify/fsnotify/issues/114)
+
+## Exec resource
+- [ ] base resource improvements
+
+## Timer resource
+- [ ] base resource [bug](https://github.com/purpleidea/mgmt/issues/15) [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove)
+- [ ] reset on recompile
+- [ ] increment algorithm (linear, exponential, etc...)
+
+## Virt (libvirt) resource
+- [ ] base resource [bug](https://github.com/purpleidea/mgmt/issues/25)
+
+## Etcd improvements
+- [ ] embedded etcd master
+
+## Language improvements
+- [ ] language design
+- [ ] lexer/parser
+- [ ] automatic language formatter, ala `gofmt`
+- [ ] gedit/gnome-builder/gtksourceview syntax highlighting
+- [ ] vim syntax highlighting
+- [ ] emacs syntax highlighting
+
+## Other
+- [ ] better error/retry handling
+- [ ] deb package target in Makefile
+- [ ] reproducible builds
+- [ ] add your suggestions!

config.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify
@@ -19,30 +19,21 @@ package main
 
 import (
 	"errors"
+	"fmt"
 	"gopkg.in/yaml.v2"
 	"io/ioutil"
 	"log"
+	"reflect"
+	"strings"
 )
 
-type noopTypeConfig struct {
-	Name string `yaml:"name"`
-}
-
-type fileTypeConfig struct {
-	Name    string `yaml:"name"`
-	Path    string `yaml:"path"`
-	Content string `yaml:"content"`
-	State   string `yaml:"state"`
-}
-
-type serviceTypeConfig struct {
-	Name    string `yaml:"name"`
-	State   string `yaml:"state"`
-	Startup string `yaml:"startup"`
+type collectorResConfig struct {
+	Kind    string `yaml:"kind"`
+	Pattern string `yaml:"pattern"` // XXX: Not Implemented
 }
 
 type vertexConfig struct {
-	Type string `yaml:"type"`
+	Kind string `yaml:"kind"`
 	Name string `yaml:"name"`
 }
 
@@ -52,18 +43,26 @@ type edgeConfig struct {
 	To   vertexConfig `yaml:"to"`
 }
 
-type graphConfig struct {
-	Graph string `yaml:"graph"`
-	Types struct {
-		Noop    []noopTypeConfig    `yaml:"noop"`
-		File    []fileTypeConfig    `yaml:"file"`
-		Service []serviceTypeConfig `yaml:"service"`
-	} `yaml:"types"`
-	Edges   []edgeConfig `yaml:"edges"`
-	Comment string       `yaml:"comment"`
+// GraphConfig is the data structure that describes a single graph to run.
+type GraphConfig struct {
+	Graph     string `yaml:"graph"`
+	Resources struct {
+		Noop  []*NoopRes  `yaml:"noop"`
+		Pkg   []*PkgRes   `yaml:"pkg"`
+		File  []*FileRes  `yaml:"file"`
+		Svc   []*SvcRes   `yaml:"svc"`
+		Exec  []*ExecRes  `yaml:"exec"`
+		Timer []*TimerRes `yaml:"timer"`
+	} `yaml:"resources"`
+	Collector []collectorResConfig `yaml:"collect"`
+	Edges     []edgeConfig         `yaml:"edges"`
+	Comment   string               `yaml:"comment"`
+	Hostname  string               `yaml:"hostname"` // uuid for the host
+	Remote    string               `yaml:"remote"`
 }
 
-func (c *graphConfig) Parse(data []byte) error {
+// Parse parses a data stream into the graph structure.
+func (c *GraphConfig) Parse(data []byte) error {
 	if err := yaml.Unmarshal(data, c); err != nil {
 		return err
 	}
@@ -73,54 +72,495 @@ func (c *graphConfig) Parse(data []byte) error {
 	return nil
 }
 
-func GraphFromConfig(filename string) *Graph {
-
-	var NoopMap map[string]*Vertex = make(map[string]*Vertex)
-	var FileMap map[string]*Vertex = make(map[string]*Vertex)
-	var ServiceMap map[string]*Vertex = make(map[string]*Vertex)
-
-	var lookup map[string]map[string]*Vertex = make(map[string]map[string]*Vertex)
-	lookup["noop"] = NoopMap
-	lookup["file"] = FileMap
-	lookup["service"] = ServiceMap
-
+// ParseConfigFromFile takes a filename and returns the graph config structure.
+func ParseConfigFromFile(filename string) *GraphConfig {
 	data, err := ioutil.ReadFile(filename)
 	if err != nil {
-		log.Fatal(err)
+		log.Printf("Config: Error: ParseConfigFromFile: File: %v", err)
+		return nil
 	}
 
-	var config graphConfig
+	var config GraphConfig
 	if err := config.Parse(data); err != nil {
-		log.Fatal(err)
-	}
-	//fmt.Printf("%+v\n", config)	// debug
-
-	g := NewGraph(config.Graph)
-
-	for _, t := range config.Types.Noop {
-		NoopMap[t.Name] = NewVertex(t.Name, "noop")
-		// FIXME: duplicate of name stored twice... where should it go?
-		NoopMap[t.Name].Associate(NewNoopType(t.Name))
-		g.AddVertex(NoopMap[t.Name]) // call standalone in case not part of an edge
+		log.Printf("Config: Error: ParseConfigFromFile: Parse: %v", err)
+		return nil
 	}
 
-	for _, t := range config.Types.File {
-		FileMap[t.Name] = NewVertex(t.Name, "file")
-		// FIXME: duplicate of name stored twice... where should it go?
-		FileMap[t.Name].Associate(NewFileType(t.Name, t.Path, t.Content, t.State))
-		g.AddVertex(FileMap[t.Name]) // call standalone in case not part of an edge
+	return &config
+}
+
+// NewGraphFromConfig returns a new graph from existing input, such as from the
+// existing graph, and a GraphConfig struct.
+func (g *Graph) NewGraphFromConfig(config *GraphConfig, embdEtcd *EmbdEtcd, noop bool) (*Graph, error) {
+	if config.Hostname == "" {
+		return nil, fmt.Errorf("Config: Error: Hostname can't be empty!")
 	}
 
-	for _, t := range config.Types.Service {
-		ServiceMap[t.Name] = NewVertex(t.Name, "service")
-		// FIXME: duplicate of name stored twice... where should it go?
-		ServiceMap[t.Name].Associate(NewServiceType(t.Name, t.State, t.Startup))
-		g.AddVertex(ServiceMap[t.Name]) // call standalone in case not part of an edge
+	var graph *Graph // new graph to return
+	if g == nil {    // FIXME: how can we check for an empty graph?
+		graph = NewGraph("Graph") // give graph a default name
+	} else {
+		graph = g.Copy() // same vertices, since they're pointers!
+	}
+
+	var lookup = make(map[string]map[string]*Vertex)
+
+	//log.Printf("%+v", config) // debug
+
+	// TODO: if defined (somehow)...
+	graph.SetName(config.Graph) // set graph name
+
+	var keep []*Vertex  // list of vertex which are the same in new graph
+	var resources []Res // list of resources to export
+	// use reflection to avoid duplicating code... better options welcome!
+	value := reflect.Indirect(reflect.ValueOf(config.Resources))
+	vtype := value.Type()
+	for i := 0; i < vtype.NumField(); i++ { // number of fields in struct
+		name := vtype.Field(i).Name // string of field name
+		field := value.FieldByName(name)
+		iface := field.Interface() // interface type of value
+		slice := reflect.ValueOf(iface)
+		// XXX: should we just drop these everywhere and have the kind strings be all lowercase?
+		kind := FirstToUpper(name)
+		if DEBUG {
+			log.Printf("Config: Processing: %v...", kind)
+		}
+		for j := 0; j < slice.Len(); j++ { // loop through resources of same kind
+			x := slice.Index(j).Interface()
+			res, ok := x.(Res) // convert to Res type
+			if !ok {
+				return nil, fmt.Errorf("Config: Error: Can't convert: %v of type: %T to Res.", x, x)
+			}
+			if noop {
+				res.Meta().Noop = noop
+			}
+			if _, exists := lookup[kind]; !exists {
+				lookup[kind] = make(map[string]*Vertex)
+			}
+			// XXX: should we export based on a @@ prefix, or a metaparam
+			// like exported => true || exported => (host pattern)||(other pattern?)
+			if !strings.HasPrefix(res.GetName(), "@@") { // not exported resource
+				// XXX: we don't have a way of knowing if any of the
+				// metaparams are undefined, and as a result to set the
+				// defaults that we want! I hate the go yaml parser!!!
+				v := graph.GetVertexMatch(res)
+				if v == nil { // no match found
+					res.Init()
+					v = NewVertex(res)
+					graph.AddVertex(v) // call standalone in case not part of an edge
+				}
+				lookup[kind][res.GetName()] = v // used for constructing edges
+				keep = append(keep, v)          // append
+
+			} else if !noop { // do not export any resources if noop
+				// store for addition to etcd storage...
+				res.SetName(res.GetName()[2:]) //slice off @@
+				res.setKind(kind)              // cheap init
+				resources = append(resources, res)
+			}
+		}
+	}
+	// store in etcd
+	if err := EtcdSetResources(embdEtcd, config.Hostname, resources); err != nil {
+		return nil, fmt.Errorf("Config: Could not export resources: %v", err)
+	}
+
+	// lookup from etcd
+	var hostnameFilter []string // empty to get from everyone
+	kindFilter := []string{}
+	for _, t := range config.Collector {
+		// XXX: should we just drop these everywhere and have the kind strings be all lowercase?
+		kind := FirstToUpper(t.Kind)
+		kindFilter = append(kindFilter, kind)
+	}
+	// do all the graph look ups in one single step, so that if the etcd
+	// database changes, we don't have a partial state of affairs...
+	if len(kindFilter) > 0 { // if kindFilter is empty, don't need to do lookups!
+		var err error
+		resources, err = EtcdGetResources(embdEtcd, hostnameFilter, kindFilter)
+		if err != nil {
+			return nil, fmt.Errorf("Config: Could not collect resources: %v", err)
+		}
+	}
+	for _, res := range resources {
+		matched := false
+		// see if we find a collect pattern that matches
+		for _, t := range config.Collector {
+			// XXX: should we just drop these everywhere and have the kind strings be all lowercase?
+			kind := FirstToUpper(t.Kind)
+			// use t.Kind and optionally t.Pattern to collect from etcd storage
+			log.Printf("Collect: %v; Pattern: %v", kind, t.Pattern)
+
+			// XXX: expand to more complex pattern matching here...
+			if res.Kind() != kind {
+				continue
+			}
+
+			if matched {
+				// we've already matched this resource, should we match again?
+				log.Printf("Config: Warning: Matching %v[%v] again!", kind, res.GetName())
+			}
+			matched = true
+
+			// collect resources but add the noop metaparam
+			if noop {
+				res.Meta().Noop = noop
+			}
+
+			if t.Pattern != "" { // XXX: simplistic for now
+				res.CollectPattern(t.Pattern) // res.Dirname = t.Pattern
+			}
+
+			log.Printf("Collect: %v[%v]: collected!", kind, res.GetName())
+
+			// XXX: similar to other resource add code:
+			if _, exists := lookup[kind]; !exists {
+				lookup[kind] = make(map[string]*Vertex)
+			}
+			v := graph.GetVertexMatch(res)
+			if v == nil { // no match found
+				res.Init() // initialize go channels or things won't work!!!
+				v = NewVertex(res)
+				graph.AddVertex(v) // call standalone in case not part of an edge
+			}
+			lookup[kind][res.GetName()] = v // used for constructing edges
+			keep = append(keep, v)          // append
+
+			//break // let's see if another resource even matches
+		}
+	}
+
+	// get rid of any vertices we shouldn't "keep" (that aren't in new graph)
+	for _, v := range graph.GetVertices() {
+		if !VertexContains(v, keep) {
+			// wait for exit before starting new graph!
+			v.SendEvent(eventExit, true, false)
+			graph.DeleteVertex(v)
+		}
 	}
 
 	for _, e := range config.Edges {
-		g.AddEdge(lookup[e.From.Type][e.From.Name], lookup[e.To.Type][e.To.Name], NewEdge(e.Name))
+		if _, ok := lookup[FirstToUpper(e.From.Kind)]; !ok {
+			return nil, fmt.Errorf("Can't find 'from' resource!")
+		}
+		if _, ok := lookup[FirstToUpper(e.To.Kind)]; !ok {
+			return nil, fmt.Errorf("Can't find 'to' resource!")
+		}
+		if _, ok := lookup[FirstToUpper(e.From.Kind)][e.From.Name]; !ok {
+			return nil, fmt.Errorf("Can't find 'from' name!")
+		}
+		if _, ok := lookup[FirstToUpper(e.To.Kind)][e.To.Name]; !ok {
+			return nil, fmt.Errorf("Can't find 'to' name!")
+		}
+		graph.AddEdge(lookup[FirstToUpper(e.From.Kind)][e.From.Name], lookup[FirstToUpper(e.To.Kind)][e.To.Name], NewEdge(e.Name))
 	}
 
-	return g
+	return graph, nil
+}
+
+// add edges to the vertex in a graph based on if it matches a uuid list
+func (g *Graph) addEdgesByMatchingUUIDS(v *Vertex, uuids []ResUUID) []bool {
+	// search for edges and see what matches!
+	var result []bool
+
+	// loop through each uuid, and see if it matches any vertex
+	for _, uuid := range uuids {
+		var found = false
+		// uuid is a ResUUID object
+		for _, vv := range g.GetVertices() { // search
+			if v == vv { // skip self
+				continue
+			}
+			if DEBUG {
+				log.Printf("Compile: AutoEdge: Match: %v[%v] with UUID: %v[%v]", vv.Kind(), vv.GetName(), uuid.Kind(), uuid.GetName())
+			}
+			// we must match to an effective UUID for the resource,
+			// that is to say, the name value of a res is a helpful
+			// handle, but it is not necessarily a unique identity!
+			// remember, resources can return multiple UUID's each!
+			if UUIDExistsInUUIDs(uuid, vv.GetUUIDs()) {
+				// add edge from: vv -> v
+				if uuid.Reversed() {
+					txt := fmt.Sprintf("AutoEdge: %v[%v] -> %v[%v]", vv.Kind(), vv.GetName(), v.Kind(), v.GetName())
+					log.Printf("Compile: Adding %v", txt)
+					g.AddEdge(vv, v, NewEdge(txt))
+				} else { // edges go the "normal" way, eg: pkg resource
+					txt := fmt.Sprintf("AutoEdge: %v[%v] -> %v[%v]", v.Kind(), v.GetName(), vv.Kind(), vv.GetName())
+					log.Printf("Compile: Adding %v", txt)
+					g.AddEdge(v, vv, NewEdge(txt))
+				}
+				found = true
+				break
+			}
+		}
+		result = append(result, found)
+	}
+	return result
+}
+
+// AutoEdges adds the automatic edges to the graph.
+func (g *Graph) AutoEdges() {
+	log.Println("Compile: Adding AutoEdges...")
+	for _, v := range g.GetVertices() { // for each vertexes autoedges
+		if !v.Meta().AutoEdge { // is the metaparam true?
+			continue
+		}
+		autoEdgeObj := v.AutoEdges()
+		if autoEdgeObj == nil {
+			log.Printf("%v[%v]: Config: No auto edges were found!", v.Kind(), v.GetName())
+			continue // next vertex
+		}
+
+		for { // while the autoEdgeObj has more uuids to add...
+			uuids := autoEdgeObj.Next() // get some!
+			if uuids == nil {
+				log.Printf("%v[%v]: Config: The auto edge list is empty!", v.Kind(), v.GetName())
+				break // inner loop
+			}
+			if DEBUG {
+				log.Println("Compile: AutoEdge: UUIDS:")
+				for i, u := range uuids {
+					log.Printf("Compile: AutoEdge: UUID%d: %v", i, u)
+				}
+			}
+
+			// match and add edges
+			result := g.addEdgesByMatchingUUIDS(v, uuids)
+
+			// report back, and find out if we should continue
+			if !autoEdgeObj.Test(result) {
+				break
+			}
+		}
+	}
+}
+
+// AutoGrouper is the required interface to implement for an autogroup algorithm
+type AutoGrouper interface {
+	// listed in the order these are typically called in...
+	name() string                                  // friendly identifier
+	init(*Graph) error                             // only call once
+	vertexNext() (*Vertex, *Vertex, error)         // mostly algorithmic
+	vertexCmp(*Vertex, *Vertex) error              // can we merge these ?
+	vertexMerge(*Vertex, *Vertex) (*Vertex, error) // vertex merge fn to use
+	edgeMerge(*Edge, *Edge) *Edge                  // edge merge fn to use
+	vertexTest(bool) (bool, error)                 // call until false
+}
+
+// baseGrouper is the base type for implementing the AutoGrouper interface
+type baseGrouper struct {
+	graph    *Graph    // store a pointer to the graph
+	vertices []*Vertex // cached list of vertices
+	i        int
+	j        int
+	done     bool
+}
+
+// name provides a friendly name for the logs to see
+func (ag *baseGrouper) name() string {
+	return "baseGrouper"
+}
+
+// init is called only once and before using other AutoGrouper interface methods
+// the name method is the only exception: call it any time without side effects!
+func (ag *baseGrouper) init(g *Graph) error {
+	if ag.graph != nil {
+		return fmt.Errorf("The init method has already been called!")
+	}
+	ag.graph = g                               // pointer
+	ag.vertices = ag.graph.GetVerticesSorted() // cache in deterministic order!
+	ag.i = 0
+	ag.j = 0
+	if len(ag.vertices) == 0 { // empty graph
+		ag.done = true
+		return nil
+	}
+	return nil
+}
+
+// vertexNext is a simple iterator that loops through vertex (pair) combinations
+// an intelligent algorithm would selectively offer only valid pairs of vertices
+// these should satisfy logical grouping requirements for the autogroup designs!
+// the desired algorithms can override, but keep this method as a base iterator!
+func (ag *baseGrouper) vertexNext() (v1, v2 *Vertex, err error) {
+	// this does a for v... { for w... { return v, w }} but stepwise!
+	l := len(ag.vertices)
+	if ag.i < l {
+		v1 = ag.vertices[ag.i]
+	}
+	if ag.j < l {
+		v2 = ag.vertices[ag.j]
+	}
+
+	// in case the vertex was deleted
+	if !ag.graph.HasVertex(v1) {
+		v1 = nil
+	}
+	if !ag.graph.HasVertex(v2) {
+		v2 = nil
+	}
+
+	// two nested loops...
+	if ag.j < l {
+		ag.j++
+	}
+	if ag.j == l {
+		ag.j = 0
+		if ag.i < l {
+			ag.i++
+		}
+		if ag.i == l {
+			ag.done = true
+		}
+	}
+
+	return
+}
+
+func (ag *baseGrouper) vertexCmp(v1, v2 *Vertex) error {
+	if v1 == nil || v2 == nil {
+		return fmt.Errorf("Vertex is nil!")
+	}
+	if v1 == v2 { // skip yourself
+		return fmt.Errorf("Vertices are the same!")
+	}
+	if v1.Kind() != v2.Kind() { // we must group similar kinds
+		// TODO: maybe future resources won't need this limitation?
+		return fmt.Errorf("The two resources aren't the same kind!")
+	}
+	// someone doesn't want to group!
+	if !v1.Meta().AutoGroup || !v2.Meta().AutoGroup {
+		return fmt.Errorf("One of the autogroup flags is false!")
+	}
+	if v1.Res.IsGrouped() { // already grouped!
+		return fmt.Errorf("Already grouped!")
+	}
+	if len(v2.Res.GetGroup()) > 0 { // already has children grouped!
+		return fmt.Errorf("Already has groups!")
+	}
+	if !v1.Res.GroupCmp(v2.Res) { // resource groupcmp failed!
+		return fmt.Errorf("The GroupCmp failed!")
+	}
+	return nil // success
+}
+
+func (ag *baseGrouper) vertexMerge(v1, v2 *Vertex) (v *Vertex, err error) {
+	// NOTE: it's important to use w.Res instead of w, b/c
+	// the w by itself is the *Vertex obj, not the *Res obj
+	// which is contained within it! They both satisfy the
+	// Res interface, which is why both will compile! :(
+	err = v1.Res.GroupRes(v2.Res) // GroupRes skips stupid groupings
+	return                        // success or fail, and no need to merge the actual vertices!
+}
+
+func (ag *baseGrouper) edgeMerge(e1, e2 *Edge) *Edge {
+	return e1 // noop
+}
+
+// vertexTest processes the results of the grouping for the algorithm to know
+// return an error if something went horribly wrong, and bool false to stop
+func (ag *baseGrouper) vertexTest(b bool) (bool, error) {
+	// NOTE: this particular baseGrouper version doesn't track what happens
+	// because since we iterate over every pair, we don't care which merge!
+	if ag.done {
+		return false, nil
+	}
+	return true, nil
+}
+
+// TODO: this algorithm may not be correct in all cases. replace if needed!
+type nonReachabilityGrouper struct {
+	baseGrouper // "inherit" what we want, and reimplement the rest
+}
+
+func (ag *nonReachabilityGrouper) name() string {
+	return "nonReachabilityGrouper"
+}
+
+// this algorithm relies on the observation that if there's a path from a to b,
+// then they *can't* be merged (b/c of the existing dependency) so therefore we
+// merge anything that *doesn't* satisfy this condition or that of the reverse!
+func (ag *nonReachabilityGrouper) vertexNext() (v1, v2 *Vertex, err error) {
+	for {
+		v1, v2, err = ag.baseGrouper.vertexNext() // get all iterable pairs
+		if err != nil {
+			log.Fatalf("Error running autoGroup(vertexNext): %v", err)
+		}
+
+		if v1 != v2 { // ignore self cmp early (perf optimization)
+			// if NOT reachable, they're viable...
+			out1 := ag.graph.Reachability(v1, v2)
+			out2 := ag.graph.Reachability(v2, v1)
+			if len(out1) == 0 && len(out2) == 0 {
+				return // return v1 and v2, they're viable
+			}
+		}
+
+		// if we got here, it means we're skipping over this candidate!
+		if ok, err := ag.baseGrouper.vertexTest(false); err != nil {
+			log.Fatalf("Error running autoGroup(vertexTest): %v", err)
+		} else if !ok {
+			return nil, nil, nil // done!
+		}
+
+		// the vertexTest passed, so loop and try with a new pair...
+	}
+}
+
+// autoGroup is the mechanical auto group "runner" that runs the interface spec
+func (g *Graph) autoGroup(ag AutoGrouper) chan string {
+	strch := make(chan string) // output log messages here
+	go func(strch chan string) {
+		strch <- fmt.Sprintf("Compile: Grouping: Algorithm: %v...", ag.name())
+		if err := ag.init(g); err != nil {
+			log.Fatalf("Error running autoGroup(init): %v", err)
+		}
+
+		for {
+			var v, w *Vertex
+			v, w, err := ag.vertexNext() // get pair to compare
+			if err != nil {
+				log.Fatalf("Error running autoGroup(vertexNext): %v", err)
+			}
+			merged := false
+			// save names since they change during the runs
+			vStr := fmt.Sprintf("%s", v) // valid even if it is nil
+			wStr := fmt.Sprintf("%s", w)
+
+			if err := ag.vertexCmp(v, w); err != nil { // cmp ?
+				if DEBUG {
+					strch <- fmt.Sprintf("Compile: Grouping: !GroupCmp for: %s into %s", wStr, vStr)
+				}
+
+				// remove grouped vertex and merge edges (res is safe)
+			} else if err := g.VertexMerge(v, w, ag.vertexMerge, ag.edgeMerge); err != nil { // merge...
+				strch <- fmt.Sprintf("Compile: Grouping: !VertexMerge for: %s into %s", wStr, vStr)
+
+			} else { // success!
+				strch <- fmt.Sprintf("Compile: Grouping: Success for: %s into %s", wStr, vStr)
+				merged = true // woo
+			}
+
+			// did these get used?
+			if ok, err := ag.vertexTest(merged); err != nil {
+				log.Fatalf("Error running autoGroup(vertexTest): %v", err)
+			} else if !ok {
+				break // done!
+			}
+		}
+
+		close(strch)
+		return
+	}(strch) // call function
+	return strch
+}
+
+// AutoGroup runs the auto grouping on the graph and prints out log messages
+func (g *Graph) AutoGroup() {
+	// receive log messages from channel...
+	// this allows test cases to avoid printing them when they're unwanted!
+	// TODO: this algorithm may not be correct in all cases. replace if needed!
+	for str := range g.autoGroup(&nonReachabilityGrouper{}) {
+		log.Println(str)
+	}
 }

configwatch.go

@@ -0,0 +1,224 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"gopkg.in/fsnotify.v1"
+	//"github.com/go-fsnotify/fsnotify" // git master of "gopkg.in/fsnotify.v1"
+	"log"
+	"math"
+	"path"
+	"strings"
+	"sync"
+	"syscall"
+)
+
+// ConfigWatcher returns events on a channel anytime one of its files events.
+type ConfigWatcher struct {
+	ch        chan string
+	wg        sync.WaitGroup
+	closechan chan struct{}
+}
+
+// NewConfigWatcher creates a new ConfigWatcher struct.
+func NewConfigWatcher() *ConfigWatcher {
+	return &ConfigWatcher{
+		ch:        make(chan string),
+		closechan: make(chan struct{}),
+	}
+}
+
+// The Add method adds a new file path to watch for events on.
+func (obj *ConfigWatcher) Add(file ...string) {
+	if len(file) == 0 {
+		return
+	}
+	if len(file) > 1 {
+		for _, f := range file { // add all the files...
+			obj.Add(f) // recurse
+		}
+		return
+	}
+	// otherwise, add the one file passed in...
+	obj.wg.Add(1)
+	go func() {
+		defer obj.wg.Done()
+		ch := ConfigWatch(file[0])
+		for {
+			select {
+			case <-ch:
+				obj.ch <- file[0]
+				continue
+			case <-obj.closechan:
+				return
+			}
+		}
+	}()
+}
+
+// Events returns a channel to listen on for file events. It closes when it is
+// emptied after the Close() method is called. You can test for closure with the
+// f, more := <-obj.Events() pattern.
+func (obj *ConfigWatcher) Events() chan string {
+	return obj.ch
+}
+
+// Close shuts down the ConfigWatcher object. It closes the Events channel after
+// all the currently pending events have been emptied.
+func (obj *ConfigWatcher) Close() {
+	if obj.ch == nil {
+		return
+	}
+	close(obj.closechan)
+	obj.wg.Wait() // wait until everyone is done sending on obj.ch
+	//obj.ch <- "" // send finished message
+	close(obj.ch)
+	obj.ch = nil
+}
+
+// ConfigWatch writes on the channel everytime an event is seen for the path.
+// XXX: it would be great if we could reuse code between this and the file resource
+// XXX: patch this to submit it as part of go-fsnotify if they're interested...
+func ConfigWatch(file string) chan bool {
+	ch := make(chan bool)
+	go func() {
+		var safename = path.Clean(file) // no trailing slash
+
+		watcher, err := fsnotify.NewWatcher()
+		if err != nil {
+			log.Fatal(err)
+		}
+		defer watcher.Close()
+
+		patharray := PathSplit(safename) // tokenize the path
+		var index = len(patharray)       // starting index
+		var current string               // current "watcher" location
+		var deltaDepth int               // depth delta between watcher and event
+		var send = false                 // send event?
+
+		for {
+			current = strings.Join(patharray[0:index], "/")
+			if current == "" { // the empty string top is the root dir ("/")
+				current = "/"
+			}
+			if DEBUG {
+				log.Printf("Watching: %v", current) // attempting to watch...
+			}
+			// initialize in the loop so that we can reset on rm-ed handles
+			err = watcher.Add(current)
+			if err != nil {
+				if err == syscall.ENOENT {
+					index-- // usually not found, move up one dir
+				} else if err == syscall.ENOSPC {
+					// XXX: occasionally: no space left on device,
+					// XXX: probably due to lack of inotify watches
+					log.Printf("Out of inotify watches for config(%v)", file)
+					log.Fatal(err)
+				} else {
+					log.Printf("Unknown config(%v) error:", file)
+					log.Fatal(err)
+				}
+				index = int(math.Max(1, float64(index)))
+				continue
+			}
+
+			select {
+			case event := <-watcher.Events:
+				// the deeper you go, the bigger the deltaDepth is...
+				// this is the difference between what we're watching,
+				// and the event... doesn't mean we can't watch deeper
+				if current == event.Name {
+					deltaDepth = 0 // i was watching what i was looking for
+
+				} else if HasPathPrefix(event.Name, current) {
+					deltaDepth = len(PathSplit(current)) - len(PathSplit(event.Name)) // -1 or less
+
+				} else if HasPathPrefix(current, event.Name) {
+					deltaDepth = len(PathSplit(event.Name)) - len(PathSplit(current)) // +1 or more
+
+				} else {
+					// TODO different watchers get each others events!
+					// https://github.com/go-fsnotify/fsnotify/issues/95
+					// this happened with two values such as:
+					// event.Name: /tmp/mgmt/f3 and current: /tmp/mgmt/f2
+					continue
+				}
+				//log.Printf("The delta depth is: %v", deltaDepth)
+
+				// if we have what we wanted, awesome, send an event...
+				if event.Name == safename {
+					//log.Println("Event!")
+					// TODO: filter out some of the events, is Write a sufficient minimum?
+					if event.Op&fsnotify.Write == fsnotify.Write {
+						send = true
+					}
+
+					// file removed, move the watch upwards
+					if deltaDepth >= 0 && (event.Op&fsnotify.Remove == fsnotify.Remove) {
+						//log.Println("Removal!")
+						watcher.Remove(current)
+						index--
+					}
+
+					// we must be a parent watcher, so descend in
+					if deltaDepth < 0 {
+						watcher.Remove(current)
+						index++
+					}
+
+					// if safename starts with event.Name, we're above, and no event should be sent
+				} else if HasPathPrefix(safename, event.Name) {
+					//log.Println("Above!")
+
+					if deltaDepth >= 0 && (event.Op&fsnotify.Remove == fsnotify.Remove) {
+						log.Println("Removal!")
+						watcher.Remove(current)
+						index--
+					}
+
+					if deltaDepth < 0 {
+						log.Println("Parent!")
+						if PathPrefixDelta(safename, event.Name) == 1 { // we're the parent dir
+							//send = true
+						}
+						watcher.Remove(current)
+						index++
+					}
+
+					// if event.Name startswith safename, send event, we're already deeper
+				} else if HasPathPrefix(event.Name, safename) {
+					//log.Println("Event2!")
+					//send = true
+				}
+
+			case err := <-watcher.Errors:
+				log.Printf("error: %v", err)
+				log.Fatal(err)
+
+			}
+
+			// do our event sending all together to avoid duplicate msgs
+			if send {
+				send = false
+				ch <- true
+			}
+		}
+		//close(ch)
+	}()
+	return ch
+}

converger.go

@@ -0,0 +1,379 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"fmt"
+	"sync"
+	"time"
+)
+
+// TODO: we could make a new function that masks out the state of certain
+// UUID's, but at the moment the new Timer code has obsoleted the need...
+
+// Converger is the general interface for implementing a convergence watcher
+type Converger interface { // TODO: need a better name
+	Register() ConvergerUUID
+	IsConverged(ConvergerUUID) bool         // is the UUID converged ?
+	SetConverged(ConvergerUUID, bool) error // set the converged state of the UUID
+	Unregister(ConvergerUUID)
+	Start()
+	Pause()
+	Loop(bool)
+	ConvergedTimer(ConvergerUUID) <-chan time.Time
+	Status() map[uint64]bool
+	Timeout() int                // returns the timeout that this was created with
+	SetStateFn(func(bool) error) // sets the stateFn
+}
+
+// ConvergerUUID is the interface resources can use to notify with if converged
+// you'll need to use part of the Converger interface to Register initially too
+type ConvergerUUID interface {
+	ID() uint64   // get Id
+	Name() string // get a friendly name
+	SetName(string)
+	IsValid() bool // has Id been initialized ?
+	InvalidateID() // set Id to nil
+	IsConverged() bool
+	SetConverged(bool) error
+	Unregister()
+	ConvergedTimer() <-chan time.Time
+	StartTimer() (func() error, error) // cancellable is the same as StopTimer()
+	ResetTimer() error                 // resets counter to zero
+	StopTimer() error
+}
+
+// converger is an implementation of the Converger interface
+type converger struct {
+	timeout   int              // must be zero (instant) or greater seconds to run
+	stateFn   func(bool) error // run on converged state changes with state bool
+	converged bool             // did we converge (state changes of this run Fn)
+	channel   chan struct{}    // signal here to run an isConverged check
+	control   chan bool        // control channel for start/pause
+	mutex     sync.RWMutex     // used for controlling access to status and lastid
+	lastid    uint64
+	status    map[uint64]bool
+}
+
+// convergerUUID is an implementation of the ConvergerUUID interface
+type convergerUUID struct {
+	converger Converger
+	id        uint64
+	name      string // user defined, friendly name
+	mutex     sync.Mutex
+	timer     chan struct{}
+	running   bool // is the above timer running?
+}
+
+// NewConverger builds a new converger struct
+func NewConverger(timeout int, stateFn func(bool) error) *converger {
+	return &converger{
+		timeout: timeout,
+		stateFn: stateFn,
+		channel: make(chan struct{}),
+		control: make(chan bool),
+		lastid:  0,
+		status:  make(map[uint64]bool),
+	}
+}
+
+// Register assigns a ConvergerUUID to the caller
+func (obj *converger) Register() ConvergerUUID {
+	obj.mutex.Lock()
+	defer obj.mutex.Unlock()
+	obj.lastid++
+	obj.status[obj.lastid] = false // initialize as not converged
+	return &convergerUUID{
+		converger: obj,
+		id:        obj.lastid,
+		name:      fmt.Sprintf("%d", obj.lastid), // some default
+		timer:     nil,
+		running:   false,
+	}
+}
+
+// IsConverged gets the converged status of a uuid
+func (obj *converger) IsConverged(uuid ConvergerUUID) bool {
+	if !uuid.IsValid() {
+		panic(fmt.Sprintf("Id of ConvergerUUID(%s) is nil!", uuid.Name()))
+	}
+	obj.mutex.RLock()
+	isConverged, found := obj.status[uuid.ID()] // lookup
+	obj.mutex.RUnlock()
+	if !found {
+		panic("Id of ConvergerUUID is unregistered!")
+	}
+	return isConverged
+}
+
+// SetConverged updates the converger with the converged state of the UUID
+func (obj *converger) SetConverged(uuid ConvergerUUID, isConverged bool) error {
+	if !uuid.IsValid() {
+		return fmt.Errorf("Id of ConvergerUUID(%s) is nil!", uuid.Name())
+	}
+	obj.mutex.Lock()
+	if _, found := obj.status[uuid.ID()]; !found {
+		panic("Id of ConvergerUUID is unregistered!")
+	}
+	obj.status[uuid.ID()] = isConverged // set
+	obj.mutex.Unlock()                  // unlock *before* poke or deadlock!
+	if isConverged != obj.converged {   // only poke if it would be helpful
+		// run in a go routine so that we never block... just queue up!
+		// this allows us to send events, even if we haven't started...
+		go func() { obj.channel <- struct{}{} }()
+	}
+	return nil
+}
+
+// isConverged returns true if *every* registered uuid has converged
+func (obj *converger) isConverged() bool {
+	obj.mutex.RLock() // take a read lock
+	defer obj.mutex.RUnlock()
+	for _, v := range obj.status {
+		if !v { // everyone must be converged for this to be true
+			return false
+		}
+	}
+	return true
+}
+
+// Unregister dissociates the ConvergedUUID from the converged checking
+func (obj *converger) Unregister(uuid ConvergerUUID) {
+	if !uuid.IsValid() {
+		panic(fmt.Sprintf("Id of ConvergerUUID(%s) is nil!", uuid.Name()))
+	}
+	obj.mutex.Lock()
+	uuid.StopTimer() // ignore any errors
+	delete(obj.status, uuid.ID())
+	obj.mutex.Unlock()
+	uuid.InvalidateID()
+}
+
+// Start causes a Converger object to start or resume running
+func (obj *converger) Start() {
+	obj.control <- true
+}
+
+// Pause causes a Converger object to stop running temporarily
+func (obj *converger) Pause() { // FIXME: add a sync ACK on pause before return
+	obj.control <- false
+}
+
+// Loop is the main loop for a Converger object; it usually runs in a goroutine
+// TODO: we could eventually have each resource tell us as soon as it converges
+// and then keep track of the time delays here, to avoid callers needing select
+// NOTE: when we have very short timeouts, if we start before all the resources
+// have joined the map, then it might appears as if we converged before we did!
+func (obj *converger) Loop(startPaused bool) {
+	if obj.control == nil {
+		panic("Converger not initialized correctly")
+	}
+	if startPaused { // start paused without racing
+		select {
+		case e := <-obj.control:
+			if !e {
+				panic("Converger expected true!")
+			}
+		}
+	}
+	for {
+		select {
+		case e := <-obj.control: // expecting "false" which means pause!
+			if e {
+				panic("Converger expected false!")
+			}
+			// now i'm paused...
+			select {
+			case e := <-obj.control:
+				if !e {
+					panic("Converger expected true!")
+				}
+				// restart
+				// kick once to refresh the check...
+				go func() { obj.channel <- struct{}{} }()
+				continue
+			}
+
+		case <-obj.channel:
+			if !obj.isConverged() {
+				if obj.converged { // we're doing a state change
+					if obj.stateFn != nil {
+						// call an arbitrary function
+						if err := obj.stateFn(false); err != nil {
+							// FIXME: what to do on error ?
+						}
+					}
+				}
+				obj.converged = false
+				continue
+			}
+
+			// we have converged!
+			if obj.timeout >= 0 { // only run if timeout is valid
+				if !obj.converged { // we're doing a state change
+					if obj.stateFn != nil {
+						// call an arbitrary function
+						if err := obj.stateFn(true); err != nil {
+							// FIXME: what to do on error ?
+						}
+					}
+				}
+			}
+			obj.converged = true
+			// loop and wait again...
+		}
+	}
+}
+
+// ConvergedTimer adds a timeout to a select call and blocks until then
+// TODO: this means we could eventually have per resource converged timeouts
+func (obj *converger) ConvergedTimer(uuid ConvergerUUID) <-chan time.Time {
+	// be clever: if i'm already converged, this timeout should block which
+	// avoids unnecessary new signals being sent! this avoids fast loops if
+	// we have a low timeout, or in particular a timeout == 0
+	if uuid.IsConverged() {
+		// blocks the case statement in select forever!
+		return TimeAfterOrBlock(-1)
+	}
+	return TimeAfterOrBlock(obj.timeout)
+}
+
+// Status returns a map of the converged status of each UUID.
+func (obj *converger) Status() map[uint64]bool {
+	status := make(map[uint64]bool)
+	obj.mutex.RLock() // take a read lock
+	defer obj.mutex.RUnlock()
+	for k, v := range obj.status { // make a copy to avoid the mutex
+		status[k] = v
+	}
+	return status
+}
+
+// Timeout returns the timeout in seconds that converger was created with. This
+// is useful to avoid passing in the timeout value separately when you're
+// already passing in the Converger struct.
+func (obj *converger) Timeout() int {
+	return obj.timeout
+}
+
+// SetStateFn sets the state function to be run on change of converged state.
+func (obj *converger) SetStateFn(stateFn func(bool) error) {
+	obj.stateFn = stateFn
+}
+
+// Id returns the unique id of this UUID object
+func (obj *convergerUUID) ID() uint64 {
+	return obj.id
+}
+
+// Name returns a user defined name for the specific convergerUUID.
+func (obj *convergerUUID) Name() string {
+	return obj.name
+}
+
+// SetName sets a user defined name for the specific convergerUUID.
+func (obj *convergerUUID) SetName(name string) {
+	obj.name = name
+}
+
+// IsValid tells us if the id is valid or has already been destroyed
+func (obj *convergerUUID) IsValid() bool {
+	return obj.id != 0 // an id of 0 is invalid
+}
+
+// InvalidateID marks the id as no longer valid
+func (obj *convergerUUID) InvalidateID() {
+	obj.id = 0 // an id of 0 is invalid
+}
+
+// IsConverged is a helper function to the regular IsConverged method
+func (obj *convergerUUID) IsConverged() bool {
+	return obj.converger.IsConverged(obj)
+}
+
+// SetConverged is a helper function to the regular SetConverged notification
+func (obj *convergerUUID) SetConverged(isConverged bool) error {
+	return obj.converger.SetConverged(obj, isConverged)
+}
+
+// Unregister is a helper function to unregister myself
+func (obj *convergerUUID) Unregister() {
+	obj.converger.Unregister(obj)
+}
+
+// ConvergedTimer is a helper around the regular ConvergedTimer method
+func (obj *convergerUUID) ConvergedTimer() <-chan time.Time {
+	return obj.converger.ConvergedTimer(obj)
+}
+
+// StartTimer runs an invisible timer that automatically converges on timeout.
+func (obj *convergerUUID) StartTimer() (func() error, error) {
+	obj.mutex.Lock()
+	if !obj.running {
+		obj.timer = make(chan struct{})
+		obj.running = true
+	} else {
+		obj.mutex.Unlock()
+		return obj.StopTimer, fmt.Errorf("Timer already started!")
+	}
+	obj.mutex.Unlock()
+	go func() {
+		for {
+			select {
+			case _, ok := <-obj.timer: // reset signal channel
+				if !ok { // channel is closed
+					return // false to exit
+				}
+				obj.SetConverged(false)
+
+			case <-obj.ConvergedTimer():
+				obj.SetConverged(true) // converged!
+				select {
+				case _, ok := <-obj.timer: // reset signal channel
+					if !ok { // channel is closed
+						return // false to exit
+					}
+				}
+			}
+		}
+	}()
+	return obj.StopTimer, nil
+}
+
+// ResetTimer resets the counter to zero if using a StartTimer internally.
+func (obj *convergerUUID) ResetTimer() error {
+	obj.mutex.Lock()
+	defer obj.mutex.Unlock()
+	if obj.running {
+		obj.timer <- struct{}{} // send the reset message
+		return nil
+	}
+	return fmt.Errorf("Timer hasn't been started!")
+}
+
+// StopTimer stops the running timer permanently until a StartTimer is run.
+func (obj *convergerUUID) StopTimer() error {
+	obj.mutex.Lock()
+	defer obj.mutex.Unlock()
+	if !obj.running {
+		return fmt.Errorf("Timer isn't running!")
+	}
+	close(obj.timer)
+	obj.running = false
+	return nil
+}

docker/Dockerfile

@@ -0,0 +1,22 @@
+FROM golang:1.6.2
+
+MAINTAINER Michał Czeraszkiewicz <contact@czerasz.com>
+
+# Set the reset cache variable
+# Read more here: http://czerasz.com/2014/11/13/docker-tip-and-tricks/#use-refreshedat-variable-for-better-cache-control
+ENV REFRESHED_AT 2016-05-10
+
+# Update the package list to be able to use required packages
+RUN apt-get update
+
+# Change the working directory
+WORKDIR /go/src/mgmt
+
+# Copy all the files to the working directory
+COPY . /go/src/mgmt
+
+# Install dependencies
+RUN make deps
+
+# Build the binary
+RUN make build

docker/Dockerfile.development

@@ -0,0 +1,31 @@
+FROM golang:1.6.2
+
+MAINTAINER Michał Czeraszkiewicz <contact@czerasz.com>
+
+# Set the reset cache variable
+# Read more here: http://czerasz.com/2014/11/13/docker-tip-and-tricks/#use-refreshedat-variable-for-better-cache-control
+ENV REFRESHED_AT 2016-05-14
+
+RUN apt-get update
+
+# Setup User to match Host User
+# Give the nre user superuser permissions
+ARG USER_ID=1000
+ARG GROUP_ID=1000
+ARG USER_NAME=mgmt
+ARG GROUP_NAME=$USER_NAME
+RUN groupadd --gid $GROUP_ID $GROUP_NAME && \
+	useradd --create-home --home /home/$USER_NAME --uid ${USER_ID} --gid $GROUP_NAME --groups sudo $USER_NAME && \
+	echo "$USER_NAME ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
+
+# Copy all the files to the working directory
+COPY . /home/$USER_NAME/mgmt
+
+# Change working directory
+WORKDIR /home/$USER_NAME/mgmt
+
+# Install dependencies
+RUN make deps
+
+# Change user
+USER ${USER_NAME}

docker/scripts/build

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+script_directory="$( cd "$( dirname "$0" )" && pwd )"
+project_directory=$script_directory/../..
+
+# Specify the Docker image name
+image_name='purpleidea/mgmt'
+
+# Build the image which contains the compiled binary
+docker build -t $image_name \
+	--file=$project_directory/docker/Dockerfile $project_directory
+
+# Remove the container if it already exists
+docker rm -f mgmt-export 2> /dev/null
+
+# Start the container in background so we can "copy out" the binary
+docker run -d --name=mgmt-export $image_name bash -c 'while true; sleep 1000; done'
+
+# Remove the current binary
+rm $project_directory/mgmt 2> /dev/null
+
+# Get the binary from the container
+docker cp mgmt-export:/go/src/mgmt/mgmt $project_directory/mgmt
+
+# Remove the container
+docker rm -f mgmt-export 2> /dev/null

docker/scripts/build-development

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+# Stop on any error
+set -e
+
+script_directory="$( cd "$( dirname "$0" )" && pwd )"
+project_directory=$script_directory/../..
+
+# Specify the Docker image name
+image_name='purpleidea/mgmt:development'
+
+# Build the image
+docker build -t $image_name \
+	--file=$project_directory/docker/Dockerfile.development $project_directory

docker/scripts/run-development

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+# Stop on any error
+set -e
+
+script_directory="$( cd "$( dirname "$0" )" && pwd )"
+project_directory=$script_directory/../..
+
+# Specify the Docker image name
+image_name='purpleidea/mgmt:development'
+
+# Run container in development mode
+docker run --rm --name=mgm_development --user=mgmt \
+	-v $project_directory:/home/mgmt/mgmt \
+	-it $image_name bash

event.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify
@@ -17,20 +17,80 @@
 
 package main
 
-import (
-	"code.google.com/p/go-uuid/uuid"
+//go:generate stringer -type=eventName -output=eventname_stringer.go
+type eventName int
+
+const (
+	eventNil eventName = iota
+	eventExit
+	eventStart
+	eventPause
+	eventPoke
+	eventBackPoke
 )
 
+// Resp is a channel to be used for boolean responses.
+type Resp chan bool
+
+// Event is the main struct that stores event information and responses.
 type Event struct {
-	uuid string
-	Name string
-	Type string
+	Name eventName
+	Resp Resp // channel to send an ack response on, nil to skip
+	//Wg   *sync.WaitGroup // receiver barrier to Wait() for everyone else on
+	Msg      string // some words for fun
+	Activity bool   // did something interesting happen?
 }
 
-func NewEvent(name, t string) *Event {
-	return &Event{
-		uuid: uuid.New(),
-		Name: name,
-		Type: t,
+// ACK sends a single acknowledgement on the channel if one was requested.
+func (event *Event) ACK() {
+	if event.Resp != nil { // if they've requested an ACK
+		event.Resp.ACK()
 	}
 }
+
+// NACK sends a negative acknowledgement message on the channel if one was requested.
+func (event *Event) NACK() {
+	if event.Resp != nil { // if they've requested a NACK
+		event.Resp.NACK()
+	}
+}
+
+// NewResp is just a helper to return the right type of response channel.
+func NewResp() Resp {
+	resp := make(chan bool)
+	return resp
+}
+
+// ACK sends a true value to resp.
+func (resp Resp) ACK() {
+	if resp != nil {
+		resp <- true
+	}
+}
+
+// NACK sends a false value to resp.
+func (resp Resp) NACK() {
+	if resp != nil {
+		resp <- false
+	}
+}
+
+// Wait waits for any response from a Resp channel and returns it.
+func (resp Resp) Wait() bool {
+	return <-resp
+}
+
+// ACKWait waits for a +ive Ack from a Resp channel.
+func (resp Resp) ACKWait() {
+	for {
+		// wait until true value
+		if resp.Wait() {
+			return
+		}
+	}
+}
+
+// GetActivity returns the activity value.
+func (event *Event) GetActivity() bool {
+	return event.Activity
+}

examples/autoedges1.yaml

@@ -0,0 +1,19 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1
+    meta:
+      autoedge: true
+    path: "/tmp/foo/bar/f1"
+    content: |
+      i am f1
+    state: exists
+  - name: file2
+    meta:
+      autoedge: true
+    path: "/tmp/foo/"
+    content: |
+      i am f2
+    state: exists
+edges: []

examples/autoedges2.yaml

@@ -0,0 +1,24 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1
+    meta:
+      autoedge: true
+    path: "/etc/drbd.conf"
+    content: |
+      # this is an mgmt test
+    state: exists
+  - name: file2
+    meta:
+      autoedge: true
+    path: "/tmp/foo/"
+    content: |
+      i am f2
+    state: exists
+  pkg:
+  - name: drbd-utils
+    meta:
+      autoedge: true
+    state: installed
+edges: []

examples/autoedges3.yaml

@@ -0,0 +1,29 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: drbd-utils
+    meta:
+      autoedge: true
+    state: installed
+  file:
+  - name: file1
+    meta:
+      autoedge: true
+    path: "/etc/drbd.conf"
+    content: |
+      # this is an mgmt test
+    state: exists
+  - name: file2
+    meta:
+      autoedge: true
+    path: "/etc/drbd.d/"
+    content: |
+      i am a directory
+    state: exists
+  svc:
+  - name: drbd
+    meta:
+      autoedge: true
+    state: stopped
+edges: []

examples/autogroup1.yaml

@@ -0,0 +1,21 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: drbd-utils
+    meta:
+      autogroup: false
+    state: installed
+  - name: powertop
+    meta:
+      autogroup: true
+    state: installed
+  - name: sl
+    meta:
+      autogroup: true
+    state: installed
+  - name: cowsay
+    meta:
+      autogroup: true
+    state: installed
+edges: []

examples/autogroup2.yaml

@@ -0,0 +1,17 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: powertop
+    meta:
+      autogroup: true
+    state: installed
+  - name: sl
+    meta:
+      autogroup: true
+    state: installed
+  - name: cowsay
+    meta:
+      autogroup: true
+    state: installed
+edges: []

examples/etcd1a.yaml

@@ -0,0 +1,18 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1a
+    path: "/tmp/mgmtA/f1a"
+    content: |
+      i am f1
+    state: exists
+  - name: "@@file2a"
+    path: "/tmp/mgmtA/f2a"
+    content: |
+      i am f2, exported from host A
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtA/"
+edges: []

examples/etcd1b.yaml

@@ -0,0 +1,18 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1b
+    path: "/tmp/mgmtB/f1b"
+    content: |
+      i am f1
+    state: exists
+  - name: "@@file2b"
+    path: "/tmp/mgmtB/f2b"
+    content: |
+      i am f2, exported from host B
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtB/"
+edges: []

examples/etcd1c.yaml

@@ -0,0 +1,18 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1c
+    path: "/tmp/mgmtC/f1c"
+    content: |
+      i am f1
+    state: exists
+  - name: "@@file2c"
+    path: "/tmp/mgmtC/f2c"
+    content: |
+      i am f2, exported from host C
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtC/"
+edges: []

examples/etcd1d.yaml

@@ -0,0 +1,18 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1d
+    path: "/tmp/mgmtD/f1d"
+    content: |
+      i am f1
+    state: exists
+  - name: "@@file2d"
+    path: "/tmp/mgmtD/f2d"
+    content: |
+      i am f2, exported from host D
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtD/"
+edges: []

examples/exec1.yaml

@@ -0,0 +1,59 @@
+---
+graph: mygraph
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec3
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec4
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec2
+- name: e2
+  from:
+    kind: exec
+    name: exec2
+  to:
+    kind: exec
+    name: exec3

examples/exec1a.yaml

@@ -0,0 +1,32 @@
+---
+graph: mygraph
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec2

examples/exec1b.yaml

@@ -0,0 +1,32 @@
+---
+graph: mygraph
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: 'true'
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: 'true'
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec2

examples/exec1c.yaml

@@ -0,0 +1,32 @@
+---
+graph: mygraph
+resources:
+  exec:
+  - name: exec1
+    cmd: echo hello from exec1
+    shell: ''
+    timeout: 0
+    watchcmd: sleep 10s
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: echo hello from exec2
+    shell: ''
+    timeout: 0
+    watchcmd: sleep 10s
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec2

examples/exec1d.yaml

@@ -0,0 +1,15 @@
+---
+graph: mygraph
+resources:
+  exec:
+  - name: exec1
+    cmd: echo hello from exec1
+    shell: ''
+    timeout: 0
+    watchcmd: sleep 5s
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges: []

examples/exec2.yaml

@@ -0,0 +1,83 @@
+---
+graph: mygraph
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec3
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec4
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec5
+    cmd: sleep 15s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec2
+- name: e2
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec3
+- name: e3
+  from:
+    kind: exec
+    name: exec2
+  to:
+    kind: exec
+    name: exec4
+- name: e4
+  from:
+    kind: exec
+    name: exec3
+  to:
+    kind: exec
+    name: exec4

examples/exec3.yaml

@@ -0,0 +1,59 @@
+---
+graph: parallel
+resources:
+  exec:
+  - name: pkg10
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: svc10
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec10
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: pkg15
+    cmd: sleep 15s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: pkg10
+  to:
+    kind: exec
+    name: svc10
+- name: e2
+  from:
+    kind: exec
+    name: svc10
+  to:
+    kind: exec
+    name: exec10

examples/file1.yaml

@@ -1,41 +1,41 @@
 ---
 graph: mygraph
-types:
+resources:
   noop:
   - name: noop1
   file:
   - name: file1
-    path: /tmp/mgmt/f1
+    path: "/tmp/mgmt/f1"
     content: |
       i am f1
     state: exists
   - name: file2
-    path: /tmp/mgmt/f2
+    path: "/tmp/mgmt/f2"
     content: |
       i am f2
     state: exists
   - name: file3
-    path: /tmp/mgmt/f3
+    path: "/tmp/mgmt/f3"
     content: |
       i am f3
     state: exists
   - name: file4
-    path: /tmp/mgmt/f4
+    path: "/tmp/mgmt/f4"
     content: |
       i am f4 and i should not be here
     state: absent
 edges:
 - name: e1
   from:
-    type: file
+    kind: file
     name: file1
   to:
-    type: file
+    kind: file
     name: file2
 - name: e2
   from:
-    type: file
+    kind: file
     name: file2
   to:
-    type: file
+    kind: file
     name: file3

examples/graph0.yaml

@@ -0,0 +1,20 @@
+---
+graph: mygraph
+comment: hello world example
+resources:
+  noop:
+  - name: noop1
+  file:
+  - name: file1
+    path: "/tmp/mgmt-hello-world"
+    content: |
+      hello world from @purpleidea
+    state: exists
+edges:
+- name: e1
+  from:
+    kind: noop
+    name: noop1
+  to:
+    kind: file
+    name: file1

examples/graph10.yaml

@@ -0,0 +1,128 @@
+---
+graph: mygraph
+comment: simple exec fan in to fan out example to demonstrate optimization
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec3
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec4
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec5
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec6
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec7
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec8
+    cmd: sleep 15s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec4
+- name: e2
+  from:
+    kind: exec
+    name: exec2
+  to:
+    kind: exec
+    name: exec4
+- name: e3
+  from:
+    kind: exec
+    name: exec3
+  to:
+    kind: exec
+    name: exec4
+- name: e4
+  from:
+    kind: exec
+    name: exec4
+  to:
+    kind: exec
+    name: exec5
+- name: e5
+  from:
+    kind: exec
+    name: exec4
+  to:
+    kind: exec
+    name: exec6
+- name: e6
+  from:
+    kind: exec
+    name: exec4
+  to:
+    kind: exec
+    name: exec7

examples/graph1a.yaml

@@ -0,0 +1,22 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1
+    path: "/tmp/mgmt/f1"
+    content: |
+      i am f1
+    state: exists
+  - name: file2
+    path: "/tmp/mgmt/f2"
+    content: |
+      i am f2
+    state: exists
+edges:
+- name: e1
+  from:
+    kind: file
+    name: file1
+  to:
+    kind: file
+    name: file2

examples/graph1b.yaml

@@ -0,0 +1,22 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file2
+    path: "/tmp/mgmt/f2"
+    content: |
+      i am f2
+    state: exists
+  - name: file3
+    path: "/tmp/mgmt/f3"
+    content: |
+      i am f3
+    state: exists
+edges:
+- name: e2
+  from:
+    kind: file
+    name: file2
+  to:
+    kind: file
+    name: file3

examples/graph3a.yaml

@@ -0,0 +1,28 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1a
+    path: "/tmp/mgmtA/f1a"
+    content: |
+      i am f1
+    state: exists
+  - name: file2a
+    path: "/tmp/mgmtA/f2a"
+    content: |
+      i am f2
+    state: exists
+  - name: "@@file3a"
+    path: "/tmp/mgmtA/f3a"
+    content: |
+      i am f3, exported from host A
+    state: exists
+  - name: "@@file4a"
+    path: "/tmp/mgmtA/f4a"
+    content: |
+      i am f4, exported from host A
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtA/"
+edges: []

examples/graph3b.yaml

@@ -0,0 +1,28 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1b
+    path: "/tmp/mgmtB/f1b"
+    content: |
+      i am f1
+    state: exists
+  - name: file2b
+    path: "/tmp/mgmtB/f2b"
+    content: |
+      i am f2
+    state: exists
+  - name: "@@file3b"
+    path: "/tmp/mgmtB/f3b"
+    content: |
+      i am f3, exported from host B
+    state: exists
+  - name: "@@file4b"
+    path: "/tmp/mgmtB/f4b"
+    content: |
+      i am f4, exported from host B
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtB/"
+edges: []

examples/graph3c.yaml

@@ -0,0 +1,28 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1c
+    path: "/tmp/mgmtC/f1c"
+    content: |
+      i am f1
+    state: exists
+  - name: file2c
+    path: "/tmp/mgmtC/f2c"
+    content: |
+      i am f2
+    state: exists
+  - name: "@@file3c"
+    path: "/tmp/mgmtC/f3c"
+    content: |
+      i am f3, exported from host C
+    state: exists
+  - name: "@@file4c"
+    path: "/tmp/mgmtC/f4c"
+    content: |
+      i am f4, exported from host C
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtC/"
+edges: []

examples/graph4.yaml

@@ -0,0 +1,18 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1
+    path: "/tmp/mgmt/f1"
+    content: |
+      i am f1
+    state: exists
+  - name: "@@file3"
+    path: "/tmp/mgmt/f3"
+    content: |
+      i am f3, exported from host A
+    state: exists
+collect:
+- kind: file
+  pattern: ''
+edges:

examples/graph5.yaml

@@ -0,0 +1,13 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1
+    path: "/tmp/mgmt/f1"
+    content: |
+      i am f1
+    state: exists
+collect:
+- kind: file
+  pattern: ''
+edges:

examples/graph6.yaml

@@ -0,0 +1,6 @@
+---
+graph: mygraph
+resources:
+  noop:
+  - name: noop1
+edges:

examples/graph7.yaml

@@ -0,0 +1,17 @@
+---
+graph: mygraph
+resources:
+  noop:
+  - name: noop1
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:

examples/graph9.yaml

@@ -0,0 +1,77 @@
+---
+graph: mygraph
+comment: simple exec fan in example to demonstrate optimization
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec3
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec4
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec5
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec5
+- name: e2
+  from:
+    kind: exec
+    name: exec2
+  to:
+    kind: exec
+    name: exec5
+- name: e3
+  from:
+    kind: exec
+    name: exec3
+  to:
+    kind: exec
+    name: exec5

examples/noop1.yaml

@@ -0,0 +1,24 @@
+---
+graph: mygraph
+comment: noop example
+resources:
+  noop:
+  - name: noop1
+    meta:
+      noop: true
+  file:
+  - name: file1
+    path: "/tmp/mgmt-hello-noop"
+    content: |
+      hello world from @purpleidea
+    state: exists
+    meta:
+      noop: true
+edges:
+- name: e1
+  from:
+    kind: noop
+    name: noop1
+  to:
+    kind: file
+    name: file1

examples/pkg1.yaml

@@ -0,0 +1,7 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: powertop
+    state: installed
+edges: []

examples/pkg2.yaml

@@ -0,0 +1,7 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: powertop
+    state: uninstalled
+edges: []

examples/purpleidea.service

@@ -0,0 +1,8 @@
+[Unit]
+Description=Fake service for testing
+
+[Service]
+ExecStart=/usr/bin/sleep 8h
+
+[Install]
+WantedBy=multi-user.target

examples/remote1.yaml

@@ -0,0 +1,23 @@
+---
+graph: mygraph
+comment: remote noop example
+resources:
+  noop:
+  - name: noop1
+    meta:
+      noop: true
+  file:
+  - name: file1
+    path: "/tmp/mgmt-remote-hello"
+    content: |
+      hello world from @purpleidea
+    state: exists
+edges:
+- name: e1
+  from:
+    kind: noop
+    name: noop1
+  to:
+    kind: file
+    name: file1
+remote: "ssh://root:password@hostname:22"

examples/svc1.yaml

@@ -1,30 +1,30 @@
 ---
 graph: mygraph
-types:
+resources:
   noop:
   - name: noop1
   file:
   - name: file1
-    path: /tmp/mgmt/f1
+    path: "/tmp/mgmt/f1"
     content: |
       i am f1
     state: exists
-  service:
+  svc:
   - name: purpleidea
     state: running
     startup: enabled
 edges:
 - name: e1
   from:
-    type: noop
+    kind: noop
     name: noop1
   to:
-    type: file
+    kind: file
     name: file1
 - name: e2
   from:
-    type: file
+    kind: file
     name: file1
   to:
-    type: service
+    kind: svc
     name: purpleidea

examples/timer1.yaml

@@ -0,0 +1,25 @@
+---
+graph: mygraph
+comment: timer example
+resources:
+  timer:
+  - name: timer1
+    interval: 30
+  exec:
+  - name: exec1
+    cmd: echo hello world
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: timer
+    name: timer1
+  to:
+    kind: exec
+    name: exec1

examples/timer2.yaml

@@ -0,0 +1,43 @@
+---
+graph: mygraph
+comment: example of multiple timers
+resources:
+  timer:
+  - name: timer1
+    interval: 30
+  - name: timer2
+    interval: 60
+  exec:
+  - name: exec1
+    cmd: echo hello world 30
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: echo hello world 60
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: timer
+    name: timer1
+  to:
+    kind: exec
+    name: exec1
+- name: e2
+  from:
+    kind: timer
+    name: timer2
+  to:
+    kind: exec
+    name: exec2

exec.go

@@ -0,0 +1,439 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"bufio"
+	"bytes"
+	"encoding/gob"
+	"errors"
+	"log"
+	"os/exec"
+	"strings"
+)
+
+func init() {
+	gob.Register(&ExecRes{})
+}
+
+// ExecRes is an exec resource for running commands.
+type ExecRes struct {
+	BaseRes    `yaml:",inline"`
+	State      string `yaml:"state"`      // state: exists/present?, absent, (undefined?)
+	Cmd        string `yaml:"cmd"`        // the command to run
+	Shell      string `yaml:"shell"`      // the (optional) shell to use to run the cmd
+	Timeout    int    `yaml:"timeout"`    // the cmd timeout in seconds
+	WatchCmd   string `yaml:"watchcmd"`   // the watch command to run
+	WatchShell string `yaml:"watchshell"` // the (optional) shell to use to run the watch cmd
+	IfCmd      string `yaml:"ifcmd"`      // the if command to run
+	IfShell    string `yaml:"ifshell"`    // the (optional) shell to use to run the if cmd
+	PollInt    int    `yaml:"pollint"`    // the poll interval for the ifcmd
+}
+
+// NewExecRes is a constructor for this resource. It also calls Init() for you.
+func NewExecRes(name, cmd, shell string, timeout int, watchcmd, watchshell, ifcmd, ifshell string, pollint int, state string) *ExecRes {
+	obj := &ExecRes{
+		BaseRes: BaseRes{
+			Name: name,
+		},
+		Cmd:        cmd,
+		Shell:      shell,
+		Timeout:    timeout,
+		WatchCmd:   watchcmd,
+		WatchShell: watchshell,
+		IfCmd:      ifcmd,
+		IfShell:    ifshell,
+		PollInt:    pollint,
+		State:      state,
+	}
+	obj.Init()
+	return obj
+}
+
+// Init runs some startup code for this resource.
+func (obj *ExecRes) Init() {
+	obj.BaseRes.kind = "Exec"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+}
+
+// validate if the params passed in are valid data
+// FIXME: where should this get called ?
+func (obj *ExecRes) Validate() bool {
+	if obj.Cmd == "" { // this is the only thing that is really required
+		return false
+	}
+
+	// if we have a watch command, then we don't poll with the if command!
+	if obj.WatchCmd != "" && obj.PollInt > 0 {
+		return false
+	}
+
+	return true
+}
+
+// wraps the scanner output in a channel
+func (obj *ExecRes) BufioChanScanner(scanner *bufio.Scanner) (chan string, chan error) {
+	ch, errch := make(chan string), make(chan error)
+	go func() {
+		for scanner.Scan() {
+			ch <- scanner.Text() // blocks here ?
+			if e := scanner.Err(); e != nil {
+				errch <- e // send any misc errors we encounter
+				//break // TODO ?
+			}
+		}
+		close(ch)
+		errch <- scanner.Err() // eof or some err
+		close(errch)
+	}()
+	return ch, errch
+}
+
+// Watch is the primary listener for this resource and it outputs events.
+func (obj *ExecRes) Watch(processChan chan Event) {
+	if obj.IsWatching() {
+		return
+	}
+	obj.SetWatching(true)
+	defer obj.SetWatching(false)
+	cuuid := obj.converger.Register()
+	defer cuuid.Unregister()
+
+	var send = false // send event?
+	var exit = false
+	bufioch, errch := make(chan string), make(chan error)
+
+	if obj.WatchCmd != "" {
+		var cmdName string
+		var cmdArgs []string
+		if obj.WatchShell == "" {
+			// call without a shell
+			// FIXME: are there still whitespace splitting issues?
+			split := strings.Fields(obj.WatchCmd)
+			cmdName = split[0]
+			//d, _ := os.Getwd() // TODO: how does this ever error ?
+			//cmdName = path.Join(d, cmdName)
+			cmdArgs = split[1:]
+		} else {
+			cmdName = obj.Shell // usually bash, or sh
+			cmdArgs = []string{"-c", obj.WatchCmd}
+		}
+		cmd := exec.Command(cmdName, cmdArgs...)
+		//cmd.Dir = "" // look for program in pwd ?
+
+		cmdReader, err := cmd.StdoutPipe()
+		if err != nil {
+			log.Printf("%v[%v]: Error creating StdoutPipe for Cmd: %v", obj.Kind(), obj.GetName(), err)
+			log.Fatal(err) // XXX: how should we handle errors?
+		}
+		scanner := bufio.NewScanner(cmdReader)
+
+		defer cmd.Wait() // XXX: is this necessary?
+		defer func() {
+			// FIXME: without wrapping this in this func it panic's
+			// when running examples/graph8d.yaml
+			cmd.Process.Kill() // TODO: is this necessary?
+		}()
+		if err := cmd.Start(); err != nil {
+			log.Printf("%v[%v]: Error starting Cmd: %v", obj.Kind(), obj.GetName(), err)
+			log.Fatal(err) // XXX: how should we handle errors?
+		}
+
+		bufioch, errch = obj.BufioChanScanner(scanner)
+	}
+
+	for {
+		obj.SetState(resStateWatching) // reset
+		select {
+		case text := <-bufioch:
+			cuuid.SetConverged(false)
+			// each time we get a line of output, we loop!
+			log.Printf("%v[%v]: Watch output: %s", obj.Kind(), obj.GetName(), text)
+			if text != "" {
+				send = true
+			}
+
+		case err := <-errch:
+			cuuid.SetConverged(false) // XXX ?
+			if err == nil {           // EOF
+				// FIXME: add an "if watch command ends/crashes"
+				// restart or generate error option
+				log.Printf("%v[%v]: Reached EOF", obj.Kind(), obj.GetName())
+				return
+			}
+			log.Printf("%v[%v]: Error reading input?: %v", obj.Kind(), obj.GetName(), err)
+			log.Fatal(err)
+			// XXX: how should we handle errors?
+
+		case event := <-obj.events:
+			cuuid.SetConverged(false)
+			if exit, send = obj.ReadEvent(&event); exit {
+				return // exit
+			}
+
+		case <-cuuid.ConvergedTimer():
+			cuuid.SetConverged(true) // converged!
+			continue
+		}
+
+		// do all our event sending all together to avoid duplicate msgs
+		if send {
+			send = false
+			// it is okay to invalidate the clean state on poke too
+			obj.isStateOK = false // something made state dirty
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
+		}
+	}
+}
+
+// CheckApply checks the resource state and applies the resource if the bool
+// input is true. It returns error info and if the state check passed or not.
+// TODO: expand the IfCmd to be a list of commands
+func (obj *ExecRes) CheckApply(apply bool) (checkok bool, err error) {
+	log.Printf("%v[%v]: CheckApply(%t)", obj.Kind(), obj.GetName(), apply)
+
+	// if there is a watch command, but no if command, run based on state
+	if obj.WatchCmd != "" && obj.IfCmd == "" {
+		if obj.isStateOK {
+			return true, nil
+		}
+
+		// if there is no watcher, but there is an onlyif check, run it to see
+	} else if obj.IfCmd != "" { // && obj.WatchCmd == ""
+		// there is a watcher, but there is also an if command
+		//} else if obj.IfCmd != "" && obj.WatchCmd != "" {
+
+		if obj.PollInt > 0 { // && obj.WatchCmd == ""
+			// XXX have the Watch() command output onlyif poll events...
+			// XXX we can optimize by saving those results for returning here
+			// return XXX
+		}
+
+		var cmdName string
+		var cmdArgs []string
+		if obj.IfShell == "" {
+			// call without a shell
+			// FIXME: are there still whitespace splitting issues?
+			split := strings.Fields(obj.IfCmd)
+			cmdName = split[0]
+			//d, _ := os.Getwd() // TODO: how does this ever error ?
+			//cmdName = path.Join(d, cmdName)
+			cmdArgs = split[1:]
+		} else {
+			cmdName = obj.IfShell // usually bash, or sh
+			cmdArgs = []string{"-c", obj.IfCmd}
+		}
+		err = exec.Command(cmdName, cmdArgs...).Run()
+		if err != nil {
+			// TODO: check exit value
+			return true, nil // don't run
+		}
+
+		// if there is no watcher and no onlyif check, assume we should run
+	} else { // if obj.WatchCmd == "" && obj.IfCmd == "" {
+		// just run if state is dirty
+		if obj.isStateOK {
+			return true, nil
+		}
+	}
+
+	// state is not okay, no work done, exit, but without error
+	if !apply {
+		return false, nil
+	}
+
+	// apply portion
+	log.Printf("%v[%v]: Apply", obj.Kind(), obj.GetName())
+	var cmdName string
+	var cmdArgs []string
+	if obj.Shell == "" {
+		// call without a shell
+		// FIXME: are there still whitespace splitting issues?
+		// TODO: we could make the split character user selectable...!
+		split := strings.Fields(obj.Cmd)
+		cmdName = split[0]
+		//d, _ := os.Getwd() // TODO: how does this ever error ?
+		//cmdName = path.Join(d, cmdName)
+		cmdArgs = split[1:]
+	} else {
+		cmdName = obj.Shell // usually bash, or sh
+		cmdArgs = []string{"-c", obj.Cmd}
+	}
+	cmd := exec.Command(cmdName, cmdArgs...)
+	//cmd.Dir = "" // look for program in pwd ?
+	var out bytes.Buffer
+	cmd.Stdout = &out
+
+	if err = cmd.Start(); err != nil {
+		log.Printf("%v[%v]: Error starting Cmd: %v", obj.Kind(), obj.GetName(), err)
+		return false, err
+	}
+
+	timeout := obj.Timeout
+	if timeout == 0 { // zero timeout means no timer, so disable it
+		timeout = -1
+	}
+	done := make(chan error)
+	go func() { done <- cmd.Wait() }()
+
+	select {
+	case err = <-done:
+		if err != nil {
+			log.Printf("%v[%v]: Error waiting for Cmd: %v", obj.Kind(), obj.GetName(), err)
+			return false, err
+		}
+
+	case <-TimeAfterOrBlock(timeout):
+		log.Printf("%v[%v]: Timeout waiting for Cmd", obj.Kind(), obj.GetName())
+		//cmd.Process.Kill() // TODO: is this necessary?
+		return false, errors.New("Timeout waiting for Cmd!")
+	}
+
+	// TODO: if we printed the stdout while the command is running, this
+	// would be nice, but it would require terminal log output that doesn't
+	// interleave all the parallel parts which would mix it all up...
+	if s := out.String(); s == "" {
+		log.Printf("Exec[%v]: Command output is empty!", obj.Name)
+	} else {
+		log.Printf("Exec[%v]: Command output is:", obj.Name)
+		log.Printf(out.String())
+	}
+	// XXX: return based on exit value!!
+
+	// the state tracking is for exec resources that can't "detect" their
+	// state, and assume it's invalid when the Watch() function triggers.
+	// if we apply state successfully, we should reset it here so that we
+	// know that we have applied since the state was set not ok by event!
+	obj.isStateOK = true // reset
+	return false, nil    // success
+}
+
+// ExecUUID is the UUID struct for ExecRes.
+type ExecUUID struct {
+	BaseUUID
+	Cmd   string
+	IfCmd string
+	// TODO: add more elements here
+}
+
+// if and only if they are equivalent, return true
+// if they are not equivalent, return false
+func (obj *ExecUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*ExecUUID)
+	if !ok {
+		return false
+	}
+	if obj.Cmd != res.Cmd {
+		return false
+	}
+	// TODO: add more checks here
+	//if obj.Shell != res.Shell {
+	//	return false
+	//}
+	//if obj.Timeout != res.Timeout {
+	//	return false
+	//}
+	//if obj.WatchCmd != res.WatchCmd {
+	//	return false
+	//}
+	//if obj.WatchShell != res.WatchShell {
+	//	return false
+	//}
+	if obj.IfCmd != res.IfCmd {
+		return false
+	}
+	//if obj.PollInt != res.PollInt {
+	//	return false
+	//}
+	//if obj.State != res.State {
+	//	return false
+	//}
+	return true
+}
+
+// The AutoEdges method returns the AutoEdges. In this case none are used.
+func (obj *ExecRes) AutoEdges() AutoEdge {
+	// TODO: parse as many exec params to look for auto edges, for example
+	// the path of the binary in the Cmd variable might be from in a pkg
+	return nil
+}
+
+// GetUUIDs includes all params to make a unique identification of this object.
+// Most resources only return one, although some resources can return multiple.
+func (obj *ExecRes) GetUUIDs() []ResUUID {
+	x := &ExecUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		Cmd:      obj.Cmd,
+		IfCmd:    obj.IfCmd,
+		// TODO: add more params here
+	}
+	return []ResUUID{x}
+}
+
+// GroupCmp returns whether two resources can be grouped together or not.
+func (obj *ExecRes) GroupCmp(r Res) bool {
+	_, ok := r.(*ExecRes)
+	if !ok {
+		return false
+	}
+	return false // not possible atm
+}
+
+// Compare two resources and return if they are equivalent.
+func (obj *ExecRes) Compare(res Res) bool {
+	switch res.(type) {
+	case *ExecRes:
+		res := res.(*ExecRes)
+		if !obj.BaseRes.Compare(res) { // call base Compare
+			return false
+		}
+
+		if obj.Name != res.Name {
+			return false
+		}
+		if obj.Cmd != res.Cmd {
+			return false
+		}
+		if obj.Shell != res.Shell {
+			return false
+		}
+		if obj.Timeout != res.Timeout {
+			return false
+		}
+		if obj.WatchCmd != res.WatchCmd {
+			return false
+		}
+		if obj.WatchShell != res.WatchShell {
+			return false
+		}
+		if obj.IfCmd != res.IfCmd {
+			return false
+		}
+		if obj.PollInt != res.PollInt {
+			return false
+		}
+		if obj.State != res.State {
+			return false
+		}
+	default:
+		return false
+	}
+	return true
+}

file.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify
@@ -18,12 +18,11 @@
 package main
 
 import (
-	"code.google.com/p/go-uuid/uuid"
 	"crypto/sha256"
 	"encoding/hex"
-	"fmt"
 	"gopkg.in/fsnotify.v1"
 	//"github.com/go-fsnotify/fsnotify" // git master of "gopkg.in/fsnotify.v1"
+	"encoding/gob"
 	"io"
 	"log"
 	"math"
@@ -33,40 +32,95 @@ import (
 	"syscall"
 )
 
-type FileType struct {
-	uuid      string
-	Type      string      // always "file"
-	Name      string      // name variable
-	Events    chan string // FIXME: eventually a struct for the event?
-	Path      string      // path variable (should default to name)
-	Content   string
-	State     string // state: exists/present?, absent, (undefined?)
+func init() {
+	gob.Register(&FileRes{})
+}
+
+// FileRes is a file and directory resource.
+type FileRes struct {
+	BaseRes   `yaml:",inline"`
+	Path      string `yaml:"path"` // path variable (should default to name)
+	Dirname   string `yaml:"dirname"`
+	Basename  string `yaml:"basename"`
+	Content   string `yaml:"content"`
+	State     string `yaml:"state"` // state: exists/present?, absent, (undefined?)
 	sha256sum string
 }
 
-func NewFileType(name, path, content, state string) *FileType {
+// NewFileRes is a constructor for this resource. It also calls Init() for you.
+func NewFileRes(name, path, dirname, basename, content, state string) *FileRes {
 	// FIXME if path = nil, path = name ...
-	return &FileType{
-		uuid:      uuid.New(),
-		Type:      "file",
-		Name:      name,
-		Events:    make(chan string, 1), // XXX: chan size?
+	obj := &FileRes{
+		BaseRes: BaseRes{
+			Name: name,
+		},
 		Path:      path,
+		Dirname:   dirname,
+		Basename:  basename,
 		Content:   content,
 		State:     state,
 		sha256sum: "",
 	}
+	obj.Init()
+	return obj
 }
 
-// File watcher for files and directories
-// Modify with caution, probably important to write some test cases first!
-func (obj FileType) Watch(v *Vertex) {
-	// obj.Path: file or directory
-	//var recursive bool = false
-	//var isdir = (obj.Path[len(obj.Path)-1:] == "/") // dirs have trailing slashes
-	//fmt.Printf("IsDirectory: %v\n", isdir)
+// Init runs some startup code for this resource.
+func (obj *FileRes) Init() {
+	obj.BaseRes.kind = "File"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+}
 
-	var safename = path.Clean(obj.Path) // no trailing slash
+// GetPath returns the actual path to use for this resource. It computes this
+// after analysis of the path, dirname and basename values.
+func (obj *FileRes) GetPath() string {
+	d := Dirname(obj.Path)
+	b := Basename(obj.Path)
+	if !obj.Validate() || (obj.Dirname == "" && obj.Basename == "") {
+		return obj.Path
+	} else if obj.Dirname == "" {
+		return d + obj.Basename
+	} else if obj.Basename == "" {
+		return obj.Dirname + b
+	} else { // if obj.dirname != "" && obj.basename != "" {
+		return obj.Dirname + obj.Basename
+	}
+}
+
+// validate if the params passed in are valid data
+func (obj *FileRes) Validate() bool {
+	if obj.Dirname != "" {
+		// must end with /
+		if obj.Dirname[len(obj.Dirname)-1:] != "/" {
+			return false
+		}
+	}
+	if obj.Basename != "" {
+		// must not start with /
+		if obj.Basename[0:1] == "/" {
+			return false
+		}
+	}
+	return true
+}
+
+// Watch is the primary listener for this resource and it outputs events.
+// This one is a file watcher for files and directories.
+// Modify with caution, it is probably important to write some test cases first!
+// obj.GetPath(): file or directory
+func (obj *FileRes) Watch(processChan chan Event) {
+	if obj.IsWatching() {
+		return
+	}
+	obj.SetWatching(true)
+	defer obj.SetWatching(false)
+	cuuid := obj.converger.Register()
+	defer cuuid.Unregister()
+
+	//var recursive bool = false
+	//var isdir = (obj.GetPath()[len(obj.GetPath())-1:] == "/") // dirs have trailing slashes
+	//log.Printf("IsDirectory: %v", isdir)
+	var safename = path.Clean(obj.GetPath()) // no trailing slash
 
 	watcher, err := fsnotify.NewWatcher()
 	if err != nil {
@@ -77,90 +131,84 @@ func (obj FileType) Watch(v *Vertex) {
 	patharray := PathSplit(safename) // tokenize the path
 	var index = len(patharray)       // starting index
 	var current string               // current "watcher" location
-	var delta_depth int              // depth delta between watcher and event
+	var deltaDepth int               // depth delta between watcher and event
 	var send = false                 // send event?
-	var extraCheck = false
+	var exit = false
+	var dirty = false
 
 	for {
 		current = strings.Join(patharray[0:index], "/")
 		if current == "" { // the empty string top is the root dir ("/")
 			current = "/"
 		}
-		log.Printf("Watching: %v\n", current) // attempting to watch...
-
+		if DEBUG {
+			log.Printf("File[%v]: Watching: %v", obj.GetName(), current) // attempting to watch...
+		}
 		// initialize in the loop so that we can reset on rm-ed handles
 		err = watcher.Add(current)
 		if err != nil {
+			if DEBUG {
+				log.Printf("File[%v]: watcher.Add(%v): Error: %v", obj.GetName(), current, err)
+			}
 			if err == syscall.ENOENT {
 				index-- // usually not found, move up one dir
 			} else if err == syscall.ENOSPC {
-				// XXX: i sometimes see: no space left on device
-				// XXX: why causes this to happen ?
-				log.Printf("Strange file[%v] error: %+v\n", obj.Name, err.Error) // 0x408da0
+				// XXX: occasionally: no space left on device,
+				// XXX: probably due to lack of inotify watches
+				log.Printf("%v[%v]: Out of inotify watches!", obj.Kind(), obj.GetName())
 				log.Fatal(err)
 			} else {
-				log.Printf("Unknown file[%v] error:\n", obj.Name)
+				log.Printf("Unknown file[%v] error:", obj.Name)
 				log.Fatal(err)
 			}
 			index = int(math.Max(1, float64(index)))
 			continue
 		}
 
-		// XXX: check state after inotify started
-		// SMALL RACE: after we terminate watch, till when it's started
-		// something could have gotten created/changed/etc... right?
-		if extraCheck {
-			extraCheck = false
-			// XXX
-			//if exists ... {
-			//	send signal
-			//	continue
-			//	change index? i don't think so. be thorough and check
-			//}
-		}
-
+		obj.SetState(resStateWatching) // reset
 		select {
 		case event := <-watcher.Events:
-			// the deeper you go, the bigger the delta_depth is...
+			if DEBUG {
+				log.Printf("File[%v]: Watch(%v), Event(%v): %v", obj.GetName(), current, event.Name, event.Op)
+			}
+			cuuid.SetConverged(false) // XXX: technically i can detect if the event is erroneous or not first
+			// the deeper you go, the bigger the deltaDepth is...
 			// this is the difference between what we're watching,
 			// and the event... doesn't mean we can't watch deeper
 			if current == event.Name {
-				delta_depth = 0 // i was watching what i was looking for
+				deltaDepth = 0 // i was watching what i was looking for
 
 			} else if HasPathPrefix(event.Name, current) {
-				delta_depth = len(PathSplit(current)) - len(PathSplit(event.Name)) // -1 or less
+				deltaDepth = len(PathSplit(current)) - len(PathSplit(event.Name)) // -1 or less
 
 			} else if HasPathPrefix(current, event.Name) {
-				delta_depth = len(PathSplit(event.Name)) - len(PathSplit(current)) // +1 or more
+				deltaDepth = len(PathSplit(event.Name)) - len(PathSplit(current)) // +1 or more
 
 			} else {
-				// XXX multiple watchers receive each others events
+				// TODO different watchers get each others events!
 				// https://github.com/go-fsnotify/fsnotify/issues/95
 				// this happened with two values such as:
 				// event.Name: /tmp/mgmt/f3 and current: /tmp/mgmt/f2
-				// are the different watchers getting each others events??
-				//log.Printf("The delta depth is NaN...\n")
-				//log.Printf("Value of event.Name is: %v\n", event.Name)
-				//log.Printf("........    current is: %v\n", current)
-				//log.Fatal("The delta depth is NaN!")
 				continue
 			}
-			//log.Printf("The delta depth is: %v\n", delta_depth)
+			//log.Printf("The delta depth is: %v", deltaDepth)
 
 			// if we have what we wanted, awesome, send an event...
 			if event.Name == safename {
 				//log.Println("Event!")
+				// FIXME: should all these below cases trigger?
 				send = true
+				dirty = true
 
 				// file removed, move the watch upwards
-				if delta_depth >= 0 && (event.Op&fsnotify.Remove == fsnotify.Remove) {
+				if deltaDepth >= 0 && (event.Op&fsnotify.Remove == fsnotify.Remove) {
 					//log.Println("Removal!")
 					watcher.Remove(current)
 					index--
 				}
 
 				// we must be a parent watcher, so descend in
-				if delta_depth < 0 {
+				if deltaDepth < 0 {
 					watcher.Remove(current)
 					index++
 				}
@@ -169,13 +217,18 @@ func (obj FileType) Watch(v *Vertex) {
 			} else if HasPathPrefix(safename, event.Name) {
 				//log.Println("Above!")
 
-				if delta_depth >= 0 && (event.Op&fsnotify.Remove == fsnotify.Remove) {
+				if deltaDepth >= 0 && (event.Op&fsnotify.Remove == fsnotify.Remove) {
 					log.Println("Removal!")
 					watcher.Remove(current)
 					index--
 				}
 
-				if delta_depth < 0 {
+				if deltaDepth < 0 {
+					log.Println("Parent!")
+					if PathPrefixDelta(safename, event.Name) == 1 { // we're the parent dir
+						send = true
+						dirty = true
+					}
 					watcher.Remove(current)
 					index++
 				}
@@ -184,37 +237,45 @@ func (obj FileType) Watch(v *Vertex) {
 			} else if HasPathPrefix(event.Name, safename) {
 				//log.Println("Event2!")
 				send = true
+				dirty = true
 			}
 
 		case err := <-watcher.Errors:
-			log.Println("error:", err)
+			cuuid.SetConverged(false) // XXX ?
+			log.Printf("error: %v", err)
 			log.Fatal(err)
-			v.Events <- fmt.Sprintf("file: %v", "error")
+			//obj.events <- fmt.Sprintf("file: %v", "error") // XXX: how should we handle errors?
 
-		case exit := <-obj.Events:
-			if exit == "exit" {
-				return
-			} else {
-				log.Fatal("Unknown event: %v\n", exit)
+		case event := <-obj.events:
+			cuuid.SetConverged(false)
+			if exit, send = obj.ReadEvent(&event); exit {
+				return // exit
 			}
+			//dirty = false // these events don't invalidate state
+
+		case <-cuuid.ConvergedTimer():
+			cuuid.SetConverged(true) // converged!
+			continue
 		}
 
 		// do all our event sending all together to avoid duplicate msgs
 		if send {
 			send = false
-			//log.Println("Sending event!")
-			//v.Events <- fmt.Sprintf("file(%v): %v", obj.Path, event.Op)
-			v.Events <- fmt.Sprintf("file(%v): %v", obj.Path, "event!") // FIXME: use struct
+			// only invalid state on certain types of events
+			if dirty {
+				dirty = false
+				obj.isStateOK = false // something made state dirty
+			}
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
 		}
 	}
 }
 
-func (obj FileType) Exit() bool {
-	obj.Events <- "exit"
-	return true
-}
-
-func (obj FileType) HashSHA256fromContent() string {
+// HashSHA256fromContent computes the hash of the file contents and returns it.
+// It also caches the value if it can.
+func (obj *FileRes) HashSHA256fromContent() string {
 	if obj.sha256sum != "" { // return if already computed
 		return obj.sha256sum
 	}
@@ -225,115 +286,244 @@ func (obj FileType) HashSHA256fromContent() string {
 	return obj.sha256sum
 }
 
-func (obj FileType) StateOK() bool {
-	if _, err := os.Stat(obj.Path); os.IsNotExist(err) {
-		// no such file or directory
-		if obj.State == "absent" {
-			return true // missing file should be missing, phew :)
-		} else {
-			// state invalid, skip expensive checksums
-			return false
-		}
+// FileHashSHA256Check computes the hash of the actual file and compares it to
+// the computed hash of the resources file contents.
+func (obj *FileRes) FileHashSHA256Check() (bool, error) {
+	if PathIsDir(obj.GetPath()) { // assert
+		log.Fatal("This should only be called on a File resource.")
 	}
-
-	// TODO: add file mode check here...
-
-	if PathIsDir(obj.Path) {
-		return obj.StateOKDir()
-	} else {
-		return obj.StateOKFile()
-	}
-}
-
-func (obj FileType) StateOKFile() bool {
-	if PathIsDir(obj.Path) {
-		log.Fatal("This should only be called on a File type.")
-	}
-
-	// run a diff, and return true if needs changing
-
+	// run a diff, and return true if it needs changing
 	hash := sha256.New()
-
-	f, err := os.Open(obj.Path)
+	f, err := os.Open(obj.GetPath())
 	if err != nil {
-		//log.Fatal(err)
-		return false
+		if e, ok := err.(*os.PathError); ok && (e.Err.(syscall.Errno) == syscall.ENOENT) {
+			return false, nil // no "error", file is just absent
+		}
+		return false, err
 	}
 	defer f.Close()
-
 	if _, err := io.Copy(hash, f); err != nil {
-		//log.Fatal(err)
-		return false
+		return false, err
 	}
-
 	sha256sum := hex.EncodeToString(hash.Sum(nil))
-	//fmt.Printf("sha256sum: %v\n", sha256sum)
-
+	//log.Printf("sha256sum: %v", sha256sum)
 	if obj.HashSHA256fromContent() == sha256sum {
-		return true
+		return true, nil
 	}
-
-	return false
+	return false, nil
 }
 
-func (obj FileType) StateOKDir() bool {
-	if !PathIsDir(obj.Path) {
-		log.Fatal("This should only be called on a Dir type.")
-	}
-
-	// XXX: not implemented
-	log.Fatal("Not implemented!")
-	return false
-}
-
-func (obj FileType) Apply() bool {
-	fmt.Printf("Apply->%v[%v]\n", obj.Type, obj.Name)
-
-	if PathIsDir(obj.Path) {
-		return obj.ApplyDir()
-	} else {
-		return obj.ApplyFile()
-	}
-}
-
-func (obj FileType) ApplyFile() bool {
-
-	if PathIsDir(obj.Path) {
-		log.Fatal("This should only be called on a File type.")
+// FileApply writes the resource file contents out to the correct path. This
+// implementation doesn't try to be particularly clever in any way.
+func (obj *FileRes) FileApply() error {
+	if PathIsDir(obj.GetPath()) {
+		log.Fatal("This should only be called on a File resource.")
 	}
 
 	if obj.State == "absent" {
-		log.Printf("About to remove: %v\n", obj.Path)
-		err := os.Remove(obj.Path)
-		if err != nil {
-			return false
-		}
-		return true
+		log.Printf("About to remove: %v", obj.GetPath())
+		err := os.Remove(obj.GetPath())
+		return err // either nil or not, for success or failure
 	}
 
-	//fmt.Println("writing: " + filename)
-	f, err := os.Create(obj.Path)
+	f, err := os.Create(obj.GetPath())
 	if err != nil {
-		log.Println("error:", err)
-		return false
+		return nil
 	}
 	defer f.Close()
 
 	_, err = io.WriteString(f, obj.Content)
 	if err != nil {
-		log.Println("error:", err)
+		return err
+	}
+
+	return nil // success
+}
+
+// CheckApply checks the resource state and applies the resource if the bool
+// input is true. It returns error info and if the state check passed or not.
+func (obj *FileRes) CheckApply(apply bool) (checkok bool, err error) {
+	log.Printf("%v[%v]: CheckApply(%t)", obj.Kind(), obj.GetName(), apply)
+
+	if obj.isStateOK { // cache the state
+		return true, nil
+	}
+
+	if _, err = os.Stat(obj.GetPath()); os.IsNotExist(err) {
+		// no such file or directory
+		if obj.State == "absent" {
+			// missing file should be missing, phew :)
+			obj.isStateOK = true
+			return true, nil
+		}
+	}
+	err = nil // reset
+
+	// FIXME: add file mode check here...
+
+	if PathIsDir(obj.GetPath()) {
+		log.Fatal("Not implemented!") // XXX
+	} else {
+		ok, err := obj.FileHashSHA256Check()
+		if err != nil {
+			return false, err
+		}
+		if ok {
+			obj.isStateOK = true
+			return true, nil
+		}
+		// if no err, but !ok, then we continue on...
+	}
+
+	// state is not okay, no work done, exit, but without error
+	if !apply {
+		return false, nil
+	}
+
+	// apply portion
+	log.Printf("%v[%v]: Apply", obj.Kind(), obj.GetName())
+	if PathIsDir(obj.GetPath()) {
+		log.Fatal("Not implemented!") // XXX
+	} else {
+		err = obj.FileApply()
+		if err != nil {
+			return false, err
+		}
+	}
+
+	obj.isStateOK = true
+	return false, nil // success
+}
+
+// FileUUID is the UUID struct for FileRes.
+type FileUUID struct {
+	BaseUUID
+	path string
+}
+
+// if and only if they are equivalent, return true
+// if they are not equivalent, return false
+func (obj *FileUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*FileUUID)
+	if !ok {
 		return false
 	}
-
-	return true
+	return obj.path == res.path
 }
 
-func (obj FileType) ApplyDir() bool {
-	if !PathIsDir(obj.Path) {
-		log.Fatal("This should only be called on a Dir type.")
+// FileResAutoEdges holds the state of the auto edge generator.
+type FileResAutoEdges struct {
+	data    []ResUUID
+	pointer int
+	found   bool
+}
+
+// Next returns the next automatic edge.
+func (obj *FileResAutoEdges) Next() []ResUUID {
+	if obj.found {
+		log.Fatal("Shouldn't be called anymore!")
 	}
+	if len(obj.data) == 0 { // check length for rare scenarios
+		return nil
+	}
+	value := obj.data[obj.pointer]
+	obj.pointer++
+	return []ResUUID{value} // we return one, even though api supports N
+}
 
-	// XXX: not implemented
-	log.Fatal("Not implemented!")
+// Test gets results of the earlier Next() call, & returns if we should continue!
+func (obj *FileResAutoEdges) Test(input []bool) bool {
+	// if there aren't any more remaining
+	if len(obj.data) <= obj.pointer {
+		return false
+	}
+	if obj.found { // already found, done!
+		return false
+	}
+	if len(input) != 1 { // in case we get given bad data
+		log.Fatal("Expecting a single value!")
+	}
+	if input[0] { // if a match is found, we're done!
+		obj.found = true // no more to find!
+		return false
+	}
+	return true // keep going
+}
+
+// AutoEdges generates a simple linear sequence of each parent directory from
+// the bottom up!
+func (obj *FileRes) AutoEdges() AutoEdge {
+	var data []ResUUID                             // store linear result chain here...
+	values := PathSplitFullReversed(obj.GetPath()) // build it
+	_, values = values[0], values[1:]              // get rid of first value which is me!
+	for _, x := range values {
+		var reversed = true // cheat by passing a pointer
+		data = append(data, &FileUUID{
+			BaseUUID: BaseUUID{
+				name:     obj.GetName(),
+				kind:     obj.Kind(),
+				reversed: &reversed,
+			},
+			path: x, // what matters
+		}) // build list
+	}
+	return &FileResAutoEdges{
+		data:    data,
+		pointer: 0,
+		found:   false,
+	}
+}
+
+// GetUUIDs includes all params to make a unique identification of this object.
+// Most resources only return one, although some resources can return multiple.
+func (obj *FileRes) GetUUIDs() []ResUUID {
+	x := &FileUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		path:     obj.GetPath(),
+	}
+	return []ResUUID{x}
+}
+
+// GroupCmp returns whether two resources can be grouped together or not.
+func (obj *FileRes) GroupCmp(r Res) bool {
+	_, ok := r.(*FileRes)
+	if !ok {
+		return false
+	}
+	// TODO: we might be able to group directory children into a single
+	// recursive watcher in the future, thus saving fanotify watches
+	return false // not possible atm
+}
+
+// Compare two resources and return if they are equivalent.
+func (obj *FileRes) Compare(res Res) bool {
+	switch res.(type) {
+	case *FileRes:
+		res := res.(*FileRes)
+		if !obj.BaseRes.Compare(res) { // call base Compare
+			return false
+		}
+
+		if obj.Name != res.Name {
+			return false
+		}
+		if obj.GetPath() != res.Path {
+			return false
+		}
+		if obj.Content != res.Content {
+			return false
+		}
+		if obj.State != res.State {
+			return false
+		}
+	default:
+		return false
+	}
 	return true
 }
+
+// CollectPattern applies the pattern for collection resources.
+func (obj *FileRes) CollectPattern(pattern string) {
+	// XXX: currently the pattern for files can only override the Dirname variable :P
+	obj.Dirname = pattern // XXX: simplistic for now
+}

gopath/.gitignore

@@ -0,0 +1,2 @@
+bin/
+pkg/

gopath/src

@@ -0,0 +1 @@
+../vendor

main.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify
@@ -19,7 +19,10 @@ package main
 
 import (
 	"fmt"
-	"github.com/codegangsta/cli"
+	etcdtypes "github.com/coreos/etcd/pkg/types"
+	"github.com/coreos/pkg/capnslog"
+	"github.com/urfave/cli"
+	"io/ioutil"
 	"log"
 	"os"
 	"os/signal"
@@ -30,12 +33,16 @@ import (
 
 // set at compile time
 var (
-	version string
 	program string
+	version string
+	prefix  = fmt.Sprintf("/var/lib/%s/", program)
 )
 
+// variables controlling verbosity
 const (
-	DEBUG = false
+	DEBUG   = false // add additional log messages
+	TRACE   = false // add execution flow log messages
+	VERBOSE = false // add extra log message output
 )
 
 // signal handler
@@ -48,7 +55,6 @@ func waitForSignal(exit chan bool) {
 	select {
 	case e := <-signals: // any signal will do
 		if e == os.Interrupt {
-			fmt.Println() // put ^C char from terminal on its own line
 			log.Println("Interrupted by ^C")
 		} else {
 			log.Println("Interrupted by signal")
@@ -58,64 +64,367 @@ func waitForSignal(exit chan bool) {
 	}
 }
 
-func run(c *cli.Context) {
-	var start int64 = time.Now().UnixNano()
+// run is the main run target.
+func run(c *cli.Context) error {
+	var start = time.Now().UnixNano()
+	log.Printf("This is: %v, version: %v", program, version)
+	log.Printf("Main: Start: %v", start)
+
+	hostname, _ := os.Hostname()
+	// allow passing in the hostname, instead of using --hostname
+	if c.IsSet("file") {
+		if config := ParseConfigFromFile(c.String("file")); config != nil {
+			if h := config.Hostname; h != "" {
+				hostname = h
+			}
+		}
+	}
+	if c.IsSet("hostname") { // override by cli
+		if h := c.String("hostname"); h != "" {
+			hostname = h
+		}
+	}
+	noop := c.Bool("noop")
+
+	seeds, err := etcdtypes.NewURLs(
+		FlattenListWithSplit(c.StringSlice("seeds"), []string{",", ";", " "}),
+	)
+	if err != nil && len(c.StringSlice("seeds")) > 0 {
+		log.Printf("Main: Error: seeds didn't parse correctly!")
+		return cli.NewExitError("", 1)
+	}
+	clientURLs, err := etcdtypes.NewURLs(
+		FlattenListWithSplit(c.StringSlice("client-urls"), []string{",", ";", " "}),
+	)
+	if err != nil && len(c.StringSlice("client-urls")) > 0 {
+		log.Printf("Main: Error: clientURLs didn't parse correctly!")
+		return cli.NewExitError("", 1)
+	}
+	serverURLs, err := etcdtypes.NewURLs(
+		FlattenListWithSplit(c.StringSlice("server-urls"), []string{",", ";", " "}),
+	)
+	if err != nil && len(c.StringSlice("server-urls")) > 0 {
+		log.Printf("Main: Error: serverURLs didn't parse correctly!")
+		return cli.NewExitError("", 1)
+	}
+
+	idealClusterSize := uint16(c.Int("ideal-cluster-size"))
+	if idealClusterSize < 1 {
+		log.Printf("Main: Error: idealClusterSize should be at least one!")
+		return cli.NewExitError("", 1)
+	}
+
+	if c.IsSet("file") && c.IsSet("puppet") {
+		log.Println("Main: Error: the --file and --puppet parameters cannot be used together!")
+		return cli.NewExitError("", 1)
+	}
+
+	if c.Bool("no-server") && len(c.StringSlice("remote")) > 0 {
+		// TODO: in this case, we won't be able to tunnel stuff back to
+		// here, so if we're okay with every remote graph running in an
+		// isolated mode, then this is okay. Improve on this if there's
+		// someone who really wants to be able to do this.
+		log.Println("Main: Error: the --no-server and --remote parameters cannot be used together!")
+		return cli.NewExitError("", 1)
+	}
+
+	cConns := uint16(c.Int("cconns"))
+	if cConns < 0 {
+		log.Printf("Main: Error: --cconns should be at least zero!")
+		return cli.NewExitError("", 1)
+	}
+
+	if c.IsSet("converged-timeout") && cConns > 0 && len(c.StringSlice("remote")) > c.Int("cconns") {
+		log.Printf("Main: Error: combining --converged-timeout with more remotes than available connections will never converge!")
+		return cli.NewExitError("", 1)
+	}
+
+	depth := uint16(c.Int("depth"))
+	if depth < 0 { // user should not be using this argument manually
+		log.Printf("Main: Error: negative values for --depth are not permitted!")
+		return cli.NewExitError("", 1)
+	}
+
+	if c.IsSet("prefix") && c.Bool("tmp-prefix") {
+		log.Println("Main: Error: combining --prefix and the request for a tmp prefix is illogical!")
+		return cli.NewExitError("", 1)
+	}
+	if s := c.String("prefix"); c.IsSet("prefix") && s != "" {
+		prefix = s
+	}
+
+	// make sure the working directory prefix exists
+	if c.Bool("tmp-prefix") || os.MkdirAll(prefix, 0770) != nil {
+		if c.Bool("tmp-prefix") || c.Bool("allow-tmp-prefix") {
+			if prefix, err = ioutil.TempDir("", program+"-"); err != nil {
+				log.Printf("Main: Error: Can't create temporary prefix!")
+				return cli.NewExitError("", 1)
+			}
+			log.Println("Main: Warning: Working prefix directory is temporary!")
+
+		} else {
+			log.Printf("Main: Error: Can't create prefix!")
+			return cli.NewExitError("", 1)
+		}
+	}
+	log.Printf("Main: Working prefix is: %s", prefix)
+
 	var wg sync.WaitGroup
 	exit := make(chan bool) // exit signal
-	log.Printf("This is: %v, version: %v\n", program, version)
+	var G, fullGraph *Graph
 
-	// exit after `exittime` seconds for no reason at all...
-	if i := c.Int("exittime"); i > 0 {
+	// exit after `max-runtime` seconds for no reason at all...
+	if i := c.Int("max-runtime"); i > 0 {
 		go func() {
 			time.Sleep(time.Duration(i) * time.Second)
 			exit <- true
 		}()
 	}
 
-	// build the graph from a config file
-	G := GraphFromConfig(c.String("file"))
-	log.Printf("Graph: %v\n", G) // show graph
+	// setup converger
+	converger := NewConverger(
+		c.Int("converged-timeout"),
+		nil, // stateFn gets added in by EmbdEtcd
+	)
+	go converger.Loop(true) // main loop for converger, true to start paused
 
-	log.Printf("Start: %v\n", start)
-
-	for x := range G.GetVerticesChan() { // XXX ?
-		log.Printf("Main->Starting[%v]\n", x.Name)
-
-		wg.Add(1)
-		// must pass in value to avoid races...
-		// see: https://ttboj.wordpress.com/2015/07/27/golang-parallelism-issues-causing-too-many-open-files-error/
-		go func(v *Vertex) {
-			defer wg.Done()
-			v.Start()
-			log.Printf("Main->Finish[%v]\n", v.Name)
-		}(x)
-
-		// generate a startup "poke" so that an initial check happens
-		go func(v *Vertex) {
-			v.Events <- fmt.Sprintf("Startup(%v)", v.Name)
-		}(x)
+	// embedded etcd
+	if len(seeds) == 0 {
+		log.Printf("Main: Seeds: No seeds specified!")
+	} else {
+		log.Printf("Main: Seeds(%v): %v", len(seeds), seeds)
+	}
+	EmbdEtcd := NewEmbdEtcd(
+		hostname,
+		seeds,
+		clientURLs,
+		serverURLs,
+		c.Bool("no-server"),
+		idealClusterSize,
+		prefix,
+		converger,
+	)
+	if EmbdEtcd == nil {
+		// TODO: verify EmbdEtcd is not nil below...
+		log.Printf("Main: Etcd: Creation failed!")
+		exit <- true
+	} else if err := EmbdEtcd.Startup(); err != nil { // startup (returns when etcd main loop is running)
+		log.Printf("Main: Etcd: Startup failed: %v", err)
+		exit <- true
+	}
+	convergerStateFn := func(b bool) error {
+		// exit if we are using the converged-timeout and we are the
+		// root node. otherwise, if we are a child node in a remote
+		// execution hierarchy, we should only notify our converged
+		// state and wait for the parent to trigger the exit.
+		if depth == 0 && c.Int("converged-timeout") >= 0 {
+			if b {
+				log.Printf("Converged for %d seconds, exiting!", c.Int("converged-timeout"))
+				exit <- true // trigger an exit!
+			}
+			return nil
+		}
+		// send our individual state into etcd for others to see
+		return EtcdSetHostnameConverged(EmbdEtcd, hostname, b) // TODO: what should happen on error?
+	}
+	if EmbdEtcd != nil {
+		converger.SetStateFn(convergerStateFn)
 	}
 
-	log.Println("Running...")
+	exitchan := make(chan Event) // exit event
+	go func() {
+		startchan := make(chan struct{}) // start signal
+		go func() { startchan <- struct{}{} }()
+		file := c.String("file")
+		var configchan chan bool
+		var puppetchan <-chan time.Time
+		if !c.Bool("no-watch") && c.IsSet("file") {
+			configchan = ConfigWatch(file)
+		} else if c.IsSet("puppet") {
+			interval := PuppetInterval(c.String("puppet-conf"))
+			puppetchan = time.Tick(time.Duration(interval) * time.Second)
+		}
+		log.Println("Etcd: Starting...")
+		etcdchan := EtcdWatch(EmbdEtcd)
+		first := true // first loop or not
+		for {
+			log.Println("Main: Waiting...")
+			select {
+			case <-startchan: // kick the loop once at start
+				// pass
+
+			case b := <-etcdchan:
+				if !b { // ignore the message
+					continue
+				}
+				// everything else passes through to cause a compile!
+
+			case <-puppetchan:
+				// nothing, just go on
+
+			case msg := <-configchan:
+				if c.Bool("no-watch") || !msg {
+					continue // not ready to read config
+				}
+			// XXX: case compile_event: ...
+			// ...
+			case msg := <-exitchan:
+				msg.ACK()
+				return
+			}
+
+			var config *GraphConfig
+			if c.IsSet("file") {
+				config = ParseConfigFromFile(file)
+			} else if c.IsSet("puppet") {
+				config = ParseConfigFromPuppet(c.String("puppet"), c.String("puppet-conf"))
+			}
+			if config == nil {
+				log.Printf("Config: Parse failure")
+				continue
+			}
+
+			if config.Hostname != "" && config.Hostname != hostname {
+				log.Printf("Config: Hostname changed, ignoring config!")
+				continue
+			}
+			config.Hostname = hostname // set it in case it was ""
+
+			// run graph vertex LOCK...
+			if !first { // TODO: we can flatten this check out I think
+				converger.Pause() // FIXME: add sync wait?
+				G.Pause()         // sync
+			}
+
+			// build graph from yaml file on events (eg: from etcd)
+			// we need the vertices to be paused to work on them
+			if newFullgraph, err := fullGraph.NewGraphFromConfig(config, EmbdEtcd, noop); err == nil { // keep references to all original elements
+				fullGraph = newFullgraph
+			} else {
+				log.Printf("Config: Error making new graph from config: %v", err)
+				// unpause!
+				if !first {
+					G.Start(&wg, first) // sync
+					converger.Start()   // after G.Start()
+				}
+				continue
+			}
+
+			G = fullGraph.Copy() // copy to active graph
+			// XXX: do etcd transaction out here...
+			G.AutoEdges() // add autoedges; modifies the graph
+			G.AutoGroup() // run autogroup; modifies the graph
+			// TODO: do we want to do a transitive reduction?
+
+			log.Printf("Graph: %v", G) // show graph
+			err := G.ExecGraphviz(c.String("graphviz-filter"), c.String("graphviz"))
+			if err != nil {
+				log.Printf("Graphviz: %v", err)
+			} else {
+				log.Printf("Graphviz: Successfully generated graph!")
+			}
+			G.AssociateData(converger)
+			// G.Start(...) needs to be synchronous or wait,
+			// because if half of the nodes are started and
+			// some are not ready yet and the EtcdWatch
+			// loops, we'll cause G.Pause(...) before we
+			// even got going, thus causing nil pointer errors
+			G.Start(&wg, first) // sync
+			converger.Start()   // after G.Start()
+			first = false
+		}
+	}()
+
+	configWatcher := NewConfigWatcher()
+	events := configWatcher.Events()
+	if !c.Bool("no-watch") {
+		configWatcher.Add(c.StringSlice("remote")...) // add all the files...
+	} else {
+		events = nil // signal that no-watch is true
+	}
+
+	// initialize the add watcher, which calls the f callback on map changes
+	convergerCb := func(f func(map[string]bool) error) (func(), error) {
+		return EtcdAddHostnameConvergedWatcher(EmbdEtcd, f)
+	}
+
+	// build remotes struct for remote ssh
+	remotes := NewRemotes(
+		EmbdEtcd.LocalhostClientURLs().StringSlice(),
+		[]string{DefaultClientURL},
+		noop,
+		c.StringSlice("remote"), // list of files
+		events,                  // watch for file changes
+		cConns,
+		c.Bool("allow-interactive"),
+		c.String("ssh-priv-id-rsa"),
+		!c.Bool("no-caching"),
+		depth,
+		prefix,
+		converger,
+		convergerCb,
+	)
+
+	// TODO: is there any benefit to running the remotes above in the loop?
+	// wait for etcd to be running before we remote in, which we do above!
+	go remotes.Run()
+
+	if !c.IsSet("file") && !c.IsSet("puppet") {
+		converger.Start() // better start this for empty graphs
+	}
+	log.Println("Main: Running...")
 
 	waitForSignal(exit) // pass in exit channel to watch
 
+	log.Println("Destroy...")
+
+	configWatcher.Close() // stop sending file changes to remotes
+	remotes.Exit()        // tell all the remote connections to shutdown; waits!
+
 	G.Exit() // tell all the children to exit
 
+	// tell inner main loop to exit
+	resp := NewResp()
+	go func() { exitchan <- Event{eventExit, resp, "", false} }()
+
+	// cleanup etcd main loop last so it can process everything first
+	if err := EmbdEtcd.Destroy(); err != nil { // shutdown and cleanup etcd
+		log.Printf("Etcd exited poorly with: %v", err)
+	}
+
+	resp.ACKWait() // let inner main loop finish cleanly just in case
+
 	if DEBUG {
-		for i := range G.GetVerticesChan() {
-			fmt.Printf("Vertex: %v\n", i)
-		}
-		fmt.Printf("Graph: %v\n", G)
+		log.Printf("Graph: %v", G)
 	}
 
 	wg.Wait() // wait for primary go routines to exit
 
 	// TODO: wait for each vertex to exit...
 	log.Println("Goodbye!")
+	return nil
 }
 
 func main() {
+	var flags int
+	if DEBUG || true { // TODO: remove || true
+		flags = log.LstdFlags | log.Lshortfile
+	}
+	flags = (flags - log.Ldate) // remove the date for now
+	log.SetFlags(flags)
+
+	// un-hijack from capnslog...
+	log.SetOutput(os.Stderr)
+	if VERBOSE {
+		capnslog.SetFormatter(capnslog.NewLogFormatter(os.Stderr, "(etcd) ", flags))
+	} else {
+		capnslog.SetFormatter(capnslog.NewNilFormatter())
+	}
+
+	// test for sanity
+	if program == "" || version == "" {
+		log.Fatal("Program was not compiled correctly. Please see Makefile.")
+	}
 	app := cli.NewApp()
 	app.Name = program
 	app.Usage = "next generation config management"
@@ -130,14 +439,136 @@ func main() {
 			Action:  run,
 			Flags: []cli.Flag{
 				cli.StringFlag{
-					Name:  "file, f",
+					Name:   "file, f",
+					Value:  "",
+					Usage:  "graph definition to run",
+					EnvVar: "MGMT_FILE",
+				},
+				cli.BoolFlag{
+					Name:  "no-watch",
+					Usage: "do not update graph on watched graph definition file changes",
+				},
+				cli.StringFlag{
+					Name:  "code, c",
 					Value: "",
-					Usage: "graph definition to run",
+					Usage: "code definition to run",
+				},
+				cli.StringFlag{
+					Name:  "graphviz, g",
+					Value: "",
+					Usage: "output file for graphviz data",
+				},
+				cli.StringFlag{
+					Name:  "graphviz-filter, gf",
+					Value: "dot", // directed graph default
+					Usage: "graphviz filter to use",
+				},
+				// useful for testing multiple instances on same machine
+				cli.StringFlag{
+					Name:  "hostname",
+					Value: "",
+					Usage: "hostname to use",
+				},
+				// if empty, it will startup a new server
+				cli.StringSliceFlag{
+					Name:   "seeds, s",
+					Value:  &cli.StringSlice{}, // empty slice
+					Usage:  "default etc client endpoint",
+					EnvVar: "MGMT_SEEDS",
+				},
+				// port 2379 and 4001 are common
+				cli.StringSliceFlag{
+					Name:   "client-urls",
+					Value:  &cli.StringSlice{},
+					Usage:  "list of URLs to listen on for client traffic",
+					EnvVar: "MGMT_CLIENT_URLS",
+				},
+				// port 2380 and 7001 are common
+				cli.StringSliceFlag{
+					Name:   "server-urls, peer-urls",
+					Value:  &cli.StringSlice{},
+					Usage:  "list of URLs to listen on for server (peer) traffic",
+					EnvVar: "MGMT_SERVER_URLS",
+				},
+				cli.BoolFlag{
+					Name:  "no-server",
+					Usage: "do not let other servers peer with me",
 				},
 				cli.IntFlag{
-					Name:  "exittime",
-					Value: 0,
-					Usage: "exit after a maximum of approximately this many seconds",
+					Name:   "ideal-cluster-size",
+					Value:  defaultIdealClusterSize,
+					Usage:  "ideal number of server peers in cluster, only read by initial server",
+					EnvVar: "MGMT_IDEAL_CLUSTER_SIZE",
+				},
+				cli.IntFlag{
+					Name:   "converged-timeout, t",
+					Value:  -1,
+					Usage:  "exit after approximately this many seconds in a converged state",
+					EnvVar: "MGMT_CONVERGED_TIMEOUT",
+				},
+				cli.IntFlag{
+					Name:   "max-runtime",
+					Value:  0,
+					Usage:  "exit after a maximum of approximately this many seconds",
+					EnvVar: "MGMT_MAX_RUNTIME",
+				},
+				cli.BoolFlag{
+					Name:  "noop",
+					Usage: "globally force all resources into no-op mode",
+				},
+				cli.StringFlag{
+					Name:  "puppet, p",
+					Value: "",
+					Usage: "load graph from puppet, optionally takes a manifest or path to manifest file",
+				},
+				cli.StringFlag{
+					Name:  "puppet-conf",
+					Value: "",
+					Usage: "supply the path to an alternate puppet.conf file to use",
+				},
+				cli.StringSliceFlag{
+					Name:  "remote",
+					Value: &cli.StringSlice{},
+					Usage: "list of remote graph definitions to run",
+				},
+				cli.BoolFlag{
+					Name:  "allow-interactive",
+					Usage: "allow interactive prompting, such as for remote passwords",
+				},
+				cli.StringFlag{
+					Name:   "ssh-priv-id-rsa",
+					Value:  "~/.ssh/id_rsa",
+					Usage:  "default path to ssh key file, set empty to never touch",
+					EnvVar: "MGMT_SSH_PRIV_ID_RSA",
+				},
+				cli.IntFlag{
+					Name:   "cconns",
+					Value:  0,
+					Usage:  "number of maximum concurrent remote ssh connections to run, 0 for unlimited",
+					EnvVar: "MGMT_CCONNS",
+				},
+				cli.BoolFlag{
+					Name:  "no-caching",
+					Usage: "don't allow remote caching of remote execution binary",
+				},
+				cli.IntFlag{
+					Name:   "depth",
+					Hidden: true, // internal use only
+					Value:  0,
+					Usage:  "specify depth in remote hierarchy",
+				},
+				cli.StringFlag{
+					Name:   "prefix",
+					Usage:  "specify a path to the working prefix directory",
+					EnvVar: "MGMT_PREFIX",
+				},
+				cli.BoolFlag{
+					Name:  "tmp-prefix",
+					Usage: "request a pseudo-random, temporary prefix to be used",
+				},
+				cli.BoolFlag{
+					Name:  "allow-tmp-prefix",
+					Usage: "allow creation of a new temporary prefix if main prefix is unavailable",
 				},
 			},
 		},

main_test.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify

misc.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify
@@ -18,22 +18,185 @@
 package main
 
 import (
+	"github.com/godbus/dbus"
 	"path"
+	"sort"
 	"strings"
+	"time"
 )
 
-// Similar to the GNU dirname command
+// FirstToUpper returns the string with the first character capitalized.
+func FirstToUpper(str string) string {
+	if str == "" {
+		return ""
+	}
+	return strings.ToUpper(str[0:1]) + str[1:]
+}
+
+// StrInList returns true if a string exists inside a list, otherwise false.
+func StrInList(needle string, haystack []string) bool {
+	for _, x := range haystack {
+		if needle == x {
+			return true
+		}
+	}
+	return false
+}
+
+// Uint64KeyFromStrInMap returns true if needle is found in haystack of keys
+// that have uint64 type.
+func Uint64KeyFromStrInMap(needle string, haystack map[uint64]string) (uint64, bool) {
+	for k, v := range haystack {
+		if v == needle {
+			return k, true
+		}
+	}
+	return 0, false
+}
+
+// StrRemoveDuplicatesInList removes any duplicate values in the list.
+// This is a possibly sub-optimal, O(n^2)? implementation.
+func StrRemoveDuplicatesInList(list []string) []string {
+	unique := []string{}
+	for _, x := range list {
+		if !StrInList(x, unique) {
+			unique = append(unique, x)
+		}
+	}
+	return unique
+}
+
+// StrFilterElementsInList removes any of the elements in filter, if they exist
+// in the list.
+func StrFilterElementsInList(filter []string, list []string) []string {
+	result := []string{}
+	for _, x := range list {
+		if !StrInList(x, filter) {
+			result = append(result, x)
+		}
+	}
+	return result
+}
+
+// StrListIntersection removes any of the elements in filter, if they don't
+// exist in the list. This is an in order intersection of two lists.
+func StrListIntersection(list1 []string, list2 []string) []string {
+	result := []string{}
+	for _, x := range list1 {
+		if StrInList(x, list2) {
+			result = append(result, x)
+		}
+	}
+	return result
+}
+
+// ReverseStringList reverses a list of strings.
+func ReverseStringList(in []string) []string {
+	var out []string // empty list
+	l := len(in)
+	for i := range in {
+		out = append(out, in[l-i-1])
+	}
+	return out
+}
+
+// StrMapKeys return the sorted list of string keys in a map with string keys.
+// NOTE: i thought it would be nice for this to use: map[string]interface{} but
+// it turns out that's not allowed. I know we don't have generics, but come on!
+func StrMapKeys(m map[string]string) []string {
+	result := []string{}
+	for k := range m {
+		result = append(result, k)
+	}
+	sort.Strings(result) // deterministic order
+	return result
+}
+
+// StrMapKeysUint64 return the sorted list of string keys in a map with string
+// keys but uint64 values.
+func StrMapKeysUint64(m map[string]uint64) []string {
+	result := []string{}
+	for k := range m {
+		result = append(result, k)
+	}
+	sort.Strings(result) // deterministic order
+	return result
+}
+
+// BoolMapValues returns the sorted list of bool values in a map with string
+// values.
+func BoolMapValues(m map[string]bool) []bool {
+	result := []bool{}
+	for _, v := range m {
+		result = append(result, v)
+	}
+	//sort.Bools(result) // TODO: deterministic order
+	return result
+}
+
+// StrMapValues returns the sorted list of string values in a map with string
+// values.
+func StrMapValues(m map[string]string) []string {
+	result := []string{}
+	for _, v := range m {
+		result = append(result, v)
+	}
+	sort.Strings(result) // deterministic order
+	return result
+}
+
+// StrMapValuesUint64 return the sorted list of string values in a map with
+// string values.
+func StrMapValuesUint64(m map[uint64]string) []string {
+	result := []string{}
+	for _, v := range m {
+		result = append(result, v)
+	}
+	sort.Strings(result) // deterministic order
+	return result
+}
+
+// BoolMapTrue returns true if everyone in the list is true.
+func BoolMapTrue(l []bool) bool {
+	for _, b := range l {
+		if !b {
+			return false
+		}
+	}
+	return true
+}
+
+// Dirname is similar to the GNU dirname command.
 func Dirname(p string) string {
+	if p == "/" {
+		return ""
+	}
 	d, _ := path.Split(path.Clean(p))
 	return d
 }
 
-// Split a path into an array of tokens excluding any trailing empty tokens
+// Basename is the base of a path string.
+func Basename(p string) string {
+	_, b := path.Split(path.Clean(p))
+	if p == "" {
+		return ""
+	}
+	if p[len(p)-1:] == "/" { // don't loose the tail slash
+		b += "/"
+	}
+	return b
+}
+
+// PathSplit splits a path into an array of tokens excluding any trailing empty
+// tokens.
 func PathSplit(p string) []string {
+	if p == "/" { // TODO: can't this all be expressed nicely in one line?
+		return []string{""}
+	}
 	return strings.Split(path.Clean(p), "/")
 }
 
-// Does path string contain the given path prefix in it?
+// HasPathPrefix tells us if a path string contain the given path prefix in it.
 func HasPathPrefix(p, prefix string) bool {
 
 	patharray := PathSplit(p)
@@ -52,6 +215,155 @@ func HasPathPrefix(p, prefix string) bool {
 	return true
 }
 
+// StrInPathPrefixList returns true if the needle is a PathPrefix in the
+// haystack.
+func StrInPathPrefixList(needle string, haystack []string) bool {
+	for _, x := range haystack {
+		if HasPathPrefix(x, needle) {
+			return true
+		}
+	}
+	return false
+}
+
+// RemoveCommonFilePrefixes removes redundant file path prefixes that are under
+// the tree of other files.
+func RemoveCommonFilePrefixes(paths []string) []string {
+	var result = make([]string, len(paths))
+	for i := 0; i < len(paths); i++ { // copy, b/c append can modify the args!!
+		result[i] = paths[i]
+	}
+	// is there a string path which is common everywhere?
+	// if so, remove it, and iterate until nothing common is left
+	// return what's left over, that's the most common superset
+loop:
+	for {
+		if len(result) <= 1 {
+			return result
+		}
+		for i := 0; i < len(result); i++ {
+			var copied = make([]string, len(result))
+			for j := 0; j < len(result); j++ { // copy, b/c append can modify the args!!
+				copied[j] = result[j]
+			}
+			noi := append(copied[:i], copied[i+1:]...) // rm i
+			if StrInPathPrefixList(result[i], noi) {
+				// delete the element common to everyone
+				result = noi
+				continue loop
+			}
+		}
+		break
+	}
+	return result
+}
+
+// PathPrefixDelta returns the delta of the path prefix, which tells you how
+// many path tokens different the prefix is.
+func PathPrefixDelta(p, prefix string) int {
+
+	if !HasPathPrefix(p, prefix) {
+		return -1
+	}
+	patharray := PathSplit(p)
+	prefixarray := PathSplit(prefix)
+	return len(patharray) - len(prefixarray)
+}
+
+// PathIsDir returns true if there is a trailing slash.
 func PathIsDir(p string) bool {
 	return p[len(p)-1:] == "/" // a dir has a trailing slash in this context
 }
+
+// PathSplitFullReversed returns the full list of "dependency" paths for a given
+// path in reverse order.
+func PathSplitFullReversed(p string) []string {
+	var result []string
+	split := PathSplit(p)
+	count := len(split)
+	var x string
+	for i := 0; i < count; i++ {
+		x = "/" + path.Join(split[0:i+1]...)
+		if i != 0 && !(i+1 == count && !PathIsDir(p)) {
+			x += "/" // add trailing slash
+		}
+		result = append(result, x)
+	}
+	return ReverseStringList(result)
+}
+
+// DirifyFileList adds trailing slashes to any likely dirs in a package manager
+// fileList if removeDirs is true, otherwise, don't keep the dirs in our output.
+func DirifyFileList(fileList []string, removeDirs bool) []string {
+	dirs := []string{}
+	for _, file := range fileList {
+		dir, _ := path.Split(file) // dir
+		dir = path.Clean(dir)      // clean so cmp is easier
+		if !StrInList(dir, dirs) {
+			dirs = append(dirs, dir)
+		}
+	}
+
+	result := []string{}
+	for _, file := range fileList {
+		cleanFile := path.Clean(file)
+		if !StrInList(cleanFile, dirs) { // we're not a directory!
+			result = append(result, file) // pass through
+		} else if !removeDirs {
+			result = append(result, cleanFile+"/")
+		}
+	}
+
+	return result
+}
+
+// FlattenListWithSplit flattens a list of input by splitting each element by
+// any and all of the strings listed in the split array
+func FlattenListWithSplit(input []string, split []string) []string {
+	if len(split) == 0 { // nothing to split by
+		return input
+	}
+	out := []string{}
+	for _, x := range input {
+		s := []string{}
+		if len(split) == 1 {
+			s = strings.Split(x, split[0]) // split by only string
+		} else {
+			s = []string{x} // initial
+			for i := range split {
+				s = FlattenListWithSplit(s, []string{split[i]}) // recurse
+			}
+		}
+		out = append(out, s...)
+	}
+	return out
+}
+
+// TimeAfterOrBlock is aspecial version of time.After that blocks when given a
+// negative integer. When used in a case statement, the timer restarts on each
+// select call to it.
+func TimeAfterOrBlock(t int) <-chan time.Time {
+	if t < 0 {
+		return make(chan time.Time) // blocks forever
+	}
+	return time.After(time.Duration(t) * time.Second)
+}
+
+// SystemBusPrivateUsable makes using the private bus usable
+// TODO: should be upstream: https://github.com/godbus/dbus/issues/15
+func SystemBusPrivateUsable() (conn *dbus.Conn, err error) {
+	conn, err = dbus.SystemBusPrivate()
+	if err != nil {
+		return nil, err
+	}
+	if err = conn.Auth(nil); err != nil {
+		conn.Close()
+		conn = nil
+		return
+	}
+	if err = conn.Hello(); err != nil {
+		conn.Close()
+		conn = nil
+	}
+	return conn, nil // success
+}

misc/bashrc.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+_cli_bash_autocomplete_mgmt() {
+	local cur prev opts base
+	COMPREPLY=()
+	cur="${COMP_WORDS[COMP_CWORD]}"
+	prev="${COMP_WORDS[COMP_CWORD-1]}"
+	opts=$( ${COMP_WORDS[@]:0:$COMP_CWORD} --generate-bash-completion )
+	COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
+	return 0
+}
+
+complete -F _cli_bash_autocomplete_mgmt mgmt

misc/centos-ci.py

@@ -0,0 +1,72 @@
+#!/usr/bin/python
+
+# modified from:
+# https://github.com/kbsingh/centos-ci-scripts/blob/master/build_python_script.py
+# usage: centos-ci.py giturl [branch [commands]]
+
+import os
+import sys
+import json
+import urllib
+import subprocess
+
+# static argv to be used if running script inline
+argv = [
+	#'https://github.com/purpleidea/mgmt', # giturl
+	#'master',
+	#'make test',
+]
+argv.insert(0, '') # add a fake argv[0]
+url_base = 'http://admin.ci.centos.org:8080'
+apikey = '' # put api key here if running inline
+if apikey == '':
+	apikey = os.environ.get('DUFFY_API_KEY')
+if apikey is None or apikey == '':
+	apikey = open('duffy.key', 'r').read().strip()
+ver = '7'
+arch = 'x86_64'
+count = 1
+
+if len(argv) <= 1: argv = sys.argv # use system argv because ours is empty
+if len(argv) <= 1:
+	print 'Not enough arguments supplied!'
+	sys.exit(1)
+
+git_url = argv[1]
+branch = 'master'
+if len(argv) > 2: branch = argv[2]
+folder = os.path.basename(git_url)	# should be project name
+run = 'make vtest' # the omv vtest cmd is a good option to run from this target
+if len(argv) > 3: run = ' '.join(argv[3:])
+
+get_nodes_url = "%s/Node/get?key=%s&ver=%s&arch=%s&i_count=%s" % (url_base, apikey, ver, arch, count)
+data = json.loads(urllib.urlopen(get_nodes_url).read()) # request host(s)
+hosts = data['hosts']
+ssid = data['ssid']
+done_nodes_url = "%s/Node/done?key=%s&ssid=%s" % (url_base, apikey, ssid)
+
+host = hosts[0]
+ssh = "ssh -tt -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o SendEnv=JENKINS_URL root@%s" % host
+yum = 'yum -y install git wget tree psmisc'
+omv = 'wget https://github.com/purpleidea/oh-my-vagrant/raw/master/extras/install-omv.sh && chmod u+x install-omv.sh && ./install-omv.sh && wget https://github.com/purpleidea/mgmt/raw/master/misc/make-path.sh && chmod u+x make-path.sh && ./make-path.sh'
+cmd = "%s '%s && %s'" % (ssh, yum, omv) # setup
+print cmd
+r = subprocess.call(cmd, shell=True)
+if r != 0:
+	# NOTE: we don't clean up the host here, so that it can be inspected!
+	print "Error configuring omv on: %s" % host
+	sys.exit(r)
+
+# the second ssh call will run with the omv /etc/profile.d/ script loaded
+git = "git clone --recursive %s %s && cd %s && git checkout %s" % (git_url, folder, folder, branch)
+cmd = "%s 'export JENKINS_URL=%s && %s && %s'" % (ssh, os.getenv('JENKINS_URL', ''), git, run) # run
+print cmd
+r = subprocess.call(cmd, shell=True)
+if r != 0:
+	print "Error running job on: %s" % host
+
+output = urllib.urlopen(done_nodes_url).read() # free host(s)
+if output != 'Done':
+	print "Error freeing host: %s" % host
+
+sys.exit(r)

misc/copr-build.py

@@ -0,0 +1,66 @@
+#!/usr/bin/python
+
+# README:
+# for initial setup, browse to: https://copr.fedoraproject.org/api/
+# and it will have a ~/.config/copr config that you can download.
+# happy hacking!
+
+import os
+import sys
+import copr
+import time
+
+COPR = 'mgmt'
+
+if len(sys.argv) != 2:
+	print("Usage: %s <srpm url>" % sys.argv[0])
+	sys.exit(1)
+
+url = sys.argv[1]
+
+client = copr.CoprClient.create_from_file_config(os.path.expanduser("~/.config/copr"))
+
+result = client.create_new_build(COPR, [url])
+if result.output != 'ok':
+	print(result.error)
+	sys.exit(1)
+print(result.message)
+
+# modified from: https://python-copr.readthedocs.org/en/latest/Examples.html#work-with-builds
+for bw in result.builds_list:
+	print("Build #{}: {}".format(bw.build_id, bw.handle.get_build_details().status))
+
+# cancel all created build
+#for bw in result.builds_list:
+#	bw.handle.cancel_build()
+
+# get build status for each chroot
+#for bw in result.builds_list:
+#	print("build: {}".format(bw.build_id))
+#	for ch, status in bw.handle.get_build_details().data["chroots"].items():
+#		print("\t chroot {}:\t {}".format(ch, status))
+
+# simple build progress:
+
+watched = set(result.builds_list)
+done = set()
+state = {}
+for bw in watched:	# store initial states
+	state[bw.build_id] = bw.handle.get_build_details().status
+
+while watched != done:
+	for bw in watched:
+		if bw in done:
+			continue
+		status = bw.handle.get_build_details().status
+		if status != state.get(bw.build_id):
+			print("Build #{}: {}".format(bw.build_id, status))
+			state[bw.build_id] = status	# update status
+
+		if status in ['skipped', 'failed', 'succeeded']:
+			done.add(bw)
+
+	if watched == done: break	# avoid long while sleep
+	else: time.sleep(10)
+
+print 'Done!'

misc/example.conf

@@ -0,0 +1 @@
+# example mgmt configuration file, currently has no options at the moment!

misc/go

@@ -0,0 +1,15 @@
+#!/bin/bash
+# hack around stupid $GOPATH semantics, with ~/bin/go helper
+# thanks to Nilium in #go-nuts for 1/3 of the idea
+[ -z "$GOPATH" ] && echo '$GOPATH is not set!' && exit 1
+GO="$(which -a go | sed -e '2q;d')"	# TODO: pick /usr/bin/go in a better way
+if [ "$1" = "generate" ]; then
+	exec $GO "$@"	# go generate is stupid and gets confused by $GOPATH
+fi
+# the idea is to have $project/gopath/src/ be a symlink to ../vendor but you put
+# all of your vendored things in vendor/ but with this gopath can be per project
+if [ -d "$PWD/vendor/" ] && [ -d "$PWD/gopath/" ] && [ "`readlink $PWD/gopath/src`" = "../vendor" ] ; then
+	GOPATH="$PWD/gopath/:$GOPATH" $GO "$@"
+else
+	$GO "$@"
+fi

misc/make-deps.sh

@@ -0,0 +1,51 @@
+#!/bin/bash
+# setup a simple go environment
+XPWD=`pwd`
+ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && cd .. && pwd )"	# dir!
+cd "${ROOT}" >/dev/null
+
+travis=0
+if env | grep -q '^TRAVIS=true$'; then
+	travis=1
+fi
+
+sudo_command=$(which sudo)
+
+if [ $travis -eq 0 ]; then
+	YUM=`which yum 2>/dev/null`
+	APT=`which apt-get 2>/dev/null`
+	if [ -z "$YUM" -a -z "$APT" ]; then
+		echo "The package managers can't be found."
+		exit 1
+	fi
+	if [ ! -z "$YUM" ]; then
+		# some go dependencies are stored in mercurial
+		$sudo_command $YUM install -y golang golang-googlecode-tools-stringer hg
+
+	fi
+	if [ ! -z "$APT" ]; then
+		$sudo_command $APT update
+		$sudo_command $APT install -y golang make gcc packagekit mercurial
+		# one of these two golang tools packages should work on debian
+		$sudo_command $APT install -y golang-golang-x-tools || true
+		$sudo_command $APT install -y golang-go.tools || true
+	fi
+fi
+
+# if golang is too old, we don't want to fail with an obscure error later
+if go version | grep 'go1\.[0123]\.'; then
+	echo "mgmt requires go1.4 or higher."
+	exit 1
+fi
+
+go get ./...	# get all the go dependencies
+[ -e "$GOBIN/mgmt" ] && rm -f "$GOBIN/mgmt"	# the `go get` version has no -X
+# vet is built-in in go 1.6 - we check for go vet command
+go vet 1> /dev/null 2>&1
+ret=$?
+if [[ $ret != 0 ]]; then
+	go get golang.org/x/tools/cmd/vet      # add in `go vet` for travis
+fi
+go get golang.org/x/tools/cmd/stringer	# for automatic stringer-ing
+go get github.com/golang/lint/golint	# for `golint`-ing
+cd "$XPWD" >/dev/null

misc/make-path.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+# setup a few environment path values
+
+if ! env | grep -q '^GOPATH='; then
+	export GOPATH="$HOME/gopath/"
+	mkdir "$GOPATH"
+	if ! grep -q '^export GOPATH=' ~/.bashrc; then
+	        echo "export GOPATH=~/gopath/" >> ~/.bashrc
+	fi
+	echo "setting go path to: $GOPATH"
+fi
+
+echo "gopath is: $GOPATH"
+
+# some versions of golang apparently require this to run go get :(
+if ! env | grep -q '^GOBIN='; then
+	export GOBIN="${GOPATH}bin/"
+	mkdir "$GOBIN"
+	if ! grep -q '^export GOBIN=' ~/.bashrc; then
+	        echo 'export GOBIN="${GOPATH}bin/"' >> ~/.bashrc
+	fi
+	echo "setting go bin to: $GOBIN"
+fi
+
+echo "gobin is: $GOBIN"
+
+# add gobin to $PATH
+if ! env | grep '^PATH=' | grep -q "$GOBIN"; then
+	if ! grep -q '^export PATH="'"${GOBIN}"':${PATH}"' ~/.bashrc; then
+	        echo 'export PATH="'"${GOBIN}"':${PATH}"' >> ~/.bashrc
+	fi
+	export PATH="${GOBIN}:${PATH}"
+	echo "setting path to: $PATH"
+fi
+
+echo "path is: $PATH"
+
+# add ~/bin/ to $PATH
+if ! env | grep '^PATH=' | grep -q "$HOME/bin"; then
+	mkdir -p "${HOME}/bin"
+	if ! grep -q '^export PATH="'"${HOME}/bin"':${PATH}"' ~/.bashrc; then
+	        echo 'export PATH="'"${HOME}/bin"':${PATH}"' >> ~/.bashrc
+	fi
+	export PATH="${HOME}/bin:${PATH}"
+	echo "setting path to: $PATH"
+fi
+
+echo "path is: $PATH"

misc/mgmt.service

@@ -0,0 +1,13 @@
+[Unit]
+Description=Run mgmt configuration management
+Documentation=https://github.com/purpleidea/mgmt/
+After=systemd-networkd.service
+Requires=systemd-networkd.service
+
+[Service]
+ExecStart=/usr/bin/mgmt run ${OPTS}
+RestartSec=5s
+Restart=always
+
+[Install]
+WantedBy=multi-user.target

misc_test.go

@@ -1,5 +1,5 @@
 // Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
 // Written by James Shubin <james@shubin.ca> and the project contributors
 //
 // This program is free software: you can redistribute it and/or modify
@@ -18,6 +18,8 @@
 package main
 
 import (
+	"reflect"
+	"sort"
 	"testing"
 )
 
@@ -31,7 +33,31 @@ func TestMiscT1(t *testing.T) {
 		t.Errorf("Result is incorrect.")
 	}
 
-	if Dirname("/") != "/" {
+	if Dirname("/foo/") != "/" {
+		t.Errorf("Result is incorrect.")
+	}
+
+	if Dirname("/") != "" { // TODO: should this equal "/" or "" ?
+		t.Errorf("Result is incorrect.")
+	}
+
+	if Basename("/foo/bar/baz") != "baz" {
+		t.Errorf("Result is incorrect.")
+	}
+
+	if Basename("/foo/bar/baz/") != "baz/" {
+		t.Errorf("Result is incorrect.")
+	}
+
+	if Basename("/foo/") != "foo/" {
+		t.Errorf("Result is incorrect.")
+	}
+
+	if Basename("/") != "/" { // TODO: should this equal "" or "/" ?
+		t.Errorf("Result is incorrect.")
+	}
+
+	if Basename("") != "" { // TODO: should this equal something different?
 		t.Errorf("Result is incorrect.")
 	}
 }
@@ -39,6 +65,13 @@ func TestMiscT1(t *testing.T) {
 func TestMiscT2(t *testing.T) {
 
 	// TODO: compare the output with the actual list
+	p0 := "/"
+	r0 := []string{""} // TODO: is this correct?
+	if len(PathSplit(p0)) != len(r0) {
+		t.Errorf("Result should be: %q.", r0)
+		t.Errorf("Result should have a length of: %v.", len(r0))
+	}
+
 	p1 := "/foo/bar/baz"
 	r1 := []string{"", "foo", "bar", "baz"}
 	if len(PathSplit(p1)) != len(r1) {
@@ -78,10 +111,49 @@ func TestMiscT3(t *testing.T) {
 	if HasPathPrefix("/foo/bar/baz/", "/foo/bar/baz/dude") != false {
 		t.Errorf("Result should be false.")
 	}
+
+	if HasPathPrefix("/foo/bar/baz/boo/", "/foo/") != true {
+		t.Errorf("Result should be true.")
+	}
 }
 
 func TestMiscT4(t *testing.T) {
 
+	if PathPrefixDelta("/foo/bar/baz", "/foo/ba") != -1 {
+		t.Errorf("Result should be -1.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz", "/foo/bar") != 1 {
+		t.Errorf("Result should be 1.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz", "/foo/bar/") != 1 {
+		t.Errorf("Result should be 1.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz/", "/foo/bar") != 1 {
+		t.Errorf("Result should be 1.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz/", "/foo/bar/") != 1 {
+		t.Errorf("Result should be 1.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz/", "/foo/bar/baz/dude") != -1 {
+		t.Errorf("Result should be -1.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz/a/b/c/", "/foo/bar/baz") != 3 {
+		t.Errorf("Result should be 3.")
+	}
+
+	if PathPrefixDelta("/foo/bar/baz/", "/foo/bar/baz") != 0 {
+		t.Errorf("Result should be 0.")
+	}
+}
+
+func TestMiscT5(t *testing.T) {
+
 	if PathIsDir("/foo/bar/baz/") != true {
 		t.Errorf("Result should be false.")
 	}
@@ -97,5 +169,644 @@ func TestMiscT4(t *testing.T) {
 	if PathIsDir("/") != true {
 		t.Errorf("Result should be true.")
 	}
+}
+
+func TestMiscT8(t *testing.T) {
+
+	r0 := []string{"/"}
+	if fullList0 := PathSplitFullReversed("/"); !reflect.DeepEqual(r0, fullList0) {
+		t.Errorf("PathSplitFullReversed expected: %v; got: %v.", r0, fullList0)
+	}
+
+	r1 := []string{"/foo/bar/baz/file", "/foo/bar/baz/", "/foo/bar/", "/foo/", "/"}
+	if fullList1 := PathSplitFullReversed("/foo/bar/baz/file"); !reflect.DeepEqual(r1, fullList1) {
+		t.Errorf("PathSplitFullReversed expected: %v; got: %v.", r1, fullList1)
+	}
+
+	r2 := []string{"/foo/bar/baz/dir/", "/foo/bar/baz/", "/foo/bar/", "/foo/", "/"}
+	if fullList2 := PathSplitFullReversed("/foo/bar/baz/dir/"); !reflect.DeepEqual(r2, fullList2) {
+		t.Errorf("PathSplitFullReversed expected: %v; got: %v.", r2, fullList2)
+	}
 
 }
+
+func TestMiscT9(t *testing.T) {
+	fileListIn := []string{ // list taken from drbd-utils package
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d",
+		"/usr/share/doc/drbd-utils",
+		"/var/lib/drbd",
+	}
+	sort.Strings(fileListIn)
+
+	fileListOut := []string{ // fixed up manually
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d/",               // added trailing slash
+		"/usr/share/doc/drbd-utils/", // added trailing slash
+		"/var/lib/drbd",              // can't be fixed :(
+	}
+	sort.Strings(fileListOut)
+
+	dirify := DirifyFileList(fileListIn, false) // TODO: test with true
+	sort.Strings(dirify)
+	equals := reflect.DeepEqual(fileListOut, dirify)
+	if a, b := len(fileListOut), len(dirify); a != b {
+		t.Errorf("DirifyFileList counts didn't match: %d != %d", a, b)
+	} else if !equals {
+		t.Error("DirifyFileList did not match expected!")
+		for i := 0; i < len(dirify); i++ {
+			if fileListOut[i] != dirify[i] {
+				t.Errorf("# %d: %v <> %v", i, fileListOut[i], dirify[i])
+			}
+		}
+	}
+}
+
+func TestMiscT10(t *testing.T) {
+	fileListIn := []string{ // fake package list
+		"/etc/drbd.conf",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d",
+		"/etc/drbd.d/foo",
+		"/var/lib/drbd",
+		"/var/somedir/",
+	}
+	sort.Strings(fileListIn)
+
+	fileListOut := []string{ // fixed up manually
+		"/etc/drbd.conf",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d/", // added trailing slash
+		"/etc/drbd.d/foo",
+		"/var/lib/drbd", // can't be fixed :(
+		"/var/somedir/", // stays the same
+	}
+	sort.Strings(fileListOut)
+
+	dirify := DirifyFileList(fileListIn, false) // TODO: test with true
+	sort.Strings(dirify)
+	equals := reflect.DeepEqual(fileListOut, dirify)
+	if a, b := len(fileListOut), len(dirify); a != b {
+		t.Errorf("DirifyFileList counts didn't match: %d != %d", a, b)
+	} else if !equals {
+		t.Error("DirifyFileList did not match expected!")
+		for i := 0; i < len(dirify); i++ {
+			if fileListOut[i] != dirify[i] {
+				t.Errorf("# %d: %v <> %v", i, fileListOut[i], dirify[i])
+			}
+		}
+	}
+}
+
+func TestMiscT11(t *testing.T) {
+	in1 := []string{"/", "/usr/", "/usr/lib/", "/usr/share/"} // input
+	ex1 := []string{"/usr/lib/", "/usr/share/"}               // expected
+	sort.Strings(ex1)
+	out1 := RemoveCommonFilePrefixes(in1)
+	sort.Strings(out1)
+	if !reflect.DeepEqual(ex1, out1) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex1, out1)
+	}
+
+	in2 := []string{"/", "/usr/"}
+	ex2 := []string{"/usr/"}
+	sort.Strings(ex2)
+	out2 := RemoveCommonFilePrefixes(in2)
+	sort.Strings(out2)
+	if !reflect.DeepEqual(ex2, out2) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex2, out2)
+	}
+
+	in3 := []string{"/"}
+	ex3 := []string{"/"}
+	out3 := RemoveCommonFilePrefixes(in3)
+	if !reflect.DeepEqual(ex3, out3) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex3, out3)
+	}
+
+	in4 := []string{"/usr/bin/foo", "/usr/bin/bar", "/usr/lib/", "/usr/share/"}
+	ex4 := []string{"/usr/bin/foo", "/usr/bin/bar", "/usr/lib/", "/usr/share/"}
+	sort.Strings(ex4)
+	out4 := RemoveCommonFilePrefixes(in4)
+	sort.Strings(out4)
+	if !reflect.DeepEqual(ex4, out4) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex4, out4)
+	}
+
+	in5 := []string{"/usr/bin/foo", "/usr/bin/bar", "/usr/lib/", "/usr/share/", "/usr/bin"}
+	ex5 := []string{"/usr/bin/foo", "/usr/bin/bar", "/usr/lib/", "/usr/share/"}
+	sort.Strings(ex5)
+	out5 := RemoveCommonFilePrefixes(in5)
+	sort.Strings(out5)
+	if !reflect.DeepEqual(ex5, out5) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex5, out5)
+	}
+
+	in6 := []string{"/etc/drbd.d/", "/lib/drbd/", "/usr/lib/drbd/", "/usr/lib/systemd/system/", "/usr/lib/tmpfiles.d/", "/usr/sbin/", "/usr/share/doc/drbd-utils/", "/usr/share/man/man5/", "/usr/share/man/man8/", "/usr/share/doc/", "/var/lib/"}
+	ex6 := []string{"/etc/drbd.d/", "/lib/drbd/", "/usr/lib/drbd/", "/usr/lib/systemd/system/", "/usr/lib/tmpfiles.d/", "/usr/sbin/", "/usr/share/doc/drbd-utils/", "/usr/share/man/man5/", "/usr/share/man/man8/", "/var/lib/"}
+	sort.Strings(ex6)
+	out6 := RemoveCommonFilePrefixes(in6)
+	sort.Strings(out6)
+	if !reflect.DeepEqual(ex6, out6) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex6, out6)
+	}
+
+	in7 := []string{"/etc/", "/lib/", "/usr/lib/", "/usr/lib/systemd/", "/usr/", "/usr/share/doc/", "/usr/share/man/", "/var/"}
+	ex7 := []string{"/etc/", "/lib/", "/usr/lib/systemd/", "/usr/share/doc/", "/usr/share/man/", "/var/"}
+	sort.Strings(ex7)
+	out7 := RemoveCommonFilePrefixes(in7)
+	sort.Strings(out7)
+	if !reflect.DeepEqual(ex7, out7) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex7, out7)
+	}
+
+	in8 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d/",
+		"/usr/share/doc/drbd-utils/",
+		"/var/lib/drbd",
+	}
+	ex8 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/var/lib/drbd",
+	}
+	sort.Strings(ex8)
+	out8 := RemoveCommonFilePrefixes(in8)
+	sort.Strings(out8)
+	if !reflect.DeepEqual(ex8, out8) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex8, out8)
+	}
+
+	in9 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/",
+		"/lib/drbd/drbd",
+		"/lib/drbd/",
+		"/lib/drbd/",
+		"/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/systemd/system/",
+		"/usr/lib/tmpfiles.d/",
+		"/usr/sbin/",
+		"/usr/sbin/",
+		"/usr/share/doc/drbd-utils/",
+		"/usr/share/doc/drbd-utils/",
+		"/usr/share/man/man5/",
+		"/usr/share/man/man5/",
+		"/usr/share/man/man8/",
+		"/usr/share/man/man8/",
+		"/usr/share/man/man8/",
+		"/etc/drbd.d/",
+		"/usr/share/doc/drbd-utils/",
+		"/var/lib/drbd",
+	}
+	ex9 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/",
+		"/lib/drbd/drbd",
+		"/usr/lib/drbd/",
+		"/usr/lib/systemd/system/",
+		"/usr/lib/tmpfiles.d/",
+		"/usr/sbin/",
+		"/usr/share/doc/drbd-utils/",
+		"/usr/share/man/man5/",
+		"/usr/share/man/man8/",
+		"/var/lib/drbd",
+	}
+	sort.Strings(ex9)
+	out9 := RemoveCommonFilePrefixes(in9)
+	sort.Strings(out9)
+	if !reflect.DeepEqual(ex9, out9) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex9, out9)
+	}
+
+	in10 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/",                   // watch me, i'm a dir
+		"/etc/drbd.d/global_common.conf", // and watch me i'm a file!
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/", // watch me, i'm a dir too
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/var/lib/drbd",
+	}
+	ex10 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/var/lib/drbd",
+	}
+	sort.Strings(ex10)
+	out10 := RemoveCommonFilePrefixes(in10)
+	sort.Strings(out10)
+	if !reflect.DeepEqual(ex10, out10) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex10, out10)
+		for i := 0; i < len(ex10); i++ {
+			if ex10[i] != out10[i] {
+				t.Errorf("# %d: %v <> %v", i, ex10[i], out10[i])
+			}
+		}
+	}
+}
+
+func TestMiscFlattenListWithSplit1(t *testing.T) {
+	{
+		in := []string{} // input
+		ex := []string{} // expected
+		out := FlattenListWithSplit(in, []string{",", ";", " "})
+		sort.Strings(out)
+		sort.Strings(ex)
+		if !reflect.DeepEqual(ex, out) {
+			t.Errorf("FlattenListWithSplit expected: %v; got: %v.", ex, out)
+		}
+	}
+
+	{
+		in := []string{"hey"} // input
+		ex := []string{"hey"} // expected
+		out := FlattenListWithSplit(in, []string{",", ";", " "})
+		sort.Strings(out)
+		sort.Strings(ex)
+		if !reflect.DeepEqual(ex, out) {
+			t.Errorf("FlattenListWithSplit expected: %v; got: %v.", ex, out)
+		}
+	}
+
+	{
+		in := []string{"a", "b", "c", "d"} // input
+		ex := []string{"a", "b", "c", "d"} // expected
+		out := FlattenListWithSplit(in, []string{",", ";", " "})
+		sort.Strings(out)
+		sort.Strings(ex)
+		if !reflect.DeepEqual(ex, out) {
+			t.Errorf("FlattenListWithSplit expected: %v; got: %v.", ex, out)
+		}
+	}
+
+	{
+		in := []string{"a,b,c,d"}          // input
+		ex := []string{"a", "b", "c", "d"} // expected
+		out := FlattenListWithSplit(in, []string{",", ";", " "})
+		sort.Strings(out)
+		sort.Strings(ex)
+		if !reflect.DeepEqual(ex, out) {
+			t.Errorf("FlattenListWithSplit expected: %v; got: %v.", ex, out)
+		}
+	}
+
+	{
+		in := []string{"a,b;c d"}          // input (mixed)
+		ex := []string{"a", "b", "c", "d"} // expected
+		out := FlattenListWithSplit(in, []string{",", ";", " "})
+		sort.Strings(out)
+		sort.Strings(ex)
+		if !reflect.DeepEqual(ex, out) {
+			t.Errorf("FlattenListWithSplit expected: %v; got: %v.", ex, out)
+		}
+	}
+
+	{
+		in := []string{"a,b,c,d;e,f,g,h;i,j,k,l;m,n,o,p q,r,s,t;u,v,w,x y z"}                                                                            // input (mixed)
+		ex := []string{"a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z"} // expected
+		out := FlattenListWithSplit(in, []string{",", ";", " "})
+		sort.Strings(out)
+		sort.Strings(ex)
+		if !reflect.DeepEqual(ex, out) {
+			t.Errorf("FlattenListWithSplit expected: %v; got: %v.", ex, out)
+		}
+	}
+}

noop.go

@@ -0,0 +1,155 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"encoding/gob"
+	"log"
+)
+
+func init() {
+	gob.Register(&NoopRes{})
+}
+
+// NoopRes is a no-op resource that does nothing.
+type NoopRes struct {
+	BaseRes `yaml:",inline"`
+	Comment string `yaml:"comment"` // extra field for example purposes
+}
+
+// NewNoopRes is a constructor for this resource. It also calls Init() for you.
+func NewNoopRes(name string) *NoopRes {
+	obj := &NoopRes{
+		BaseRes: BaseRes{
+			Name: name,
+		},
+		Comment: "",
+	}
+	obj.Init()
+	return obj
+}
+
+// Init runs some startup code for this resource.
+func (obj *NoopRes) Init() {
+	obj.BaseRes.kind = "Noop"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+}
+
+// validate if the params passed in are valid data
+// FIXME: where should this get called ?
+func (obj *NoopRes) Validate() bool {
+	return true
+}
+
+// Watch is the primary listener for this resource and it outputs events.
+func (obj *NoopRes) Watch(processChan chan Event) {
+	if obj.IsWatching() {
+		return
+	}
+	obj.SetWatching(true)
+	defer obj.SetWatching(false)
+	cuuid := obj.converger.Register()
+	defer cuuid.Unregister()
+
+	var send = false // send event?
+	var exit = false
+	for {
+		obj.SetState(resStateWatching) // reset
+		select {
+		case event := <-obj.events:
+			cuuid.SetConverged(false)
+			// we avoid sending events on unpause
+			if exit, send = obj.ReadEvent(&event); exit {
+				return // exit
+			}
+
+		case <-cuuid.ConvergedTimer():
+			cuuid.SetConverged(true) // converged!
+			continue
+		}
+
+		// do all our event sending all together to avoid duplicate msgs
+		if send {
+			send = false
+			// only do this on certain types of events
+			//obj.isStateOK = false // something made state dirty
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
+		}
+	}
+}
+
+// CheckApply method for Noop resource. Does nothing, returns happy!
+func (obj *NoopRes) CheckApply(apply bool) (checkok bool, err error) {
+	log.Printf("%v[%v]: CheckApply(%t)", obj.Kind(), obj.GetName(), apply)
+	return true, nil // state is always okay
+}
+
+// NoopUUID is the UUID struct for NoopRes.
+type NoopUUID struct {
+	BaseUUID
+	name string
+}
+
+// The AutoEdges method returns the AutoEdges. In this case none are used.
+func (obj *NoopRes) AutoEdges() AutoEdge {
+	return nil
+}
+
+// GetUUIDs includes all params to make a unique identification of this object.
+// Most resources only return one, although some resources can return multiple.
+func (obj *NoopRes) GetUUIDs() []ResUUID {
+	x := &NoopUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		name:     obj.Name,
+	}
+	return []ResUUID{x}
+}
+
+// GroupCmp returns whether two resources can be grouped together or not.
+func (obj *NoopRes) GroupCmp(r Res) bool {
+	_, ok := r.(*NoopRes)
+	if !ok {
+		// NOTE: technically we could group a noop into any other
+		// resource, if that resource knew how to handle it, although,
+		// since the mechanics of inter-kind resource grouping are
+		// tricky, avoid doing this until there's a good reason.
+		return false
+	}
+	return true // noop resources can always be grouped together!
+}
+
+// Compare two resources and return if they are equivalent.
+func (obj *NoopRes) Compare(res Res) bool {
+	switch res.(type) {
+	// we can only compare NoopRes to others of the same resource
+	case *NoopRes:
+		res := res.(*NoopRes)
+		// calling base Compare is unneeded for the noop res
+		//if !obj.BaseRes.Compare(res) { // call base Compare
+		//	return false
+		//}
+		if obj.Name != res.Name {
+			return false
+		}
+	default:
+		return false
+	}
+	return true
+}

omv.yaml

@@ -0,0 +1,39 @@
+---
+:domain: example.com
+:network: 192.168.123.0/24
+:image: fedora-23
+:cpus: ''
+:memory: ''
+:disks: 0
+:disksize: 40G
+:boxurlprefix: ''
+:sync: rsync
+:syncdir: mgmt/
+:syncsrc: "../"
+:folder: ".omv"
+:extern: []
+:cd: "-"
+:puppet: false
+:classes: []
+:shell:
+- cd /vagrant/mgmt/ && make deps
+:docker: false
+:kubernetes: false
+:ansible: []
+:playbook: []
+:ansible_extras: {}
+:cachier: false
+:vms: []
+:namespace: omv
+:count: 1
+:username: ''
+:password: ''
+:poolid: true
+:repos: []
+:update: false
+:reboot: false
+:unsafe: false
+:nested: false
+:tests: []
+:comment: ''
+:reallyrm: false

packagekit.go

@@ -0,0 +1,923 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+// DOCS: https://www.freedesktop.org/software/PackageKit/gtk-doc/index.html
+
+//package packagekit // TODO
+package main
+
+import (
+	"fmt"
+	"github.com/godbus/dbus"
+	"log"
+	"runtime"
+	"strings"
+)
+
+// global tweaks of verbosity and code path
+const (
+	PK_DEBUG = false
+	PARANOID = false // enable if you see any ghosts
+)
+
+// constants which might need to be tweaked or which contain special dbus strings.
+const (
+	// FIXME: if PkBufferSize is too low, install seems to drop signals
+	PkBufferSize = 1000
+	// TODO: the PkSignalTimeout value might be too low
+	PkSignalPackageTimeout = 60 // 60 seconds, arbitrary
+	PkSignalDestroyTimeout = 15 // 15 seconds, arbitrary
+	PkPath                 = "/org/freedesktop/PackageKit"
+	PkIface                = "org.freedesktop.PackageKit"
+	PkIfaceTransaction     = PkIface + ".Transaction"
+	dbusAddMatch           = "org.freedesktop.DBus.AddMatch"
+)
+
+var (
+	// PkArchMap contains the mapping from PackageKit arch to GOARCH.
+	// GOARCH's: 386, amd64, arm, arm64, mips64, mips64le, ppc64, ppc64le
+	PkArchMap = map[string]string{ // map of PackageKit arch to GOARCH
+		// TODO: add more values
+		// noarch
+		"noarch": "ANY", // special value "ANY" (noarch as seen in Fedora)
+		"all":    "ANY", // special value "ANY" ('all' as seen in Debian)
+		// fedora
+		"x86_64":  "amd64",
+		"aarch64": "arm64",
+		// debian, from: https://www.debian.org/ports/
+		"amd64": "amd64",
+		"arm64": "arm64",
+		"i386":  "386",
+		"i486":  "386",
+		"i586":  "386",
+		"i686":  "386",
+	}
+)
+
+//type enum_filter uint64
+// https://github.com/hughsie/PackageKit/blob/master/lib/packagekit-glib2/pk-enum.c
+const ( //static const PkEnumMatch enum_filter[]
+	PK_FILTER_ENUM_UNKNOWN         uint64 = 1 << iota // "unknown"
+	PK_FILTER_ENUM_NONE                               // "none"
+	PK_FILTER_ENUM_INSTALLED                          // "installed"
+	PK_FILTER_ENUM_NOT_INSTALLED                      // "~installed"
+	PK_FILTER_ENUM_DEVELOPMENT                        // "devel"
+	PK_FILTER_ENUM_NOT_DEVELOPMENT                    // "~devel"
+	PK_FILTER_ENUM_GUI                                // "gui"
+	PK_FILTER_ENUM_NOT_GUI                            // "~gui"
+	PK_FILTER_ENUM_FREE                               // "free"
+	PK_FILTER_ENUM_NOT_FREE                           // "~free"
+	PK_FILTER_ENUM_VISIBLE                            // "visible"
+	PK_FILTER_ENUM_NOT_VISIBLE                        // "~visible"
+	PK_FILTER_ENUM_SUPPORTED                          // "supported"
+	PK_FILTER_ENUM_NOT_SUPPORTED                      // "~supported"
+	PK_FILTER_ENUM_BASENAME                           // "basename"
+	PK_FILTER_ENUM_NOT_BASENAME                       // "~basename"
+	PK_FILTER_ENUM_NEWEST                             // "newest"
+	PK_FILTER_ENUM_NOT_NEWEST                         // "~newest"
+	PK_FILTER_ENUM_ARCH                               // "arch"
+	PK_FILTER_ENUM_NOT_ARCH                           // "~arch"
+	PK_FILTER_ENUM_SOURCE                             // "source"
+	PK_FILTER_ENUM_NOT_SOURCE                         // "~source"
+	PK_FILTER_ENUM_COLLECTIONS                        // "collections"
+	PK_FILTER_ENUM_NOT_COLLECTIONS                    // "~collections"
+	PK_FILTER_ENUM_APPLICATION                        // "application"
+	PK_FILTER_ENUM_NOT_APPLICATION                    // "~application"
+	PK_FILTER_ENUM_DOWNLOADED                         // "downloaded"
+	PK_FILTER_ENUM_NOT_DOWNLOADED                     // "~downloaded"
+)
+
+// constants from packagekit c library.
+const ( //static const PkEnumMatch enum_transaction_flag[]
+	PK_TRANSACTION_FLAG_ENUM_NONE            uint64 = 1 << iota // "none"
+	PK_TRANSACTION_FLAG_ENUM_ONLY_TRUSTED                       // "only-trusted"
+	PK_TRANSACTION_FLAG_ENUM_SIMULATE                           // "simulate"
+	PK_TRANSACTION_FLAG_ENUM_ONLY_DOWNLOAD                      // "only-download"
+	PK_TRANSACTION_FLAG_ENUM_ALLOW_REINSTALL                    // "allow-reinstall"
+	PK_TRANSACTION_FLAG_ENUM_JUST_REINSTALL                     // "just-reinstall"
+	PK_TRANSACTION_FLAG_ENUM_ALLOW_DOWNGRADE                    // "allow-downgrade"
+)
+
+// constants from packagekit c library.
+const ( //typedef enum
+	PK_INFO_ENUM_UNKNOWN uint64 = 1 << iota
+	PK_INFO_ENUM_INSTALLED
+	PK_INFO_ENUM_AVAILABLE
+	PK_INFO_ENUM_LOW
+	PK_INFO_ENUM_ENHANCEMENT
+	PK_INFO_ENUM_NORMAL
+	PK_INFO_ENUM_BUGFIX
+	PK_INFO_ENUM_IMPORTANT
+	PK_INFO_ENUM_SECURITY
+	PK_INFO_ENUM_BLOCKED
+	PK_INFO_ENUM_DOWNLOADING
+	PK_INFO_ENUM_UPDATING
+	PK_INFO_ENUM_INSTALLING
+	PK_INFO_ENUM_REMOVING
+	PK_INFO_ENUM_CLEANUP
+	PK_INFO_ENUM_OBSOLETING
+	PK_INFO_ENUM_COLLECTION_INSTALLED
+	PK_INFO_ENUM_COLLECTION_AVAILABLE
+	PK_INFO_ENUM_FINISHED
+	PK_INFO_ENUM_REINSTALLING
+	PK_INFO_ENUM_DOWNGRADING
+	PK_INFO_ENUM_PREPARING
+	PK_INFO_ENUM_DECOMPRESSING
+	PK_INFO_ENUM_UNTRUSTED
+	PK_INFO_ENUM_TRUSTED
+	PK_INFO_ENUM_UNAVAILABLE
+	PK_INFO_ENUM_LAST
+)
+
+// Conn is a wrapper struct so we can pass bus connection around in the struct.
+type Conn struct {
+	conn *dbus.Conn
+}
+
+// PkPackageIDActionData is a struct that is returned by PackagesToPackageIDs in the map values.
+type PkPackageIDActionData struct {
+	Found     bool
+	Installed bool
+	Version   string
+	PackageID string
+	Newest    bool
+}
+
+// NewBus returns a new bus connection.
+func NewBus() *Conn {
+	// if we share the bus with others, we will get each others messages!!
+	bus, err := SystemBusPrivateUsable() // don't share the bus connection!
+	if err != nil {
+		return nil
+	}
+	return &Conn{
+		conn: bus,
+	}
+}
+
+// GetBus gets the dbus connection object.
+func (bus *Conn) GetBus() *dbus.Conn {
+	return bus.conn
+}
+
+// Close closes the dbus connection object.
+func (bus *Conn) Close() error {
+	return bus.conn.Close()
+}
+
+// internal helper to add signal matches to the bus, should only be called once
+func (bus *Conn) matchSignal(ch chan *dbus.Signal, path dbus.ObjectPath, iface string, signals []string) error {
+	if PK_DEBUG {
+		log.Printf("PackageKit: matchSignal(%v, %v, %v, %v)", ch, path, iface, signals)
+	}
+	// eg: gdbus monitor --system --dest org.freedesktop.PackageKit --object-path /org/freedesktop/PackageKit | grep <signal>
+	var call *dbus.Call
+	// TODO: if we make this call many times, we seem to receive signals
+	// that many times... Maybe this should be an object singleton?
+	obj := bus.GetBus().BusObject()
+	pathStr := fmt.Sprintf("%s", path)
+	if len(signals) == 0 {
+		call = obj.Call(dbusAddMatch, 0, "type='signal',path='"+pathStr+"',interface='"+iface+"'")
+	} else {
+		for _, signal := range signals {
+			call = obj.Call(dbusAddMatch, 0, "type='signal',path='"+pathStr+"',interface='"+iface+"',member='"+signal+"'")
+			if call.Err != nil {
+				break
+			}
+		}
+	}
+	if call.Err != nil {
+		return call.Err
+	}
+	// The caller has to make sure that ch is sufficiently buffered; if a
+	// message arrives when a write to c is not possible, it is discarded!
+	// This can be disastrous if we're waiting for a "Finished" signal!
+	bus.GetBus().Signal(ch)
+	return nil
+}
+
+// WatchChanges gets a signal anytime an event happens.
+func (bus *Conn) WatchChanges() (chan *dbus.Signal, error) {
+	ch := make(chan *dbus.Signal, PkBufferSize)
+	// NOTE: the TransactionListChanged signal fires much more frequently,
+	// but with much less specificity. If we're missing events, report the
+	// issue upstream! The UpdatesChanged signal is what hughsie suggested
+	var signal = "UpdatesChanged"
+	err := bus.matchSignal(ch, PkPath, PkIface, []string{signal})
+	if err != nil {
+		return nil, err
+	}
+	if PARANOID { // TODO: this filtering might not be necessary anymore...
+		// try to handle the filtering inside this function!
+		rch := make(chan *dbus.Signal)
+		go func() {
+		loop:
+			for {
+				select {
+				case event := <-ch:
+					// "A receive from a closed channel returns the
+					// zero value immediately": if i get nil here,
+					// it means the channel was closed by someone!!
+					if event == nil { // shared bus issue?
+						log.Println("PackageKit: Hrm, channel was closed!")
+						break loop // TODO: continue?
+					}
+					// i think this was caused by using the shared
+					// bus, but we might as well leave it in for now
+					if event.Path != PkPath || event.Name != fmt.Sprintf("%s.%s", PkIface, signal) {
+						log.Printf("PackageKit: Woops: Event: %+v", event)
+						continue
+					}
+					rch <- event // forward...
+				}
+			}
+			defer close(ch)
+		}()
+		return rch, nil
+	}
+	return ch, nil
+}
+
+// CreateTransaction creates and returns a transaction path.
+func (bus *Conn) CreateTransaction() (dbus.ObjectPath, error) {
+	if PK_DEBUG {
+		log.Println("PackageKit: CreateTransaction()")
+	}
+	var interfacePath dbus.ObjectPath
+	obj := bus.GetBus().Object(PkIface, PkPath)
+	call := obj.Call(fmt.Sprintf("%s.CreateTransaction", PkIface), 0).Store(&interfacePath)
+	if call != nil {
+		return "", call
+	}
+	if PK_DEBUG {
+		log.Printf("PackageKit: CreateTransaction(): %v", interfacePath)
+	}
+	return interfacePath, nil
+}
+
+// ResolvePackages runs the PackageKit Resolve method and returns the result.
+func (bus *Conn) ResolvePackages(packages []string, filter uint64) ([]string, error) {
+	packageIDs := []string{}
+	ch := make(chan *dbus.Signal, PkBufferSize)   // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction() // emits Destroy on close
+	if err != nil {
+		return []string{}, err
+	}
+
+	// add signal matches for Package and Finished which will always be last
+	var signals = []string{"Package", "Finished", "Error", "Destroy"}
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+	if PK_DEBUG {
+		log.Printf("PackageKit: ResolvePackages(): Object(%v, %v)", PkIface, interfacePath)
+	}
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("Resolve"), 0, filter, packages)
+	if PK_DEBUG {
+		log.Println("PackageKit: ResolvePackages(): Call: Success!")
+	}
+	if call.Err != nil {
+		return []string{}, call.Err
+	}
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+			if PK_DEBUG {
+				log.Printf("PackageKit: ResolvePackages(): Signal: %+v", signal)
+			}
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("Package") {
+				//pkg_int, ok := signal.Body[0].(int)
+				packageID, ok := signal.Body[1].(string)
+				// format is: name;version;arch;data
+				if !ok {
+					continue loop
+				}
+				//comment, ok := signal.Body[2].(string)
+				for _, p := range packageIDs {
+					if packageID == p {
+						continue loop // duplicate!
+					}
+				}
+				packageIDs = append(packageIDs, packageID)
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				return []string{}, fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		}
+	}
+	return packageIDs, nil
+}
+
+// IsInstalledList queries a list of packages to see if they are installed.
+func (bus *Conn) IsInstalledList(packages []string) ([]bool, error) {
+	var filter uint64             // initializes at the "zero" value of 0
+	filter += PK_FILTER_ENUM_ARCH // always search in our arch
+	packageIDs, e := bus.ResolvePackages(packages, filter)
+	if e != nil {
+		return nil, fmt.Errorf("ResolvePackages error: %v", e)
+	}
+
+	var m = make(map[string]int)
+	for _, packageID := range packageIDs {
+		s := strings.Split(packageID, ";")
+		//if len(s) != 4 { continue } // this would be a bug!
+		pkg := s[0]
+		flags := strings.Split(s[3], ":")
+		for _, f := range flags {
+			if f == "installed" {
+				if _, exists := m[pkg]; !exists {
+					m[pkg] = 0
+				}
+				m[pkg]++ // if we see pkg installed, increment
+				break
+			}
+		}
+	}
+
+	var r []bool
+	for _, p := range packages {
+		if value, exists := m[p]; exists {
+			r = append(r, value > 0) // at least 1 means installed
+		} else {
+			r = append(r, false)
+		}
+	}
+	return r, nil
+}
+
+// IsInstalled returns if a package is installed.
+// TODO: this could be optimized by making the resolve call directly
+func (bus *Conn) IsInstalled(pkg string) (bool, error) {
+	p, e := bus.IsInstalledList([]string{pkg})
+	if len(p) != 1 {
+		return false, e
+	}
+	return p[0], nil
+}
+
+// InstallPackages installs a list of packages by packageID.
+func (bus *Conn) InstallPackages(packageIDs []string, transactionFlags uint64) error {
+
+	ch := make(chan *dbus.Signal, PkBufferSize)   // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction() // emits Destroy on close
+	if err != nil {
+		return err
+	}
+
+	var signals = []string{"Package", "ErrorCode", "Finished", "Destroy"} // "ItemProgress", "Status" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("InstallPackages"), 0, transactionFlags, packageIDs)
+	if call.Err != nil {
+		return call.Err
+	}
+	timeout := -1 // disabled initially
+	finished := false
+loop:
+	for {
+		select {
+		case signal := <-ch:
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			} else if signal.Name == FmtTransactionMethod("Package") {
+				// a package was installed...
+				// only start the timer once we're here...
+				timeout = PkSignalPackageTimeout
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				finished = true
+				timeout = PkSignalDestroyTimeout // wait a bit
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				return nil // success
+			} else {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		case <-TimeAfterOrBlock(timeout):
+			if finished {
+				log.Println("PackageKit: Timeout: InstallPackages: Waiting for 'Destroy'")
+				return nil // got tired of waiting for Destroy
+			}
+			return fmt.Errorf("PackageKit: Timeout: InstallPackages: %v", strings.Join(packageIDs, ", "))
+		}
+	}
+}
+
+// RemovePackages removes a list of packages by packageID.
+func (bus *Conn) RemovePackages(packageIDs []string, transactionFlags uint64) error {
+
+	var allowDeps = true                          // TODO: configurable
+	var autoremove = false                        // unsupported on GNU/Linux
+	ch := make(chan *dbus.Signal, PkBufferSize)   // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction() // emits Destroy on close
+	if err != nil {
+		return err
+	}
+
+	var signals = []string{"Package", "ErrorCode", "Finished", "Destroy"} // "ItemProgress", "Status" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("RemovePackages"), 0, transactionFlags, packageIDs, allowDeps, autoremove)
+	if call.Err != nil {
+		return call.Err
+	}
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			} else if signal.Name == FmtTransactionMethod("Package") {
+				// a package was installed...
+				continue loop
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		}
+	}
+	return nil
+}
+
+// UpdatePackages updates a list of packages to versions that are specified.
+func (bus *Conn) UpdatePackages(packageIDs []string, transactionFlags uint64) error {
+	ch := make(chan *dbus.Signal, PkBufferSize) // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction()
+	if err != nil {
+		return err
+	}
+
+	var signals = []string{"Package", "ErrorCode", "Finished", "Destroy"} // "ItemProgress", "Status" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("UpdatePackages"), 0, transactionFlags, packageIDs)
+	if call.Err != nil {
+		return call.Err
+	}
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			} else if signal.Name == FmtTransactionMethod("Package") {
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		}
+	}
+	return nil
+}
+
+// GetFilesByPackageID gets the list of files that are contained inside a list of packageIDs.
+func (bus *Conn) GetFilesByPackageID(packageIDs []string) (files map[string][]string, err error) {
+	// NOTE: the maximum number of files in an RPM is 52116 in Fedora 23
+	// https://gist.github.com/purpleidea/b98e60dcd449e1ac3b8a
+	ch := make(chan *dbus.Signal, PkBufferSize) // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction()
+	if err != nil {
+		return
+	}
+
+	var signals = []string{"Files", "ErrorCode", "Finished", "Destroy"} // "ItemProgress", "Status" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("GetFiles"), 0, packageIDs)
+	if call.Err != nil {
+		err = call.Err
+		return
+	}
+	files = make(map[string][]string)
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				err = fmt.Errorf("PackageKit: Error: %v", signal.Body)
+				return
+
+				// one signal returned per packageID found...
+			} else if signal.Name == FmtTransactionMethod("Files") {
+				if len(signal.Body) != 2 { // bad data
+					continue loop
+				}
+				var ok bool
+				var key string
+				var fileList []string
+				if key, ok = signal.Body[0].(string); !ok {
+					continue loop
+				}
+				if fileList, ok = signal.Body[1].([]string); !ok {
+					continue loop // failed conversion
+				}
+				files[key] = fileList // build up map
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				err = fmt.Errorf("PackageKit: Error: %v", signal.Body)
+				return
+			}
+		}
+	}
+	return
+}
+
+// GetUpdates gets a list of packages that are installed and which can be updated, mod filter.
+func (bus *Conn) GetUpdates(filter uint64) ([]string, error) {
+	if PK_DEBUG {
+		log.Println("PackageKit: GetUpdates()")
+	}
+	packageIDs := []string{}
+	ch := make(chan *dbus.Signal, PkBufferSize) // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction()
+	if err != nil {
+		return nil, err
+	}
+
+	var signals = []string{"Package", "ErrorCode", "Finished", "Destroy"} // "ItemProgress" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("GetUpdates"), 0, filter)
+	if call.Err != nil {
+		return nil, call.Err
+	}
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				return nil, fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			} else if signal.Name == FmtTransactionMethod("Package") {
+
+				//pkg_int, ok := signal.Body[0].(int)
+				packageID, ok := signal.Body[1].(string)
+				// format is: name;version;arch;data
+				if !ok {
+					continue loop
+				}
+				//comment, ok := signal.Body[2].(string)
+				for _, p := range packageIDs { // optional?
+					if packageID == p {
+						continue loop // duplicate!
+					}
+				}
+				packageIDs = append(packageIDs, packageID)
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				return nil, fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		}
+	}
+	return packageIDs, nil
+}
+
+// PackagesToPackageIDs is a helper function that *might* be generally useful
+// outside mgmt. The packageMap input has the package names as keys and
+// requested states as values. These states can be: installed, uninstalled,
+// newest or a requested version str.
+func (bus *Conn) PackagesToPackageIDs(packageMap map[string]string, filter uint64) (map[string]*PkPackageIDActionData, error) {
+	count := 0
+	packages := make([]string, len(packageMap))
+	for k := range packageMap { // lol, golang has no hash.keys() function!
+		packages[count] = k
+		count++
+	}
+
+	if !(filter&PK_FILTER_ENUM_ARCH == PK_FILTER_ENUM_ARCH) {
+		filter += PK_FILTER_ENUM_ARCH // always search in our arch
+	}
+
+	if PK_DEBUG {
+		log.Printf("PackageKit: PackagesToPackageIDs(): %v", strings.Join(packages, ", "))
+	}
+	resolved, e := bus.ResolvePackages(packages, filter)
+	if e != nil {
+		return nil, fmt.Errorf("Resolve error: %v", e)
+	}
+
+	found := make([]bool, count) // default false
+	installed := make([]bool, count)
+	version := make([]string, count)
+	usePackageID := make([]string, count)
+	newest := make([]bool, count) // default true
+	for i := range newest {
+		newest[i] = true // assume, for now
+	}
+	var index int
+
+	for _, packageID := range resolved {
+		index = -1
+		//log.Printf("* %v", packageID)
+		// format is: name;version;arch;data
+		s := strings.Split(packageID, ";")
+		//if len(s) != 4 { continue } // this would be a bug!
+		pkg, ver, arch, data := s[0], s[1], s[2], s[3]
+		// we might need to allow some of this, eg: i386 .deb on amd64
+		if !IsMyArch(arch) {
+			continue
+		}
+
+		for i := range packages { // find pkg if it exists
+			if pkg == packages[i] {
+				index = i
+			}
+		}
+		if index == -1 { // can't find what we're looking for
+			continue
+		}
+		state := packageMap[pkg] // lookup the requested state/version
+		if state == "" {
+			return nil, fmt.Errorf("Empty package state for %v", pkg)
+		}
+		found[index] = true
+		stateIsVersion := (state != "installed" && state != "uninstalled" && state != "newest") // must be a ver. string
+
+		if stateIsVersion {
+			if state == ver && ver != "" { // we match what we want...
+				usePackageID[index] = packageID
+			}
+		}
+
+		if FlagInData("installed", data) {
+			installed[index] = true
+			version[index] = ver
+			// state of "uninstalled" matched during CheckApply, and
+			// states of "installed" and "newest" for fileList
+			if !stateIsVersion {
+				usePackageID[index] = packageID // save for later
+			}
+		} else { // not installed...
+			if !stateIsVersion {
+				// if there is more than one result, eg: there
+				// is the old and newest version of a package,
+				// then this section can run more than once...
+				// in that case, don't worry, we'll choose the
+				// right value in the "updates" section below!
+				usePackageID[index] = packageID
+			}
+		}
+	}
+
+	// we can't determine which packages are "newest", without searching
+	// for each one individually, so instead we check if any updates need
+	// to be done, and if so, anything that needs updating isn't newest!
+	// if something isn't installed, we can't verify it with this method
+	// FIXME: https://github.com/hughsie/PackageKit/issues/116
+	updates, e := bus.GetUpdates(filter)
+	if e != nil {
+		return nil, fmt.Errorf("Updates error: %v", e)
+	}
+	for _, packageID := range updates {
+		//log.Printf("* %v", packageID)
+		// format is: name;version;arch;data
+		s := strings.Split(packageID, ";")
+		//if len(s) != 4 { continue } // this would be a bug!
+		pkg, _, _, _ := s[0], s[1], s[2], s[3]
+		for index := range packages { // find pkg if it exists
+			if pkg == packages[index] {
+				state := packageMap[pkg] // lookup
+				newest[index] = false
+				if state == "installed" || state == "newest" {
+					// fix up in case above wasn't correct!
+					usePackageID[index] = packageID
+				}
+				break
+			}
+		}
+	}
+
+	// skip if the "newest" filter was used, otherwise we might need fixing
+	// this check is for packages that need to verify their "newest" status
+	// we need to know this so we can install the correct newest packageID!
+	recursion := make(map[string]*PkPackageIDActionData)
+	if !(filter&PK_FILTER_ENUM_NEWEST == PK_FILTER_ENUM_NEWEST) {
+		checkPackages := []string{}
+		filteredPackageMap := make(map[string]string)
+		for index, pkg := range packages {
+			state := packageMap[pkg]               // lookup the requested state/version
+			if !found[index] || installed[index] { // skip these, they're okay
+				continue
+			}
+			if !(state == "newest" || state == "installed") {
+				continue
+			}
+
+			checkPackages = append(checkPackages, pkg)
+			filteredPackageMap[pkg] = packageMap[pkg] // check me!
+		}
+
+		// we _could_ do a second resolve and then parse like this...
+		//resolved, e := bus.ResolvePackages(..., filter+PK_FILTER_ENUM_NEWEST)
+		// but that's basically what recursion here could do too!
+		if len(checkPackages) > 0 {
+			if PK_DEBUG {
+				log.Printf("PackageKit: PackagesToPackageIDs(): Recurse: %v", strings.Join(checkPackages, ", "))
+			}
+			recursion, e = bus.PackagesToPackageIDs(filteredPackageMap, filter+PK_FILTER_ENUM_NEWEST)
+			if e != nil {
+				return nil, fmt.Errorf("Recursion error: %v", e)
+			}
+		}
+	}
+
+	// fix up and build result format
+	result := make(map[string]*PkPackageIDActionData)
+	for index, pkg := range packages {
+
+		if !found[index] || !installed[index] {
+			newest[index] = false // make the results more logical!
+		}
+
+		// prefer recursion results if present
+		if lookup, ok := recursion[pkg]; ok {
+			result[pkg] = lookup
+		} else {
+			result[pkg] = &PkPackageIDActionData{
+				Found:     found[index],
+				Installed: installed[index],
+				Version:   version[index],
+				PackageID: usePackageID[index],
+				Newest:    newest[index],
+			}
+		}
+	}
+
+	return result, nil
+}
+
+// FilterPackageIDs returns a list of packageIDs which match the set of package names in packages.
+func FilterPackageIDs(m map[string]*PkPackageIDActionData, packages []string) ([]string, error) {
+	result := []string{}
+	for _, k := range packages {
+		obj, ok := m[k] // lookup single package
+		// package doesn't exist, this is an error!
+		if !ok || !obj.Found || obj.PackageID == "" {
+			return nil, fmt.Errorf("Can't find package named '%s'.", k)
+		}
+		result = append(result, obj.PackageID)
+	}
+	return result, nil
+}
+
+// FilterState returns a map of whether each package queried matches the particular state.
+func FilterState(m map[string]*PkPackageIDActionData, packages []string, state string) (result map[string]bool, err error) {
+	result = make(map[string]bool)
+	pkgs := []string{} // bad pkgs that don't have a bool state
+	for _, k := range packages {
+		obj, ok := m[k] // lookup single package
+		// package doesn't exist, this is an error!
+		if !ok || !obj.Found {
+			return nil, fmt.Errorf("Can't find package named '%s'.", k)
+		}
+		var b bool
+		if state == "installed" {
+			b = obj.Installed
+		} else if state == "uninstalled" {
+			b = !obj.Installed
+		} else if state == "newest" {
+			b = obj.Newest
+		} else {
+			// we can't filter "version" state in this function
+			pkgs = append(pkgs, k)
+			continue
+		}
+		result[k] = b // save
+	}
+	if len(pkgs) > 0 {
+		err = fmt.Errorf("Can't filter non-boolean state on: %v!", strings.Join(pkgs, ","))
+	}
+	return result, err
+}
+
+// FilterPackageState returns all packages that are in package and match the specific state.
+func FilterPackageState(m map[string]*PkPackageIDActionData, packages []string, state string) (result []string, err error) {
+	result = []string{}
+	for _, k := range packages {
+		obj, ok := m[k] // lookup single package
+		// package doesn't exist, this is an error!
+		if !ok || !obj.Found {
+			return nil, fmt.Errorf("Can't find package named '%s'.", k)
+		}
+		b := false
+		if state == "installed" && obj.Installed {
+			b = true
+		} else if state == "uninstalled" && !obj.Installed {
+			b = true
+		} else if state == "newest" && obj.Newest {
+			b = true
+		} else if state == obj.Version {
+			b = true
+		}
+		if b {
+			result = append(result, k)
+		}
+	}
+	return result, err
+}
+
+// FlagInData asks whether a flag exists inside the data portion of a packageID field?
+func FlagInData(flag, data string) bool {
+	flags := strings.Split(data, ":")
+	for _, f := range flags {
+		if f == flag {
+			return true
+		}
+	}
+	return false
+}
+
+// FmtTransactionMethod builds the transaction method string properly.
+func FmtTransactionMethod(method string) string {
+	return fmt.Sprintf("%s.%s", PkIfaceTransaction, method)
+}
+
+// IsMyArch determines if a PackageKit architecture matches the current os arch.
+func IsMyArch(arch string) bool {
+	goarch, ok := PkArchMap[arch]
+	if !ok {
+		// if you get this error, please update the PkArchMap const
+		log.Fatalf("PackageKit: Arch '%v', not found!", arch)
+	}
+	if goarch == "ANY" { // special value that corresponds to noarch
+		return true
+	}
+	return goarch == runtime.GOARCH
+}

pkg.go

@@ -0,0 +1,565 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	//"packagekit" // TODO
+	"encoding/gob"
+	"errors"
+	"fmt"
+	"log"
+	"path"
+	"strings"
+)
+
+func init() {
+	gob.Register(&PkgRes{})
+}
+
+// PkgRes is a package resource for packagekit.
+type PkgRes struct {
+	BaseRes          `yaml:",inline"`
+	State            string `yaml:"state"`            // state: installed, uninstalled, newest, <version>
+	AllowUntrusted   bool   `yaml:"allowuntrusted"`   // allow untrusted packages to be installed?
+	AllowNonFree     bool   `yaml:"allownonfree"`     // allow nonfree packages to be found?
+	AllowUnsupported bool   `yaml:"allowunsupported"` // allow unsupported packages to be found?
+	//bus              *Conn    // pk bus connection
+	fileList []string // FIXME: update if pkg changes
+}
+
+// NewPkgRes is a constructor for this resource. It also calls Init() for you.
+func NewPkgRes(name, state string, allowuntrusted, allownonfree, allowunsupported bool) *PkgRes {
+	obj := &PkgRes{
+		BaseRes: BaseRes{
+			Name: name,
+		},
+		State:            state,
+		AllowUntrusted:   allowuntrusted,
+		AllowNonFree:     allownonfree,
+		AllowUnsupported: allowunsupported,
+	}
+	obj.Init()
+	return obj
+}
+
+// Init runs some startup code for this resource.
+func (obj *PkgRes) Init() {
+	obj.BaseRes.kind = "Pkg"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+
+	bus := NewBus()
+	if bus == nil {
+		log.Fatal("Can't connect to PackageKit bus.")
+	}
+	defer bus.Close()
+
+	result, err := obj.pkgMappingHelper(bus)
+	if err != nil {
+		// FIXME: return error?
+		log.Fatalf("The pkgMappingHelper failed with: %v.", err)
+		return
+	}
+
+	data, ok := result[obj.Name] // lookup single package (init does just one)
+	// package doesn't exist, this is an error!
+	if !ok || !data.Found {
+		// FIXME: return error?
+		log.Fatalf("Can't find package named '%s'.", obj.Name)
+		return
+	}
+
+	packageIDs := []string{data.PackageID} // just one for now
+	filesMap, err := bus.GetFilesByPackageID(packageIDs)
+	if err != nil {
+		// FIXME: return error?
+		log.Fatalf("Can't run GetFilesByPackageID: %v", err)
+		return
+	}
+	if files, ok := filesMap[data.PackageID]; ok {
+		obj.fileList = DirifyFileList(files, false)
+	}
+}
+
+// Validate checks if the resource data structure was populated correctly.
+func (obj *PkgRes) Validate() bool {
+	if obj.State == "" {
+		return false
+	}
+
+	return true
+}
+
+// Watch is the primary listener for this resource and it outputs events.
+// It uses the PackageKit UpdatesChanged signal to watch for changes.
+// TODO: https://github.com/hughsie/PackageKit/issues/109
+// TODO: https://github.com/hughsie/PackageKit/issues/110
+func (obj *PkgRes) Watch(processChan chan Event) {
+	if obj.IsWatching() {
+		return
+	}
+	obj.SetWatching(true)
+	defer obj.SetWatching(false)
+	cuuid := obj.converger.Register()
+	defer cuuid.Unregister()
+
+	bus := NewBus()
+	if bus == nil {
+		log.Fatal("Can't connect to PackageKit bus.")
+	}
+	defer bus.Close()
+
+	ch, err := bus.WatchChanges()
+	if err != nil {
+		log.Fatalf("Error adding signal match: %v", err)
+	}
+
+	var send = false // send event?
+	var exit = false
+	var dirty = false
+
+	for {
+		if DEBUG {
+			log.Printf("%v: Watching...", obj.fmtNames(obj.getNames()))
+		}
+
+		obj.SetState(resStateWatching) // reset
+		select {
+		case event := <-ch:
+			cuuid.SetConverged(false)
+
+			// FIXME: ask packagekit for info on what packages changed
+			if DEBUG {
+				log.Printf("%v: Event: %v", obj.fmtNames(obj.getNames()), event.Name)
+			}
+
+			// since the chan is buffered, remove any supplemental
+			// events since they would just be duplicates anyways!
+			for len(ch) > 0 { // we can detect pending count here!
+				<-ch // discard
+			}
+
+			send = true
+			dirty = true
+
+		case event := <-obj.events:
+			cuuid.SetConverged(false)
+			if exit, send = obj.ReadEvent(&event); exit {
+				return // exit
+			}
+			dirty = false // these events don't invalidate state
+
+		case <-cuuid.ConvergedTimer():
+			cuuid.SetConverged(true) // converged!
+			continue
+		}
+
+		// do all our event sending all together to avoid duplicate msgs
+		if send {
+			send = false
+			// only invalid state on certain types of events
+			if dirty {
+				dirty = false
+				obj.isStateOK = false // something made state dirty
+			}
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
+		}
+	}
+}
+
+// get list of names when grouped or not
+func (obj *PkgRes) getNames() []string {
+	if g := obj.GetGroup(); len(g) > 0 { // grouped elements
+		names := []string{obj.GetName()}
+		for _, x := range g {
+			pkg, ok := x.(*PkgRes) // convert from Res
+			if ok {
+				names = append(names, pkg.Name)
+			}
+		}
+		return names
+	}
+	return []string{obj.GetName()}
+}
+
+// pretty print for header values
+func (obj *PkgRes) fmtNames(names []string) string {
+	if len(obj.GetGroup()) > 0 { // grouped elements
+		return fmt.Sprintf("%v[autogroup:(%v)]", obj.Kind(), strings.Join(names, ","))
+	}
+	return fmt.Sprintf("%v[%v]", obj.Kind(), obj.GetName())
+}
+
+func (obj *PkgRes) groupMappingHelper() map[string]string {
+	var result = make(map[string]string)
+	if g := obj.GetGroup(); len(g) > 0 { // add any grouped elements
+		for _, x := range g {
+			pkg, ok := x.(*PkgRes) // convert from Res
+			if !ok {
+				log.Fatalf("Grouped member %v is not a %v", x, obj.Kind())
+			}
+			result[pkg.Name] = pkg.State
+		}
+	}
+	return result
+}
+
+func (obj *PkgRes) pkgMappingHelper(bus *Conn) (map[string]*PkPackageIDActionData, error) {
+	packageMap := obj.groupMappingHelper() // get the grouped values
+	packageMap[obj.Name] = obj.State       // key is pkg name, value is pkg state
+	var filter uint64                      // initializes at the "zero" value of 0
+	filter += PK_FILTER_ENUM_ARCH          // always search in our arch (optional!)
+	// we're requesting latest version, or to narrow down install choices!
+	if obj.State == "newest" || obj.State == "installed" {
+		// if we add this, we'll still see older packages if installed
+		// this is an optimization, and is *optional*, this logic is
+		// handled inside of PackagesToPackageIDs now automatically!
+		filter += PK_FILTER_ENUM_NEWEST // only search for newest packages
+	}
+	if !obj.AllowNonFree {
+		filter += PK_FILTER_ENUM_FREE
+	}
+	if !obj.AllowUnsupported {
+		filter += PK_FILTER_ENUM_SUPPORTED
+	}
+	result, e := bus.PackagesToPackageIDs(packageMap, filter)
+	if e != nil {
+		return nil, fmt.Errorf("Can't run PackagesToPackageIDs: %v", e)
+	}
+	return result, nil
+}
+
+// CheckApply checks the resource state and applies the resource if the bool
+// input is true. It returns error info and if the state check passed or not.
+func (obj *PkgRes) CheckApply(apply bool) (checkok bool, err error) {
+	log.Printf("%v: CheckApply(%t)", obj.fmtNames(obj.getNames()), apply)
+
+	if obj.State == "" { // TODO: Validate() should replace this check!
+		log.Fatalf("%v: Package state is undefined!", obj.fmtNames(obj.getNames()))
+	}
+
+	if obj.isStateOK { // cache the state
+		return true, nil
+	}
+
+	bus := NewBus()
+	if bus == nil {
+		return false, errors.New("Can't connect to PackageKit bus.")
+	}
+	defer bus.Close()
+
+	result, err := obj.pkgMappingHelper(bus)
+	if err != nil {
+		return false, fmt.Errorf("The pkgMappingHelper failed with: %v.", err)
+	}
+
+	packageMap := obj.groupMappingHelper() // map[string]string
+	packageList := []string{obj.Name}
+	packageList = append(packageList, StrMapKeys(packageMap)...)
+	//stateList := []string{obj.State}
+	//stateList = append(stateList, StrMapValues(packageMap)...)
+
+	// TODO: at the moment, all the states are the same, but
+	// eventually we might be able to drop this constraint!
+	states, err := FilterState(result, packageList, obj.State)
+	if err != nil {
+		return false, fmt.Errorf("The FilterState method failed with: %v.", err)
+	}
+	data, _ := result[obj.Name] // if above didn't error, we won't either!
+	validState := BoolMapTrue(BoolMapValues(states))
+
+	// obj.State == "installed" || "uninstalled" || "newest" || "4.2-1.fc23"
+	switch obj.State {
+	case "installed":
+		fallthrough
+	case "uninstalled":
+		fallthrough
+	case "newest":
+		if validState {
+			obj.isStateOK = true // reset
+			return true, nil     // state is correct, exit!
+		}
+	default: // version string
+		if obj.State == data.Version && data.Version != "" {
+			obj.isStateOK = true // reset
+			return true, nil
+		}
+	}
+
+	// state is not okay, no work done, exit, but without error
+	if !apply {
+		return false, nil
+	}
+
+	// apply portion
+	log.Printf("%v: Apply", obj.fmtNames(obj.getNames()))
+	readyPackages, err := FilterPackageState(result, packageList, obj.State)
+	if err != nil {
+		return false, err // fail
+	}
+	// these are the packages that actually need their states applied!
+	applyPackages := StrFilterElementsInList(readyPackages, packageList)
+	packageIDs, _ := FilterPackageIDs(result, applyPackages) // would be same err as above
+
+	var transactionFlags uint64 // initializes at the "zero" value of 0
+	if !obj.AllowUntrusted {    // allow
+		transactionFlags += PK_TRANSACTION_FLAG_ENUM_ONLY_TRUSTED
+	}
+	// apply correct state!
+	log.Printf("%v: Set: %v...", obj.fmtNames(StrListIntersection(applyPackages, obj.getNames())), obj.State)
+	switch obj.State {
+	case "uninstalled": // run remove
+		// NOTE: packageID is different than when installed, because now
+		// it has the "installed" flag added to the data portion if it!!
+		err = bus.RemovePackages(packageIDs, transactionFlags)
+
+	case "newest": // TODO: isn't this the same operation as install, below?
+		err = bus.UpdatePackages(packageIDs, transactionFlags)
+
+	case "installed":
+		fallthrough // same method as for "set specific version", below
+	default: // version string
+		err = bus.InstallPackages(packageIDs, transactionFlags)
+	}
+	if err != nil {
+		return false, err // fail
+	}
+	log.Printf("%v: Set: %v success!", obj.fmtNames(StrListIntersection(applyPackages, obj.getNames())), obj.State)
+	obj.isStateOK = true // reset
+	return false, nil    // success
+}
+
+// PkgUUID is the UUID struct for PkgRes.
+type PkgUUID struct {
+	BaseUUID
+	name  string // pkg name
+	state string // pkg state or "version"
+}
+
+// if and only if they are equivalent, return true
+// if they are not equivalent, return false
+func (obj *PkgUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*PkgUUID)
+	if !ok {
+		return false
+	}
+	// FIXME: match on obj.state vs. res.state ?
+	return obj.name == res.name
+}
+
+// PkgResAutoEdges holds the state of the auto edge generator.
+type PkgResAutoEdges struct {
+	fileList   []string
+	svcUUIDs   []ResUUID
+	testIsNext bool   // safety
+	name       string // saved data from PkgRes obj
+	kind       string
+}
+
+// Next returns the next automatic edge.
+func (obj *PkgResAutoEdges) Next() []ResUUID {
+	if obj.testIsNext {
+		log.Fatal("Expecting a call to Test()")
+	}
+	obj.testIsNext = true // set after all the errors paths are past
+
+	// first return any matching svcUUIDs
+	if x := obj.svcUUIDs; len(x) > 0 {
+		return x
+	}
+
+	var result []ResUUID
+	// return UUID's for whatever is in obj.fileList
+	for _, x := range obj.fileList {
+		var reversed = false // cheat by passing a pointer
+		result = append(result, &FileUUID{
+			BaseUUID: BaseUUID{
+				name:     obj.name,
+				kind:     obj.kind,
+				reversed: &reversed,
+			},
+			path: x, // what matters
+		}) // build list
+	}
+	return result
+}
+
+// Test gets results of the earlier Next() call, & returns if we should continue!
+func (obj *PkgResAutoEdges) Test(input []bool) bool {
+	if !obj.testIsNext {
+		log.Fatal("Expecting a call to Next()")
+	}
+
+	// ack the svcUUID's...
+	if x := obj.svcUUIDs; len(x) > 0 {
+		if y := len(x); y != len(input) {
+			log.Fatalf("Expecting %d value(s)!", y)
+		}
+		obj.svcUUIDs = []ResUUID{} // empty
+		obj.testIsNext = false
+		return true
+	}
+
+	count := len(obj.fileList)
+	if count != len(input) {
+		log.Fatalf("Expecting %d value(s)!", count)
+	}
+	obj.testIsNext = false // set after all the errors paths are past
+
+	// while i do believe this algorithm generates the *correct* result, i
+	// don't know if it does so in the optimal way. improvements welcome!
+	// the basic logic is:
+	// 0) Next() returns whatever is in fileList
+	// 1) Test() computes the dirname of each file, and removes duplicates
+	// and dirname's that have been in the path of an ack from input results
+	// 2) It then simplifies the list by removing the common path prefixes
+	// 3) Lastly, the remaining set of files (dirs) is used as new fileList
+	// 4) We then iterate in (0) until the fileList is empty!
+	var dirs = make([]string, count)
+	done := []string{}
+	for i := 0; i < count; i++ {
+		dir := Dirname(obj.fileList[i]) // dirname of /foo/ should be /
+		dirs[i] = dir
+		if input[i] {
+			done = append(done, dir)
+		}
+	}
+	nodupes := StrRemoveDuplicatesInList(dirs)                // remove duplicates
+	nodones := StrFilterElementsInList(done, nodupes)         // filter out done
+	noempty := StrFilterElementsInList([]string{""}, nodones) // remove the "" from /
+	obj.fileList = RemoveCommonFilePrefixes(noempty)          // magic
+
+	if len(obj.fileList) == 0 { // nothing more, don't continue
+		return false
+	}
+	return true // continue, there are more files!
+}
+
+// AutoEdges produces an object which generates a minimal pkg file optimization
+// sequence of edges.
+func (obj *PkgRes) AutoEdges() AutoEdge {
+	// in contrast with the FileRes AutoEdges() function which contains
+	// more of the mechanics, most of the AutoEdge mechanics for the PkgRes
+	// is contained in the Test() method! This design is completely okay!
+
+	// add matches for any svc resources found in pkg definition!
+	var svcUUIDs []ResUUID
+	for _, x := range ReturnSvcInFileList(obj.fileList) {
+		var reversed = false
+		svcUUIDs = append(svcUUIDs, &SvcUUID{
+			BaseUUID: BaseUUID{
+				name:     obj.GetName(),
+				kind:     obj.Kind(),
+				reversed: &reversed,
+			},
+			name: x, // the svc name itself in the SvcUUID object!
+		}) // build list
+	}
+
+	return &PkgResAutoEdges{
+		fileList:   RemoveCommonFilePrefixes(obj.fileList), // clean start!
+		svcUUIDs:   svcUUIDs,
+		testIsNext: false,         // start with Next() call
+		name:       obj.GetName(), // save data for PkgResAutoEdges obj
+		kind:       obj.Kind(),
+	}
+}
+
+// GetUUIDs includes all params to make a unique identification of this object.
+// Most resources only return one, although some resources can return multiple.
+func (obj *PkgRes) GetUUIDs() []ResUUID {
+	x := &PkgUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		name:     obj.Name,
+		state:    obj.State,
+	}
+	result := []ResUUID{x}
+	return result
+}
+
+// GroupCmp returns whether two resources can be grouped together or not.
+// can these two resources be merged ?
+// (aka does this resource support doing so?)
+// will resource allow itself to be grouped _into_ this obj?
+func (obj *PkgRes) GroupCmp(r Res) bool {
+	res, ok := r.(*PkgRes)
+	if !ok {
+		return false
+	}
+	objStateIsVersion := (obj.State != "installed" && obj.State != "uninstalled" && obj.State != "newest") // must be a ver. string
+	resStateIsVersion := (res.State != "installed" && res.State != "uninstalled" && res.State != "newest") // must be a ver. string
+	if objStateIsVersion || resStateIsVersion {
+		// can't merge specific version checks atm
+		return false
+	}
+	// FIXME: keep it simple for now, only merge same states
+	if obj.State != res.State {
+		return false
+	}
+	return true
+}
+
+// Compare two resources and return if they are equivalent.
+func (obj *PkgRes) Compare(res Res) bool {
+	switch res.(type) {
+	case *PkgRes:
+		res := res.(*PkgRes)
+		if !obj.BaseRes.Compare(res) { // call base Compare
+			return false
+		}
+
+		if obj.Name != res.Name {
+			return false
+		}
+		if obj.State != res.State {
+			return false
+		}
+		if obj.AllowUntrusted != res.AllowUntrusted {
+			return false
+		}
+		if obj.AllowNonFree != res.AllowNonFree {
+			return false
+		}
+		if obj.AllowUnsupported != res.AllowUnsupported {
+			return false
+		}
+	default:
+		return false
+	}
+	return true
+}
+
+// return a list of svc names for matches like /usr/lib/systemd/system/*.service
+func ReturnSvcInFileList(fileList []string) []string {
+	result := []string{}
+	for _, x := range fileList {
+		dirname, basename := path.Split(path.Clean(x))
+		// TODO: do we also want to look for /etc/systemd/system/ ?
+		if dirname != "/usr/lib/systemd/system/" {
+			continue
+		}
+		if !strings.HasSuffix(basename, ".service") {
+			continue
+		}
+		if s := strings.TrimSuffix(basename, ".service"); !StrInList(s, result) {
+			result = append(result, s)
+		}
+	}
+	return result
+}

puppet.go

@@ -0,0 +1,141 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"bufio"
+	"io"
+	"log"
+	"os/exec"
+	"strconv"
+	"strings"
+)
+
+const (
+	// PuppetYAMLBufferSize is the maximum buffer size for the yaml input data
+	PuppetYAMLBufferSize = 65535
+)
+
+func runPuppetCommand(cmd *exec.Cmd) ([]byte, error) {
+	if DEBUG {
+		log.Printf("Puppet: running command: %v", cmd)
+	}
+
+	stdout, err := cmd.StdoutPipe()
+	if err != nil {
+		log.Printf("Puppet: Error opening pipe to puppet command: %v", err)
+		return nil, err
+	}
+	stderr, err := cmd.StderrPipe()
+	if err != nil {
+		log.Printf("Puppet: Error opening error pipe to puppet command: %v", err)
+		return nil, err
+	}
+
+	if err := cmd.Start(); err != nil {
+		log.Printf("Puppet: Error starting puppet command: %v", err)
+		return nil, err
+	}
+
+	// XXX: the current implementation is likely prone to fail
+	// as soon as the YAML data overflows the buffer.
+	data := make([]byte, PuppetYAMLBufferSize)
+	var result []byte
+	for err == nil {
+		var count int
+		count, err = stdout.Read(data)
+		if err != nil && err != io.EOF {
+			log.Printf("Puppet: Error reading YAML data from puppet: %v", err)
+			return nil, err
+		}
+		// Slicing down to the number of actual bytes is important, the YAML parser
+		// will choke on an oversized slice. http://stackoverflow.com/a/33726617/3356612
+		result = append(result, data[0:count]...)
+	}
+	if DEBUG {
+		log.Printf("Puppet: read %v bytes of data from puppet", len(result))
+	}
+	for scanner := bufio.NewScanner(stderr); scanner.Scan(); {
+		log.Printf("Puppet: (output) %v", scanner.Text())
+	}
+	if err := cmd.Wait(); err != nil {
+		log.Printf("Puppet: Error: puppet command did not complete: %v", err)
+		return nil, err
+	}
+
+	return result, nil
+}
+
+// ParseConfigFromPuppet takes a special puppet param string and config and
+// returns the graph configuration structure.
+func ParseConfigFromPuppet(puppetParam, puppetConf string) *GraphConfig {
+	var puppetConfArg string
+	if puppetConf != "" {
+		puppetConfArg = "--config=" + puppetConf
+	}
+
+	var cmd *exec.Cmd
+	if puppetParam == "agent" {
+		cmd = exec.Command("puppet", "mgmtgraph", "print", puppetConfArg)
+	} else if strings.HasSuffix(puppetParam, ".pp") {
+		cmd = exec.Command("puppet", "mgmtgraph", "print", puppetConfArg, "--manifest", puppetParam)
+	} else {
+		cmd = exec.Command("puppet", "mgmtgraph", "print", puppetConfArg, "--code", puppetParam)
+	}
+
+	log.Println("Puppet: launching translator")
+
+	var config GraphConfig
+	if data, err := runPuppetCommand(cmd); err != nil {
+		return nil
+	} else if err := config.Parse(data); err != nil {
+		log.Printf("Puppet: Error: Could not parse YAML output with Parse: %v", err)
+		return nil
+	}
+
+	return &config
+}
+
+// PuppetInterval returns the graph refresh interval from the puppet configuration.
+func PuppetInterval(puppetConf string) int {
+	if DEBUG {
+		log.Printf("Puppet: determining graph refresh interval")
+	}
+	var cmd *exec.Cmd
+	if puppetConf != "" {
+		cmd = exec.Command("puppet", "config", "print", "runinterval", "--config", puppetConf)
+	} else {
+		cmd = exec.Command("puppet", "config", "print", "runinterval")
+	}
+
+	log.Println("Puppet: inspecting runinterval configuration")
+
+	interval := 1800
+	data, err := runPuppetCommand(cmd)
+	if err != nil {
+		log.Printf("Puppet: could not determine configured run interval (%v), using default of %v", err, interval)
+		return interval
+	}
+	result, err := strconv.ParseInt(strings.TrimSpace(string(data)), 10, 0)
+	if err != nil {
+		log.Printf("Puppet: error reading numeric runinterval value (%v), using default of %v", err, interval)
+		return interval
+	}
+
+	return int(result)
+}

resources.go

@@ -0,0 +1,366 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"bytes"
+	"encoding/base64"
+	"encoding/gob"
+	"fmt"
+	"log"
+)
+
+//go:generate stringer -type=resState -output=resstate_stringer.go
+type resState int
+
+const (
+	resStateNil resState = iota
+	resStateWatching
+	resStateEvent // an event has happened, but we haven't poked yet
+	resStateCheckApply
+	resStatePoking
+)
+
+// ResUUID is a unique identifier for a resource, namely it's name, and the kind ("type").
+type ResUUID interface {
+	GetName() string
+	Kind() string
+	IFF(ResUUID) bool
+
+	Reversed() bool // true means this resource happens before the generator
+}
+
+// The BaseUUID struct is used to provide a unique resource identifier.
+type BaseUUID struct {
+	name string // name and kind are the values of where this is coming from
+	kind string
+
+	reversed *bool // piggyback edge information here
+}
+
+// The AutoEdge interface is used to implement the autoedges feature.
+type AutoEdge interface {
+	Next() []ResUUID  // call to get list of edges to add
+	Test([]bool) bool // call until false
+}
+
+// MetaParams is a struct will all params that apply to every resource.
+type MetaParams struct {
+	AutoEdge  bool `yaml:"autoedge"`  // metaparam, should we generate auto edges? // XXX should default to true
+	AutoGroup bool `yaml:"autogroup"` // metaparam, should we auto group? // XXX should default to true
+	Noop      bool `yaml:"noop"`
+}
+
+// The Base interface is everything that is common to all resources.
+// Everything here only needs to be implemented once, in the BaseRes.
+type Base interface {
+	GetName() string // can't be named "Name()" because of struct field
+	SetName(string)
+	setKind(string)
+	Kind() string
+	Meta() *MetaParams
+	AssociateData(Converger)
+	IsWatching() bool
+	SetWatching(bool)
+	GetState() resState
+	SetState(resState)
+	SendEvent(eventName, bool, bool) bool
+	ReadEvent(*Event) (bool, bool) // TODO: optional here?
+	GroupCmp(Res) bool             // TODO: is there a better name for this?
+	GroupRes(Res) error            // group resource (arg) into self
+	IsGrouped() bool               // am I grouped?
+	SetGrouped(bool)               // set grouped bool
+	GetGroup() []Res               // return everyone grouped inside me
+	SetGroup([]Res)
+}
+
+// Res is the minimum interface you need to implement to define a new resource.
+type Res interface {
+	Base // include everything from the Base interface
+	Init()
+	//Validate() bool    // TODO: this might one day be added
+	GetUUIDs() []ResUUID // most resources only return one
+	Watch(chan Event)    // send on channel to signal process() events
+	CheckApply(bool) (bool, error)
+	AutoEdges() AutoEdge
+	Compare(Res) bool
+	CollectPattern(string) // XXX: temporary until Res collection is more advanced
+}
+
+// BaseRes is the base struct that gets used in every resource.
+type BaseRes struct {
+	Name       string     `yaml:"name"`
+	MetaParams MetaParams `yaml:"meta"` // struct of all the metaparams
+	kind       string
+	events     chan Event
+	converger  Converger // converged tracking
+	state      resState
+	watching   bool  // is Watch() loop running ?
+	isStateOK  bool  // whether the state is okay based on events or not
+	isGrouped  bool  // am i contained within a group?
+	grouped    []Res // list of any grouped resources
+}
+
+// UUIDExistsInUUIDs wraps the IFF method when used with a list of UUID's.
+func UUIDExistsInUUIDs(uuid ResUUID, uuids []ResUUID) bool {
+	for _, u := range uuids {
+		if uuid.IFF(u) {
+			return true
+		}
+	}
+	return false
+}
+
+// GetName returns the name of the resource.
+func (obj *BaseUUID) GetName() string {
+	return obj.name
+}
+
+// Kind returns the kind of resource.
+func (obj *BaseUUID) Kind() string {
+	return obj.kind
+}
+
+// IFF looks at two UUID's and if and only if they are equivalent, returns true.
+// If they are not equivalent, it returns false.
+// Most resources will want to override this method, since it does the important
+// work of actually discerning if two resources are identical in function.
+func (obj *BaseUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*BaseUUID)
+	if !ok {
+		return false
+	}
+	return obj.name == res.name
+}
+
+// Reversed is part of the ResUUID interface, and true means this resource
+// happens before the generator.
+func (obj *BaseUUID) Reversed() bool {
+	if obj.reversed == nil {
+		log.Fatal("Programming error!")
+	}
+	return *obj.reversed
+}
+
+// Init initializes structures like channels if created without New constructor.
+func (obj *BaseRes) Init() {
+	obj.events = make(chan Event) // unbuffered chan size to avoid stale events
+}
+
+// GetName is used by all the resources to Get the name.
+func (obj *BaseRes) GetName() string {
+	return obj.Name
+}
+
+// SetName is used to set the name of the resource.
+func (obj *BaseRes) SetName(name string) {
+	obj.Name = name
+}
+
+// setKind sets the kind. This is used internally for exported resources.
+func (obj *BaseRes) setKind(kind string) {
+	obj.kind = kind
+}
+
+// Kind returns the kind of resource this is.
+func (obj *BaseRes) Kind() string {
+	return obj.kind
+}
+
+// Meta returns the MetaParams as a reference, which we can then get/set on.
+func (obj *BaseRes) Meta() *MetaParams {
+	return &obj.MetaParams
+}
+
+// AssociateData associates some data with the object in question.
+func (obj *BaseRes) AssociateData(converger Converger) {
+	obj.converger = converger
+}
+
+// IsWatching tells us if the Watch() function is running.
+func (obj *BaseRes) IsWatching() bool {
+	return obj.watching
+}
+
+// SetWatching stores the status of if the Watch() function is running.
+func (obj *BaseRes) SetWatching(b bool) {
+	obj.watching = b
+}
+
+// GetState returns the state of the resource.
+func (obj *BaseRes) GetState() resState {
+	return obj.state
+}
+
+// SetState sets the state of the resource.
+func (obj *BaseRes) SetState(state resState) {
+	if DEBUG {
+		log.Printf("%v[%v]: State: %v -> %v", obj.Kind(), obj.GetName(), obj.GetState(), state)
+	}
+	obj.state = state
+}
+
+// SendEvent pushes an event into the message queue for a particular vertex
+func (obj *BaseRes) SendEvent(event eventName, sync bool, activity bool) bool {
+	// TODO: isn't this race-y ?
+	if !obj.IsWatching() { // element has already exited
+		return false // if we don't return, we'll block on the send
+	}
+	if !sync {
+		obj.events <- Event{event, nil, "", activity}
+		return true
+	}
+
+	resp := make(chan bool)
+	obj.events <- Event{event, resp, "", activity}
+	for {
+		value := <-resp
+		// wait until true value
+		if value {
+			return true
+		}
+	}
+}
+
+// ReadEvent processes events when a select gets one, and handles the pause
+// code too! The return values specify if we should exit and poke respectively.
+func (obj *BaseRes) ReadEvent(event *Event) (exit, poke bool) {
+	event.ACK()
+	switch event.Name {
+	case eventStart:
+		return false, true
+
+	case eventPoke:
+		return false, true
+
+	case eventBackPoke:
+		return false, true // forward poking in response to a back poke!
+
+	case eventExit:
+		return true, false
+
+	case eventPause:
+		// wait for next event to continue
+		select {
+		case e := <-obj.events:
+			e.ACK()
+			if e.Name == eventExit {
+				return true, false
+			} else if e.Name == eventStart { // eventContinue
+				return false, false // don't poke on unpause!
+			} else {
+				// if we get a poke event here, it's a bug!
+				log.Fatalf("%v[%v]: Unknown event: %v, while paused!", obj.Kind(), obj.GetName(), e)
+			}
+		}
+
+	default:
+		log.Fatal("Unknown event: ", event)
+	}
+	return true, false // required to keep the stupid go compiler happy
+}
+
+// GroupCmp compares two resources and decides if they're suitable for grouping
+// You'll probably want to override this method when implementing a resource...
+func (obj *BaseRes) GroupCmp(res Res) bool {
+	return false // base implementation assumes false, override me!
+}
+
+// GroupRes groups resource (arg) into self.
+func (obj *BaseRes) GroupRes(res Res) error {
+	if l := len(res.GetGroup()); l > 0 {
+		return fmt.Errorf("Res: %v already contains %d grouped resources!", res, l)
+	}
+	if res.IsGrouped() {
+		return fmt.Errorf("Res: %v is already grouped!", res)
+	}
+
+	obj.grouped = append(obj.grouped, res)
+	res.SetGrouped(true) // i am contained _in_ a group
+	return nil
+}
+
+// IsGrouped determines if we are grouped.
+func (obj *BaseRes) IsGrouped() bool { // am I grouped?
+	return obj.isGrouped
+}
+
+// SetGrouped sets a flag to tell if we are grouped.
+func (obj *BaseRes) SetGrouped(b bool) {
+	obj.isGrouped = b
+}
+
+// GetGroup returns everyone grouped inside me.
+func (obj *BaseRes) GetGroup() []Res { // return everyone grouped inside me
+	return obj.grouped
+}
+
+// SetGroup sets the grouped resources into me.
+func (obj *BaseRes) SetGroup(g []Res) {
+	obj.grouped = g
+}
+
+// Compare is the base compare method, which also handles the metaparams cmp
+func (obj *BaseRes) Compare(res Res) bool {
+	if obj.Meta().Noop != res.Meta().Noop {
+		// obj is the existing res, res is the *new* resource
+		// if we go from no-noop -> noop, we can re-use the obj
+		// if we go from noop -> no-noop, we need to regenerate
+		if obj.Meta().Noop { // asymmetrical
+			return false // going from noop to no-noop!
+		}
+	}
+	return true
+}
+
+// CollectPattern is used for resource collection.
+func (obj *BaseRes) CollectPattern(pattern string) {
+	// XXX: default method is empty
+}
+
+// ResToB64 encodes a resource to a base64 encoded string (after serialization)
+func ResToB64(res Res) (string, error) {
+	b := bytes.Buffer{}
+	e := gob.NewEncoder(&b)
+	err := e.Encode(&res) // pass with &
+	if err != nil {
+		return "", fmt.Errorf("Gob failed to encode: %v", err)
+	}
+	return base64.StdEncoding.EncodeToString(b.Bytes()), nil
+}
+
+// B64ToRes decodes a resource from a base64 encoded string (after deserialization)
+func B64ToRes(str string) (Res, error) {
+	var output interface{}
+	bb, err := base64.StdEncoding.DecodeString(str)
+	if err != nil {
+		return nil, fmt.Errorf("Base64 failed to decode: %v", err)
+	}
+	b := bytes.NewBuffer(bb)
+	d := gob.NewDecoder(b)
+	err = d.Decode(&output) // pass with &
+	if err != nil {
+		return nil, fmt.Errorf("Gob failed to decode: %v", err)
+	}
+	res, ok := output.(Res)
+	if !ok {
+		return nil, fmt.Errorf("Output %v is not a Res", res)
+
+	}
+	return res, nil
+}

resources_test.go

@@ -0,0 +1,173 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"bytes"
+	"encoding/base64"
+	"encoding/gob"
+	"testing"
+)
+
+func TestMiscEncodeDecode1(t *testing.T) {
+	var err error
+	//gob.Register( &NoopRes{} ) // happens in noop.go : init()
+	//gob.Register( &FileRes{} ) // happens in file.go : init()
+	// ...
+
+	// encode
+	var input interface{} = &FileRes{}
+	b1 := bytes.Buffer{}
+	e := gob.NewEncoder(&b1)
+	err = e.Encode(&input) // pass with &
+	if err != nil {
+		t.Errorf("Gob failed to Encode: %v", err)
+	}
+	str := base64.StdEncoding.EncodeToString(b1.Bytes())
+
+	// decode
+	var output interface{}
+	bb, err := base64.StdEncoding.DecodeString(str)
+	if err != nil {
+		t.Errorf("Base64 failed to Decode: %v", err)
+	}
+	b2 := bytes.NewBuffer(bb)
+	d := gob.NewDecoder(b2)
+	err = d.Decode(&output) // pass with &
+	if err != nil {
+		t.Errorf("Gob failed to Decode: %v", err)
+	}
+
+	res1, ok := input.(Res)
+	if !ok {
+		t.Errorf("Input %v is not a Res", res1)
+		return
+	}
+	res2, ok := output.(Res)
+	if !ok {
+		t.Errorf("Output %v is not a Res", res2)
+		return
+	}
+	if !res1.Compare(res2) {
+		t.Error("The input and output Res values do not match!")
+	}
+}
+
+func TestMiscEncodeDecode2(t *testing.T) {
+	var err error
+	//gob.Register( &NoopRes{} ) // happens in noop.go : init()
+	//gob.Register( &FileRes{} ) // happens in file.go : init()
+	// ...
+
+	// encode
+	var input Res = &FileRes{}
+
+	b64, err := ResToB64(input)
+	if err != nil {
+		t.Errorf("Can't encode: %v", err)
+		return
+	}
+
+	output, err := B64ToRes(b64)
+	if err != nil {
+		t.Errorf("Can't decode: %v", err)
+		return
+	}
+
+	res1, ok := input.(Res)
+	if !ok {
+		t.Errorf("Input %v is not a Res", res1)
+		return
+	}
+	res2, ok := output.(Res)
+	if !ok {
+		t.Errorf("Output %v is not a Res", res2)
+		return
+	}
+	if !res1.Compare(res2) {
+		t.Error("The input and output Res values do not match!")
+	}
+}
+
+func TestIFF(t *testing.T) {
+	uuid := &BaseUUID{name: "/tmp/unit-test"}
+	same := &BaseUUID{name: "/tmp/unit-test"}
+	diff := &BaseUUID{name: "/tmp/other-file"}
+
+	if !uuid.IFF(same) {
+		t.Error("basic resource UUIDs with the same name should satisfy each other's IFF condition.")
+	}
+
+	if uuid.IFF(diff) {
+		t.Error("basic resource UUIDs with different names should NOT satisfy each other's IFF condition.")
+	}
+}
+
+func TestReadEvent(t *testing.T) {
+	res := FileRes{}
+
+	shouldExit := map[eventName]bool{
+		eventStart:    false,
+		eventPoke:     false,
+		eventBackPoke: false,
+		eventExit:     true,
+	}
+	shouldPoke := map[eventName]bool{
+		eventStart:    true,
+		eventPoke:     true,
+		eventBackPoke: true,
+		eventExit:     false,
+	}
+
+	for event, _ := range shouldExit {
+		exit, poke := res.ReadEvent(&Event{Name: event})
+		if exit != shouldExit[event] {
+			t.Errorf("resource.ReadEvent returned wrong exit flag for a %v event (%v, should be %v)",
+				event, exit, shouldExit[event])
+		}
+		if poke != shouldPoke[event] {
+			t.Errorf("resource.ReadEvent returned wrong poke flag for a %v event (%v, should be %v)",
+				event, poke, shouldPoke[event])
+		}
+	}
+
+	res.Init()
+	res.SetWatching(true)
+
+	// test result when a pause event is followed by start
+	go res.SendEvent(eventStart, false, false)
+	exit, poke := res.ReadEvent(&Event{Name: eventPause})
+	if exit {
+		t.Error("resource.ReadEvent returned wrong exit flag for a pause+start event (true, should be false)")
+	}
+	if poke {
+		t.Error("resource.ReadEvent returned wrong poke flag for a pause+start event (true, should be false)")
+	}
+
+	// test result when a pause event is followed by exit
+	go res.SendEvent(eventExit, false, false)
+	exit, poke = res.ReadEvent(&Event{Name: eventPause})
+	if !exit {
+		t.Error("resource.ReadEvent returned wrong exit flag for a pause+start event (false, should be true)")
+	}
+	if poke {
+		t.Error("resource.ReadEvent returned wrong poke flag for a pause+start event (true, should be false)")
+	}
+
+	// TODO: create a wrapper API around log, so that Fatals can be mocked and tested
+}

service.go

@@ -1,294 +0,0 @@
-// Mgmt
-// Copyright (C) 2013-2015+ James Shubin and the project contributors
-// Written by James Shubin <james@shubin.ca> and the project contributors
-//
-// This program is free software: you can redistribute it and/or modify
-// it under the terms of the GNU Affero General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-//
-// This program is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU Affero General Public License for more details.
-//
-// You should have received a copy of the GNU Affero General Public License
-// along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-// NOTE: docs are found at: https://godoc.org/github.com/coreos/go-systemd/dbus
-
-package main
-
-import (
-	"code.google.com/p/go-uuid/uuid"
-	"fmt"
-	systemd "github.com/coreos/go-systemd/dbus" // change namespace
-	"github.com/coreos/go-systemd/util"
-	"github.com/godbus/dbus" // namespace collides with systemd wrapper
-	"log"
-)
-
-type ServiceType struct {
-	uuid    string
-	Type    string      // always "service"
-	Name    string      // name variable
-	Events  chan string // FIXME: eventually a struct for the event?
-	State   string      // state: running, stopped
-	Startup string      // enabled, disabled, undefined
-}
-
-func NewServiceType(name, state, startup string) *ServiceType {
-	return &ServiceType{
-		uuid:    uuid.New(),
-		Type:    "service",
-		Name:    name,
-		Events:  make(chan string, 1), // XXX: chan size?
-		State:   state,
-		Startup: startup,
-	}
-}
-
-// Service watcher
-func (obj ServiceType) Watch(v *Vertex) {
-	// obj.Name: service name
-
-	if !util.IsRunningSystemd() {
-		log.Fatal("Systemd is not running.")
-	}
-
-	conn, err := systemd.NewSystemdConnection() // needs root access
-	if err != nil {
-		log.Fatal("Failed to connect to systemd: ", err)
-	}
-	defer conn.Close()
-
-	bus, err := dbus.SystemBus()
-	if err != nil {
-		log.Fatal("Failed to connect to bus: %v\n", err)
-	}
-
-	// XXX: will this detect new units?
-	bus.BusObject().Call("org.freedesktop.DBus.AddMatch", 0,
-		"type='signal',interface='org.freedesktop.systemd1.Manager',member='Reloading'")
-	buschan := make(chan *dbus.Signal, 10)
-	bus.Signal(buschan)
-
-	var service = fmt.Sprintf("%v.service", obj.Name) // systemd name
-	var send = false                                  // send event?
-	var invalid = false                               // does the service exist or not?
-	var previous bool                                 // previous invalid value
-	set := conn.NewSubscriptionSet()                  // no error should be returned
-	subChannel, subErrors := set.Subscribe()
-	var activeSet = false
-
-	for {
-		// XXX: watch for an event for new units...
-		// XXX: detect if startup enabled/disabled value changes...
-
-		previous = invalid
-		invalid = false
-
-		// firstly, does service even exist or not?
-		loadstate, err := conn.GetUnitProperty(service, "LoadState")
-		if err != nil {
-			log.Printf("Failed to get property: %v\n", err)
-			invalid = true
-		}
-
-		if !invalid {
-			var notFound = (loadstate.Value == dbus.MakeVariant("not-found"))
-			if notFound { // XXX: in the loop we'll handle changes better...
-				log.Printf("Failed to find service: %v\n", service)
-				invalid = true // XXX ?
-			}
-		}
-
-		if previous != invalid { // if invalid changed, send signal
-			send = true
-		}
-
-		if invalid {
-			log.Printf("Waiting for: %v\n", service) // waiting for service to appear...
-			if activeSet {
-				activeSet = false
-				set.Remove(service) // no return value should ever occur
-			}
-
-			select {
-			case _ = <-buschan: // XXX wait for new units event to unstick
-				// loop so that we can see the changed invalid signal
-				log.Printf("Service[%v]->DaemonReload()\n", service)
-
-			case exit := <-obj.Events:
-				if exit == "exit" {
-					return
-				} else {
-					log.Fatal("Unknown event: %v\n", exit)
-				}
-			}
-		} else {
-			if !activeSet {
-				activeSet = true
-				set.Add(service) // no return value should ever occur
-			}
-
-			log.Printf("Watching: %v\n", service) // attempting to watch...
-			select {
-			case event := <-subChannel:
-
-				log.Printf("Service event: %+v\n", event)
-				// NOTE: the value returned is a map for some reason...
-				if event[service] != nil {
-					// event[service].ActiveState is not nil
-					if event[service].ActiveState == "active" {
-						log.Printf("Service[%v]->Started()\n", service)
-					} else if event[service].ActiveState == "inactive" {
-						log.Printf("Service[%v]->Stopped!()\n", service)
-					} else {
-						log.Fatal("Unknown service state: ", event[service].ActiveState)
-					}
-				} else {
-					// service stopped (and ActiveState is nil...)
-					log.Printf("Service[%v]->Stopped\n", service)
-				}
-				send = true
-
-			case err := <-subErrors:
-				log.Println("error:", err)
-				log.Fatal(err)
-				v.Events <- fmt.Sprintf("service: %v", "error")
-
-			case exit := <-obj.Events:
-				if exit == "exit" {
-					return
-				} else {
-					log.Fatal("Unknown event: %v\n", exit)
-				}
-			}
-		}
-
-		if send {
-			send = false
-			//log.Println("Sending event!")
-			v.Events <- fmt.Sprintf("service(%v): %v", obj.Name, "event!") // FIXME: use struct
-		}
-	}
-}
-
-func (obj ServiceType) Exit() bool {
-	obj.Events <- "exit"
-	return true
-}
-
-func (obj ServiceType) StateOK() bool {
-
-	if !util.IsRunningSystemd() {
-		log.Fatal("Systemd is not running.")
-	}
-
-	conn, err := systemd.NewSystemdConnection() // needs root access
-	if err != nil {
-		log.Fatal("Failed to connect to systemd: ", err)
-	}
-	defer conn.Close()
-
-	var service = fmt.Sprintf("%v.service", obj.Name) // systemd name
-
-	loadstate, err := conn.GetUnitProperty(service, "LoadState")
-	if err != nil {
-		log.Printf("Failed to get load state: %v\n", err)
-		return false
-	}
-
-	// NOTE: we have to compare variants with other variants, they are really strings...
-	var notFound = (loadstate.Value == dbus.MakeVariant("not-found"))
-	if notFound {
-		log.Printf("Failed to find service: %v\n", service)
-		return false
-	}
-
-	// XXX: check service "enabled at boot" or not status...
-
-	//conn.GetUnitProperties(service)
-	activestate, err := conn.GetUnitProperty(service, "ActiveState")
-	if err != nil {
-		log.Fatal("Failed to get active state: ", err)
-	}
-
-	var running = (activestate.Value == dbus.MakeVariant("active"))
-
-	if obj.State == "running" {
-		if !running {
-			return false // we are in the wrong state
-		}
-	} else if obj.State == "stopped" {
-		if running {
-			return false
-		}
-	} else {
-		log.Fatal("Unknown state: ", obj.State)
-	}
-
-	return true // all is good, no state change needed
-}
-
-func (obj ServiceType) Apply() bool {
-	fmt.Printf("Apply->%v[%v]\n", obj.Type, obj.Name)
-
-	if !util.IsRunningSystemd() {
-		log.Fatal("Systemd is not running.")
-	}
-
-	conn, err := systemd.NewSystemdConnection() // needs root access
-	if err != nil {
-		log.Fatal("Failed to connect to systemd: ", err)
-	}
-	defer conn.Close()
-
-	var service = fmt.Sprintf("%v.service", obj.Name) // systemd name
-	var files = []string{service}                     // the service represented in a list
-	if obj.Startup == "enabled" {
-		_, _, err = conn.EnableUnitFiles(files, false, true)
-
-	} else if obj.Startup == "disabled" {
-		_, err = conn.DisableUnitFiles(files, false)
-	} else {
-		err = nil
-	}
-	if err != nil {
-		log.Printf("Unable to change startup status: %v\n", err)
-		return false
-	}
-
-	result := make(chan string, 1) // catch result information
-
-	if obj.State == "running" {
-		_, err := conn.StartUnit(service, "fail", result)
-		if err != nil {
-			log.Fatal("Failed to start unit: ", err)
-			return false
-		}
-	} else if obj.State == "stopped" {
-		_, err = conn.StopUnit(service, "fail", result)
-		if err != nil {
-			log.Fatal("Failed to stop unit: ", err)
-			return false
-		}
-	} else {
-		log.Fatal("Unknown state: ", obj.State)
-	}
-
-	status := <-result
-	if &status == nil {
-		log.Fatal("Result is nil")
-		return false
-	}
-	if status != "done" {
-		log.Fatal("Unknown return string: ", status)
-		return false
-	}
-
-	// XXX: also set enabled on boot
-
-	return true
-}

spec.in

@@ -0,0 +1,72 @@
+%global project_version __VERSION__
+%define debug_package %{nil}
+
+Name:		__PROGRAM__
+Version:	__VERSION__
+Release:	__RELEASE__
+Summary:	A next generation config management prototype!
+License:	AGPLv3+
+URL:		https://github.com/purpleidea/mgmt
+Source0:	https://dl.fedoraproject.org/pub/alt/purpleidea/__PROGRAM__/SOURCES/__PROGRAM__-%{project_version}.tar.bz2
+
+# graphviz should really be a "suggests", since technically it's optional
+Requires: graphviz
+
+BuildRequires:	golang
+BuildRequires:	golang-googlecode-tools-stringer
+BuildRequires:	git-core
+BuildRequires:	mercurial
+
+%description
+A next generation config management prototype!
+
+%prep
+%setup
+
+%build
+# FIXME: in the future, these could be vendor-ed in
+mkdir -p vendor/
+export GOPATH=`pwd`/vendor/
+go get github.com/coreos/etcd/client
+go get gopkg.in/yaml.v2
+go get gopkg.in/fsnotify.v1
+go get github.com/codegangsta/cli
+go get github.com/coreos/go-systemd/dbus
+go get github.com/coreos/go-systemd/util
+make build
+
+%install
+rm -rf %{buildroot}
+mkdir -p %{buildroot}/%{_unitdir}/
+install -pm 0644 misc/__PROGRAM__.service %{buildroot}/%{_unitdir}/
+
+# install the binary
+mkdir -p %{buildroot}/%{_bindir}
+install -m 0755 __PROGRAM__ %{buildroot}/%{_bindir}/__PROGRAM__
+
+# profile.d bash completion
+mkdir -p %{buildroot}%{_sysconfdir}/profile.d
+install misc/bashrc.sh -m 0755 %{buildroot}%{_sysconfdir}/profile.d/__PROGRAM__.sh
+
+# etc dir
+mkdir -p %{buildroot}%{_sysconfdir}/__PROGRAM__/
+install -m 0644 misc/example.conf %{buildroot}%{_sysconfdir}/__PROGRAM__/__PROGRAM__.conf
+
+%files
+%attr(0755, root, root) %{_sysconfdir}/profile.d/__PROGRAM__.sh
+%{_bindir}/__PROGRAM__
+%{_sysconfdir}/__PROGRAM__/*
+%{_unitdir}/__PROGRAM__.service
+
+# https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#Documentation
+# Please add docs one per line in alpha order to avoid diff churn.
+%doc AUTHORS
+%doc COPYING
+%doc COPYRIGHT
+%doc DOCUMENTATION.md
+%doc README.md
+%doc THANKS
+%doc examples/*
+
+# this changelog is auto-generated by git log
+%changelog

svc.go

@@ -0,0 +1,457 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+// DOCS: https://godoc.org/github.com/coreos/go-systemd/dbus
+
+package main
+
+import (
+	"encoding/gob"
+	"errors"
+	"fmt"
+	systemd "github.com/coreos/go-systemd/dbus" // change namespace
+	systemdUtil "github.com/coreos/go-systemd/util"
+	"github.com/godbus/dbus" // namespace collides with systemd wrapper
+	"log"
+)
+
+func init() {
+	gob.Register(&SvcRes{})
+}
+
+// SvcRes is a service resource for systemd units.
+type SvcRes struct {
+	BaseRes `yaml:",inline"`
+	State   string `yaml:"state"`   // state: running, stopped, undefined
+	Startup string `yaml:"startup"` // enabled, disabled, undefined
+}
+
+// NewSvcRes is a constructor for this resource. It also calls Init() for you.
+func NewSvcRes(name, state, startup string) *SvcRes {
+	obj := &SvcRes{
+		BaseRes: BaseRes{
+			Name: name,
+		},
+		State:   state,
+		Startup: startup,
+	}
+	obj.Init()
+	return obj
+}
+
+// Init runs some startup code for this resource.
+func (obj *SvcRes) Init() {
+	obj.BaseRes.kind = "Svc"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+}
+
+// Validate checks if the resource data structure was populated correctly.
+func (obj *SvcRes) Validate() bool {
+	if obj.State != "running" && obj.State != "stopped" && obj.State != "" {
+		return false
+	}
+	if obj.Startup != "enabled" && obj.Startup != "disabled" && obj.Startup != "" {
+		return false
+	}
+	return true
+}
+
+// Watch is the primary listener for this resource and it outputs events.
+func (obj *SvcRes) Watch(processChan chan Event) {
+	if obj.IsWatching() {
+		return
+	}
+	obj.SetWatching(true)
+	defer obj.SetWatching(false)
+	cuuid := obj.converger.Register()
+	defer cuuid.Unregister()
+
+	// obj.Name: svc name
+	if !systemdUtil.IsRunningSystemd() {
+		log.Fatal("Systemd is not running.")
+	}
+
+	conn, err := systemd.NewSystemdConnection() // needs root access
+	if err != nil {
+		log.Fatal("Failed to connect to systemd: ", err)
+	}
+	defer conn.Close()
+
+	// if we share the bus with others, we will get each others messages!!
+	bus, err := SystemBusPrivateUsable() // don't share the bus connection!
+	if err != nil {
+		log.Fatal("Failed to connect to bus: ", err)
+	}
+
+	// XXX: will this detect new units?
+	bus.BusObject().Call("org.freedesktop.DBus.AddMatch", 0,
+		"type='signal',interface='org.freedesktop.systemd1.Manager',member='Reloading'")
+	buschan := make(chan *dbus.Signal, 10)
+	bus.Signal(buschan)
+
+	var svc = fmt.Sprintf("%v.service", obj.Name) // systemd name
+	var send = false                              // send event?
+	var exit = false
+	var dirty = false
+	var invalid = false              // does the svc exist or not?
+	var previous bool                // previous invalid value
+	set := conn.NewSubscriptionSet() // no error should be returned
+	subChannel, subErrors := set.Subscribe()
+	var activeSet = false
+
+	for {
+		// XXX: watch for an event for new units...
+		// XXX: detect if startup enabled/disabled value changes...
+
+		previous = invalid
+		invalid = false
+
+		// firstly, does svc even exist or not?
+		loadstate, err := conn.GetUnitProperty(svc, "LoadState")
+		if err != nil {
+			log.Printf("Failed to get property: %v", err)
+			invalid = true
+		}
+
+		if !invalid {
+			var notFound = (loadstate.Value == dbus.MakeVariant("not-found"))
+			if notFound { // XXX: in the loop we'll handle changes better...
+				log.Printf("Failed to find svc: %v", svc)
+				invalid = true // XXX ?
+			}
+		}
+
+		if previous != invalid { // if invalid changed, send signal
+			send = true
+			dirty = true
+		}
+
+		if invalid {
+			log.Printf("Waiting for: %v", svc) // waiting for svc to appear...
+			if activeSet {
+				activeSet = false
+				set.Remove(svc) // no return value should ever occur
+			}
+
+			obj.SetState(resStateWatching) // reset
+			select {
+			case <-buschan: // XXX wait for new units event to unstick
+				cuuid.SetConverged(false)
+				// loop so that we can see the changed invalid signal
+				log.Printf("Svc[%v]->DaemonReload()", svc)
+
+			case event := <-obj.events:
+				cuuid.SetConverged(false)
+				if exit, send = obj.ReadEvent(&event); exit {
+					return // exit
+				}
+				if event.GetActivity() {
+					dirty = true
+				}
+
+			case <-cuuid.ConvergedTimer():
+				cuuid.SetConverged(true) // converged!
+				continue
+			}
+		} else {
+			if !activeSet {
+				activeSet = true
+				set.Add(svc) // no return value should ever occur
+			}
+
+			log.Printf("Watching: %v", svc) // attempting to watch...
+			obj.SetState(resStateWatching)  // reset
+			select {
+			case event := <-subChannel:
+
+				log.Printf("Svc event: %+v", event)
+				// NOTE: the value returned is a map for some reason...
+				if event[svc] != nil {
+					// event[svc].ActiveState is not nil
+					if event[svc].ActiveState == "active" {
+						log.Printf("Svc[%v]->Started()", svc)
+					} else if event[svc].ActiveState == "inactive" {
+						log.Printf("Svc[%v]->Stopped!()", svc)
+					} else {
+						log.Fatal("Unknown svc state: ", event[svc].ActiveState)
+					}
+				} else {
+					// svc stopped (and ActiveState is nil...)
+					log.Printf("Svc[%v]->Stopped", svc)
+				}
+				send = true
+				dirty = true
+
+			case err := <-subErrors:
+				cuuid.SetConverged(false) // XXX ?
+				log.Printf("error: %v", err)
+				log.Fatal(err)
+				//vertex.events <- fmt.Sprintf("svc: %v", "error") // XXX: how should we handle errors?
+
+			case event := <-obj.events:
+				cuuid.SetConverged(false)
+				if exit, send = obj.ReadEvent(&event); exit {
+					return // exit
+				}
+				if event.GetActivity() {
+					dirty = true
+				}
+
+			case <-cuuid.ConvergedTimer():
+				cuuid.SetConverged(true) // converged!
+				continue
+			}
+		}
+
+		if send {
+			send = false
+			if dirty {
+				dirty = false
+				obj.isStateOK = false // something made state dirty
+			}
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
+		}
+
+	}
+}
+
+// CheckApply checks the resource state and applies the resource if the bool
+// input is true. It returns error info and if the state check passed or not.
+func (obj *SvcRes) CheckApply(apply bool) (checkok bool, err error) {
+	log.Printf("%v[%v]: CheckApply(%t)", obj.Kind(), obj.GetName(), apply)
+
+	if obj.isStateOK { // cache the state
+		return true, nil
+	}
+
+	if !systemdUtil.IsRunningSystemd() {
+		return false, errors.New("Systemd is not running.")
+	}
+
+	conn, err := systemd.NewSystemdConnection() // needs root access
+	if err != nil {
+		return false, fmt.Errorf("Failed to connect to systemd: %v", err)
+	}
+	defer conn.Close()
+
+	var svc = fmt.Sprintf("%v.service", obj.Name) // systemd name
+
+	loadstate, err := conn.GetUnitProperty(svc, "LoadState")
+	if err != nil {
+		return false, fmt.Errorf("Failed to get load state: %v", err)
+	}
+
+	// NOTE: we have to compare variants with other variants, they are really strings...
+	var notFound = (loadstate.Value == dbus.MakeVariant("not-found"))
+	if notFound {
+		return false, fmt.Errorf("Failed to find svc: %v", svc)
+	}
+
+	// XXX: check svc "enabled at boot" or not status...
+
+	//conn.GetUnitProperties(svc)
+	activestate, err := conn.GetUnitProperty(svc, "ActiveState")
+	if err != nil {
+		return false, fmt.Errorf("Failed to get active state: %v", err)
+	}
+
+	var running = (activestate.Value == dbus.MakeVariant("active"))
+	var stateOK = ((obj.State == "") || (obj.State == "running" && running) || (obj.State == "stopped" && !running))
+	var startupOK = true // XXX DETECT AND SET
+
+	if stateOK && startupOK {
+		return true, nil // we are in the correct state
+	}
+
+	// state is not okay, no work done, exit, but without error
+	if !apply {
+		return false, nil
+	}
+
+	// apply portion
+	log.Printf("%v[%v]: Apply", obj.Kind(), obj.GetName())
+	var files = []string{svc} // the svc represented in a list
+	if obj.Startup == "enabled" {
+		_, _, err = conn.EnableUnitFiles(files, false, true)
+
+	} else if obj.Startup == "disabled" {
+		_, err = conn.DisableUnitFiles(files, false)
+	}
+
+	if err != nil {
+		return false, fmt.Errorf("Unable to change startup status: %v", err)
+	}
+
+	// XXX: do we need to use a buffered channel here?
+	result := make(chan string, 1) // catch result information
+
+	if obj.State == "running" {
+		_, err = conn.StartUnit(svc, "fail", result)
+		if err != nil {
+			return false, fmt.Errorf("Failed to start unit: %v", err)
+		}
+	} else if obj.State == "stopped" {
+		_, err = conn.StopUnit(svc, "fail", result)
+		if err != nil {
+			return false, fmt.Errorf("Failed to stop unit: %v", err)
+		}
+	}
+
+	status := <-result
+	if &status == nil {
+		return false, errors.New("Systemd service action result is nil")
+	}
+	if status != "done" {
+		return false, fmt.Errorf("Unknown systemd return string: %v", status)
+	}
+
+	// XXX: also set enabled on boot
+
+	return false, nil // success
+}
+
+// SvcUUID is the UUID struct for SvcRes.
+type SvcUUID struct {
+	// NOTE: there is also a name variable in the BaseUUID struct, this is
+	// information about where this UUID came from, and is unrelated to the
+	// information about the resource we're matching. That data which is
+	// used in the IFF function, is what you see in the struct fields here.
+	BaseUUID
+	name string // the svc name
+}
+
+// if and only if they are equivalent, return true
+// if they are not equivalent, return false
+func (obj *SvcUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*SvcUUID)
+	if !ok {
+		return false
+	}
+	return obj.name == res.name
+}
+
+// SvcResAutoEdges holds the state of the auto edge generator.
+type SvcResAutoEdges struct {
+	data    []ResUUID
+	pointer int
+	found   bool
+}
+
+// Next returns the next automatic edge.
+func (obj *SvcResAutoEdges) Next() []ResUUID {
+	if obj.found {
+		log.Fatal("Shouldn't be called anymore!")
+	}
+	if len(obj.data) == 0 { // check length for rare scenarios
+		return nil
+	}
+	value := obj.data[obj.pointer]
+	obj.pointer++
+	return []ResUUID{value} // we return one, even though api supports N
+}
+
+// Test gets results of the earlier Next() call, & returns if we should continue!
+func (obj *SvcResAutoEdges) Test(input []bool) bool {
+	// if there aren't any more remaining
+	if len(obj.data) <= obj.pointer {
+		return false
+	}
+	if obj.found { // already found, done!
+		return false
+	}
+	if len(input) != 1 { // in case we get given bad data
+		log.Fatal("Expecting a single value!")
+	}
+	if input[0] { // if a match is found, we're done!
+		obj.found = true // no more to find!
+		return false
+	}
+	return true // keep going
+}
+
+// The AutoEdges method returns the AutoEdges. In this case the systemd units.
+func (obj *SvcRes) AutoEdges() AutoEdge {
+	var data []ResUUID
+	svcFiles := []string{
+		fmt.Sprintf("/etc/systemd/system/%s.service", obj.Name),     // takes precedence
+		fmt.Sprintf("/usr/lib/systemd/system/%s.service", obj.Name), // pkg default
+	}
+	for _, x := range svcFiles {
+		var reversed = true
+		data = append(data, &FileUUID{
+			BaseUUID: BaseUUID{
+				name:     obj.GetName(),
+				kind:     obj.Kind(),
+				reversed: &reversed,
+			},
+			path: x, // what matters
+		})
+	}
+	return &FileResAutoEdges{
+		data:    data,
+		pointer: 0,
+		found:   false,
+	}
+}
+
+// GetUUIDs includes all params to make a unique identification of this object.
+// Most resources only return one, although some resources can return multiple.
+func (obj *SvcRes) GetUUIDs() []ResUUID {
+	x := &SvcUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		name:     obj.Name, // svc name
+	}
+	return []ResUUID{x}
+}
+
+// GroupCmp returns whether two resources can be grouped together or not.
+func (obj *SvcRes) GroupCmp(r Res) bool {
+	_, ok := r.(*SvcRes)
+	if !ok {
+		return false
+	}
+	// TODO: depending on if the systemd service api allows batching, we
+	// might be able to build this, although not sure how useful it is...
+	// it might just eliminate parallelism be bunching up the graph
+	return false // not possible atm
+}
+
+// Compare two resources and return if they are equivalent.
+func (obj *SvcRes) Compare(res Res) bool {
+	switch res.(type) {
+	case *SvcRes:
+		res := res.(*SvcRes)
+		if !obj.BaseRes.Compare(res) { // call base Compare
+			return false
+		}
+
+		if obj.Name != res.Name {
+			return false
+		}
+		if obj.State != res.State {
+			return false
+		}
+		if obj.Startup != res.Startup {
+			return false
+		}
+	default:
+		return false
+	}
+	return true
+}

tag.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+# TODO: don't run if current HEAD is already tagged (ensure this is idempotent)
+# take current HEAD with new version
+v=`git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --abbrev=0`
+t=`echo "${v%.*}.$((${v##*.}+1))"`	# increment version
+echo "Version $t is now tagged!"
+echo "Pushing $t to origin..."
+echo "Press ^C within 3s to abort."
+sleep 3s
+git tag $t
+git push origin $t

test.sh

@@ -1,11 +1,48 @@
 #!/bin/bash -e
-
 # test suite...
 echo running test.sh
+echo "ENV:"
+env
+
+failures=''
+function run-test()
+{
+	$@ || failures=$( [ -n "$failures" ] && echo "$failures\\n$@" || echo "$@" )
+}
 
 # ensure there is no trailing whitespace or other whitespace errors
-git diff-tree --check $(git hash-object -t tree /dev/null) HEAD
+run-test git diff-tree --check $(git hash-object -t tree /dev/null) HEAD
 
 # ensure entries to authors file are sorted
 start=$(($(grep -n '^[[:space:]]*$' AUTHORS | awk -F ':' '{print $1}' | head -1) + 1))
-diff <(tail -n +$start AUTHORS | sort) <(tail -n +$start AUTHORS)
+run-test diff <(tail -n +$start AUTHORS | sort) <(tail -n +$start AUTHORS)
+
+run-test ./test/test-gofmt.sh
+run-test ./test/test-yamlfmt.sh
+run-test ./test/test-bashfmt.sh
+run-test ./test/test-headerfmt.sh
+run-test go test
+run-test ./test/test-govet.sh
+
+# do these longer tests only when running on ci
+if env | grep -q -e '^TRAVIS=true$' -e '^JENKINS_URL=' -e '^BUILD_TAG=jenkins'; then
+	run-test go test -race
+	run-test ./test/test-shell.sh
+fi
+
+# FIXME: this now fails everywhere :(
+#run-test ./test/test-reproducible.sh
+
+# run omv tests on jenkins physical hosts only
+if env | grep -q -e '^JENKINS_URL=' -e '^BUILD_TAG=jenkins'; then
+	run-test ./test/test-omv.sh
+fi
+run-test ./test/test-golint.sh	# test last, because this test is somewhat arbitrary
+
+if [[ -n "$failures" ]]; then
+	echo 'FAIL'
+	echo 'The following tests have failed:'
+	echo -e "$failures"
+	exit 1
+fi
+echo 'ALL PASSED'

test/omv/helloworld.yaml

@@ -0,0 +1,50 @@
+---
+:domain: example.com
+:network: 192.168.123.0/24
+:image: centos-7.1
+:cpus: ''
+:memory: ''
+:disks: 0
+:disksize: 40G
+:boxurlprefix: ''
+:sync: rsync
+:syncdir: ''
+:syncsrc: ''
+:folder: ".omv"
+:extern:
+- type: git
+  repository: https://github.com/purpleidea/mgmt
+  directory: mgmt
+:cd: ''
+:puppet: false
+:classes: []
+:shell:
+- mkdir /tmp/mgmt/
+:docker: false
+:kubernetes: false
+:ansible: []
+:playbook: []
+:ansible_extras: {}
+:cachier: false
+:vms:
+- :name: mgmt0
+  :shell:
+  - iptables -F
+  - cd /vagrant/mgmt/ && make path
+  - cd /vagrant/mgmt/ && make deps && make build && cp mgmt ~/bin/
+  - cd && mgmt --help
+:namespace: omv
+:count: 0
+:username: ''
+:password: ''
+:poolid: true
+:repos: []
+:update: false
+:reboot: false
+:unsafe: false
+:nested: false
+:tests:
+- omv up mgmt0
+- omv destroy
+:comment: simple hello world test case for mgmt
+:reallyrm: false

test/omv/pkg1a.yaml

@@ -0,0 +1,51 @@
+---
+:domain: example.com
+:network: 192.168.123.0/24
+:image: centos-7.1
+:cpus: ''
+:memory: ''
+:disks: 0
+:disksize: 40G
+:boxurlprefix: ''
+:sync: rsync
+:syncdir: ''
+:syncsrc: ''
+:folder: ".omv"
+:extern:
+- type: git
+  repository: https://github.com/purpleidea/mgmt
+  directory: mgmt
+:cd: ''
+:puppet: false
+:classes: []
+:shell:
+- mkdir /tmp/mgmt/
+:docker: false
+:kubernetes: false
+:ansible: []
+:playbook: []
+:ansible_extras: {}
+:cachier: false
+:vms:
+- :name: mgmt1
+  :shell:
+  - iptables -F
+  - cd /vagrant/mgmt/ && make path
+  - cd /vagrant/mgmt/ && make deps && make build && cp mgmt ~/bin/
+  - cd && mgmt run --file /vagrant/mgmt/examples/pkg1.yaml --converged-timeout=5
+:namespace: omv
+:count: 0
+:username: ''
+:password: ''
+:poolid: true
+:repos: []
+:update: false
+:reboot: false
+:unsafe: false
+:nested: false
+:tests:
+- omv up
+- vssh root@mgmt1 -c which powertop
+- omv destroy
+:comment: simple package install test case
+:reallyrm: false

test/omv/pkg1b.yaml

@@ -0,0 +1,52 @@
+---
+:domain: example.com
+:network: 192.168.123.0/24
+:image: debian-8
+:cpus: ''
+:memory: ''
+:disks: 0
+:disksize: 40G
+:boxurlprefix: ''
+:sync: rsync
+:syncdir: ''
+:syncsrc: ''
+:folder: ".omv"
+:extern:
+- type: git
+  repository: https://github.com/purpleidea/mgmt
+  directory: mgmt
+:cd: ''
+:puppet: false
+:classes: []
+:shell:
+- mkdir /tmp/mgmt/
+:docker: false
+:kubernetes: false
+:ansible: []
+:playbook: []
+:ansible_extras: {}
+:cachier: false
+:vms:
+- :name: mgmt1
+  :shell:
+  - apt-get install -y make
+  - iptables -F
+  - cd /vagrant/mgmt/ && make path
+  - cd /vagrant/mgmt/ && make deps && make build && cp mgmt ~/bin/
+  - cd && mgmt run --file /vagrant/mgmt/examples/pkg1.yaml --converged-timeout=5
+:namespace: omv
+:count: 0
+:username: ''
+:password: ''
+:poolid: true
+:repos: []
+:update: false
+:reboot: false
+:unsafe: false
+:nested: false
+:tests:
+- omv up
+- vssh root@mgmt1 -c which powertop
+- omv destroy
+:comment: simple package install test case
+:reallyrm: false

test/shell/.gitignore

@@ -0,0 +1 @@
+mgmt

test/shell/t1.sh

@@ -0,0 +1,15 @@
+#!/bin/bash -e
+# NOTES:
+#	* this is a simple shell based `mgmt` test case
+#	* it is recommended that you run mgmt wrapped in the timeout command
+#	* it is recommended that you run mgmt with --no-watch
+#	* it is recommended that you run mgmt --converged-timeout=<seconds>
+#	* you can run mgmt with --max-runtime=<seconds> in special scenarios
+
+set -o errexit
+set -o pipefail
+
+timeout --kill-after=3s 1s ./mgmt --help # hello world!
+pid=$!
+wait $pid	# get exit status
+exit $?

test/shell/t2.sh

@@ -0,0 +1,20 @@
+#!/bin/bash -e
+
+if env | grep -q -e '^TRAVIS=true$'; then
+	# inotify doesn't seem to work properly on travis
+	echo "Travis and Jenkins give wonky results here, skipping test!"
+	exit
+fi
+
+# run till completion
+timeout --kill-after=15s 10s ./mgmt run --file t2.yaml --converged-timeout=5 --no-watch --tmp-prefix &
+pid=$!
+wait $pid	# get exit status
+e=$?
+
+test -e /tmp/mgmt/f1
+test -e /tmp/mgmt/f2
+test -e /tmp/mgmt/f3
+test ! -e /tmp/mgmt/f4
+
+exit $e

test/shell/t2.yaml

@@ -0,0 +1,41 @@
+---
+graph: mygraph
+resources:
+  noop:
+  - name: noop1
+  file:
+  - name: file1
+    path: "/tmp/mgmt/f1"
+    content: |
+      i am f1
+    state: exists
+  - name: file2
+    path: "/tmp/mgmt/f2"
+    content: |
+      i am f2
+    state: exists
+  - name: file3
+    path: "/tmp/mgmt/f3"
+    content: |
+      i am f3
+    state: exists
+  - name: file4
+    path: "/tmp/mgmt/f4"
+    content: |
+      i am f4 and i should not be here
+    state: absent
+edges:
+- name: e1
+  from:
+    kind: file
+    name: file1
+  to:
+    kind: file
+    name: file2
+- name: e2
+  from:
+    kind: file
+    name: file2
+  to:
+    kind: file
+    name: file3

test/shell/t3-a.yaml

@@ -0,0 +1,28 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1a
+    path: "/tmp/mgmt/mgmtA/f1a"
+    content: |
+      i am f1
+    state: exists
+  - name: file2a
+    path: "/tmp/mgmt/mgmtA/f2a"
+    content: |
+      i am f2
+    state: exists
+  - name: "@@file3a"
+    path: "/tmp/mgmt/mgmtA/f3a"
+    content: |
+      i am f3, exported from host A
+    state: exists
+  - name: "@@file4a"
+    path: "/tmp/mgmt/mgmtA/f4a"
+    content: |
+      i am f4, exported from host A
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmt/mgmtA/"
+edges: []

test/shell/t3-b.yaml

@@ -0,0 +1,28 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1b
+    path: "/tmp/mgmt/mgmtB/f1b"
+    content: |
+      i am f1
+    state: exists
+  - name: file2b
+    path: "/tmp/mgmt/mgmtB/f2b"
+    content: |
+      i am f2
+    state: exists
+  - name: "@@file3b"
+    path: "/tmp/mgmt/mgmtB/f3b"
+    content: |
+      i am f3, exported from host B
+    state: exists
+  - name: "@@file4b"
+    path: "/tmp/mgmt/mgmtB/f4b"
+    content: |
+      i am f4, exported from host B
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmt/mgmtB/"
+edges: []

test/shell/t3-c.yaml

@@ -0,0 +1,28 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1c
+    path: "/tmp/mgmt/mgmtC/f1c"
+    content: |
+      i am f1
+    state: exists
+  - name: file2c
+    path: "/tmp/mgmt/mgmtC/f2c"
+    content: |
+      i am f2
+    state: exists
+  - name: "@@file3c"
+    path: "/tmp/mgmt/mgmtC/f3c"
+    content: |
+      i am f3, exported from host C
+    state: exists
+  - name: "@@file4c"
+    path: "/tmp/mgmt/mgmtC/f4c"
+    content: |
+      i am f4, exported from host C
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmt/mgmtC/"
+edges: []