thanks: More people really needed to be on the list

Felix wins the points for first asking the question I knew would
eventually come but didn't document earlier.

.dockerignore

@@ -0,0 +1 @@
+docker

.editorconfig

@@ -0,0 +1,19 @@
+; This file is for unifying the coding style for different editors and IDEs.
+; Plugins are available for notepad++, emacs, vim, gedit,
+; textmate, visual studio, and more.
+;
+; See http://editorconfig.org for details.
+
+# Top-most EditorConfig file.
+root = true
+
+[*]
+end_of_line = lf
+insert_final_newline = true
+trim_trailing_whitespace = true
+
+[*.go]
+indent_style = tab
+
+[Makefile]
+indent_style = tab

.gitignore

@@ -6,4 +6,5 @@ old/
 tmp/
 *_stringer.go
 mgmt
+mgmt.static
 rpmbuild/

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "vendor/github.com/coreos/etcd"]
+	path = vendor/github.com/coreos/etcd
+	url = https://github.com/coreos/etcd/

.travis.yml

@@ -1,15 +1,19 @@
 language: go
 go:
   - 1.4.3
-  - 1.5.2
+  - 1.5.3
+  - 1.6
   - tip
-dist: trusty
+sudo: false
-sudo: required
+before_install: 'git fetch --unshallow'
 install: 'make deps'
 script: 'make test'
 matrix:
+  fast_finish: true
   allow_failures:
     - go: tip
+    - go: 1.4.3
+    - go: 1.6
 notifications:
   irc:
     channels:

AUTHORS

@@ -5,3 +5,4 @@ For a more exhaustive list please run: git log --format='%aN' | sort -u
 This list is sorted alphabetically by first name.
 
 James Shubin
+Paul Morgan

DOCUMENTATION.md

@@ -30,18 +30,23 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 1. [Overview](#overview)
 2. [Project description - What the project does](#project-description)
 3. [Setup - Getting started with mgmt](#setup)
-4. [Usage/FAQ - Notes on usage and frequently asked questions](#usage-and-frequently-asked-questions)
+4. [Features - All things mgmt can do](#features)
-5. [Reference - Detailed reference](#reference)
+	* [Autoedges - Automatic resource relationships](#autoedges)
-	* [graph.yaml](#graph.yaml)
+	* [Autogrouping - Automatic resource grouping](#autogrouping)
+	* [Automatic clustering - Automatic cluster management](#automatic-clustering)
+	* [Remote mode - Remote "agent-less" execution](#remote-agent-less-mode)
+5. [Usage/FAQ - Notes on usage and frequently asked questions](#usage-and-frequently-asked-questions)
+6. [Reference - Detailed reference](#reference)
+	* [Graph definition file](#graph-definition-file)
 	* [Command line](#command-line)
-6. [Examples - Example configurations](#examples)
+7. [Examples - Example configurations](#examples)
-7. [Development - Background on module development and reporting bugs](#development)
+8. [Development - Background on module development and reporting bugs](#development)
-8. [Authors - Authors and contact information](#authors)
+9. [Authors - Authors and contact information](#authors)
 
 ##Overview
 
-The `mgmt` tool is a research prototype to demonstrate next generation config
+The `mgmt` tool is a next generation config management prototype. It's not yet
-management techniques. Hopefully it will evolve into a useful, robust tool.
+ready for production, but we hope to get there soon. Get involved today!
 
 ##Project Description
 
@@ -49,6 +54,16 @@ The mgmt tool is a distributed, event driven, config management tool, that
 supports parallel execution, and librarification to be used as the management
 foundation in and for, new and existing software.
 
+For more information, you may like to read some blog posts from the author:
+
+* [Next generation config mgmt](https://ttboj.wordpress.com/2016/01/18/next-generation-configuration-mgmt/)
+* [Automatic edges in mgmt](https://ttboj.wordpress.com/2016/03/14/automatic-edges-in-mgmt/)
+* [Automatic grouping in mgmt](https://ttboj.wordpress.com/2016/03/30/automatic-grouping-in-mgmt/)
+* [Automatic clustering in mgmt](https://ttboj.wordpress.com/2016/06/20/automatic-clustering-in-mgmt/)
+
+There is also an [introductory video](http://meetings-archive.debian.net/pub/debian-meetings/2016/debconf16/Next_Generation_Config_Mgmt.webm) available.
+Older videos and other material [is available](https://github.com/purpleidea/mgmt/#on-the-web).
+
 ##Setup
 
 During this prototype phase, the tool can be run out of the source directory.
@@ -57,6 +72,103 @@ get started. Beware that this _can_ cause data loss. Understand what you're
 doing first, or perform these actions in a virtual environment such as the one
 provided by [Oh-My-Vagrant](https://github.com/purpleidea/oh-my-vagrant).
 
+##Features
+
+This section details the numerous features of mgmt and some caveats you might
+need to be aware of.
+
+###Autoedges
+
+Automatic edges, or AutoEdges, is the mechanism in mgmt by which it will
+automatically create dependencies for you between resources. For example,
+since mgmt can discover which files are installed by a package it will
+automatically ensure that any file resource you declare that matches a
+file installed by your package resource will only be processed after the
+package is installed.
+
+####Controlling autoedges
+
+Though autoedges is likely to be very helpful and avoid you having to declare
+all dependencies explicitly, there are cases where this behaviour is
+undesirable.
+
+Some distributions allow package installations to automatically start the
+service they ship. This can be problematic in the case of packages like MySQL
+as there are configuration options that need to be set before MySQL is ever
+started for the first time (or you'll need to wipe the data directory). In
+order to handle this situation you can disable autoedges per resource and
+explicitly declare that you want `my.cnf` to be written to disk before the
+installation of the `mysql-server` package.
+
+You can disable autoedges for a resource by setting the `autoedge` key on
+the meta attributes of that resource to `false`.
+
+####Blog post
+
+You can read the introductory blog post about this topic here:
+[https://ttboj.wordpress.com/2016/03/14/automatic-edges-in-mgmt/](https://ttboj.wordpress.com/2016/03/14/automatic-edges-in-mgmt/)
+
+###Autogrouping
+
+Automatic grouping or AutoGroup is the mechanism in mgmt by which it will
+automatically group multiple resource vertices into a single one. This is
+particularly useful for grouping multiple package resources into a single
+resource, since the multiple installations can happen together in a single
+transaction, which saves a lot of time because package resources typically have
+a large fixed cost to running (downloading and verifying the package repo) and
+if they are grouped they share this fixed cost. This grouping feature can be
+used for other use cases too.
+
+You can disable autogrouping for a resource by setting the `autogroup` key on
+the meta attributes of that resource to `false`.
+
+####Blog post
+
+You can read the introductory blog post about this topic here:
+[https://ttboj.wordpress.com/2016/03/30/automatic-grouping-in-mgmt/](https://ttboj.wordpress.com/2016/03/30/automatic-grouping-in-mgmt/)
+
+###Automatic clustering
+
+Automatic clustering is a feature by which mgmt automatically builds, scales,
+and manages the embedded etcd cluster which is compiled into mgmt itself. It is
+quite helpful for rapidly bootstrapping clusters and avoiding the extra work to
+setup etcd.
+
+If you prefer to avoid this feature. you can always opt to use an existing etcd
+cluster that is managed separately from mgmt by pointing your mgmt agents at it
+with the `--seeds` variable.
+
+####Blog post
+
+You can read the introductory blog post about this topic here:
+[https://ttboj.wordpress.com/2016/06/20/automatic-clustering-in-mgmt/](https://ttboj.wordpress.com/2016/06/20/automatic-clustering-in-mgmt/)
+
+###Remote ("agent-less") mode
+
+Remote mode is a special mode that lets you kick off mgmt runs on one or more
+remote machines which are only accessible via SSH. In this mode the initiating
+host connects over SSH, copies over the `mgmt` binary, opens an SSH tunnel, and
+runs the remote program while simultaneously passing the etcd traffic back
+through the tunnel so that the initiators etcd cluster can be used to exchange
+resource data.
+
+The interesting benefit of this architecture is that multiple hosts which can't
+connect directly use the initiator to pass the important traffic through to each
+other. Once the cluster has converged all the remote programs can shutdown
+leaving no residual agent.
+
+This mode can also be useful for bootstrapping a new host where you'd like to
+have the service run continuously and as part of an mgmt cluster normally.
+
+In particular, when combined with the `--converged-timeout` parameter, the
+entire set of running mgmt agents will need to all simultaneously converge for
+the group to exit. This is particularly useful for bootstrapping new clusters
+which need to exchange information that is only available at run time.
+
+####Blog post
+
+An introductory blog post about this topic will follow soon.
+
 ##Usage and frequently asked questions
 (Send your questions as a patch to this FAQ! I'll review it, merge it, and
 respond by commit with the answer.)
@@ -75,6 +187,58 @@ chosen, but it was also somewhat arbitrary. If there is available interest,
 good reasoning, *and* patches, then we would consider either switching or
 supporting both, but this is not a high priority at this time.
 
+###Can I use an existing etcd cluster instead of the automatic embedded servers?
+
+Yes, it's possible to use an existing etcd cluster instead of the automatic,
+elastic embedded etcd servers. To do so, simply point to the cluster with the
+`--seeds` variable, the same way you would if you were seeding a new member to
+an existing mgmt cluster.
+
+The downside to this approach is that you won't benefit from the automatic
+elastic nature of the embedded etcd servers, and that you're responsible if you
+accidentally break your etcd cluster, or if you use an unsupported version.
+
+###What does the error message about an inconsistent dataDir mean?
+
+If you get an error message similar to:
+
+```
+Etcd: Connect: CtxError...
+Etcd: CtxError: Reason: CtxDelayErr(5s): No endpoints available yet!
+Etcd: Connect: Endpoints: []
+Etcd: The dataDir (/var/lib/mgmt/etcd) might be inconsistent or corrupt.
+```
+
+This happens when there are a series of fatal connect errors in a row. This can
+happen when you start `mgmt` using a dataDir that doesn't correspond to the
+current cluster view. As a result, the embedded etcd server never finishes
+starting up, and as a result, a default endpoint never gets added. The solution
+is to either reconcile the mistake, and if there is no important data saved, you
+can remove the etcd dataDir. This is typically `/var/lib/mgmt/etcd/member/`.
+
+###Why do resources have both a `Compare` method and an `IFF` (on the UUID) method?
+
+The `Compare()` methods are for determining if two resources are effectively the
+same, which is used to make graph change delta's efficient. This is when we want
+to change from the current running graph to a new graph, but preserve the common
+vertices. Since we want to make this process efficient, we only update the parts
+that are different, and leave everything else alone. This `Compare()` method can
+tell us if two resources are the same.
+
+The `IFF()` method is part of the whole UUID system, which is for discerning if
+a resource meets the requirements another expects for an automatic edge. This is
+because the automatic edge system assumes a unified UUID pattern to test for
+equality. In the future it might be helpful or sane to merge the two similar
+comparison functions although for now they are separate because they are
+actually answer different questions.
+
+###Did you know that there is a band named `MGMT`?
+
+I didn't realize this when naming the project, and it is accidental. After much
+anguishing, I chose the name because it was short and I thought it was
+appropriately descriptive. If you need a less ambiguous search term or phrase,
+you can try using `mgmtconfig` or `mgmt config`.
+
 ###You didn't answer my question, or I have a question!
 
 It's best to ask on [IRC](https://webchat.freenode.net/?channels=#mgmtconfig)
@@ -92,11 +256,11 @@ information on these options, please view the source at:
 If you feel that a well used option needs documenting here, please patch it!
 
 ###Overview of reference
-* [graph.yaml](#graph.yaml): Main graph definition file.
+* [Graph definition file](#graph-definition-file): Main graph definition file.
 * [Command line](#command-line): Command line parameters.
 
-###graph.yaml
+###Graph definition file
-This is the compiled graph definition file. The format is currently
+graph.yaml is the compiled graph definition file. The format is currently
 undocumented, but by looking through the [examples/](https://github.com/purpleidea/mgmt/tree/master/examples)
 you can probably figure out most of it, as it's fairly intuitive.
 
@@ -114,12 +278,79 @@ Exit if the machine has converged for approximately this many seconds.
 Exit when the agent has run for approximately this many seconds. This is not
 generally recommended, but may be useful for users who know what they're doing.
 
+####`--noop`
+Globally force all resources into no-op mode. This also disables the export to
+etcd functionality, but does not disable resource collection, however all
+resources that are collected will have their individual noop settings set.
+
+####`--remote <graph.yaml>`
+Point to a graph file to run on the remote host specified within. This parameter
+can be used multiple times if you'd like to remotely run on multiple hosts in
+parallel.
+
+####`--allow-interactive`
+Allow interactive prompting for SSH passwords if there is no authentication
+method that works.
+
+####`--ssh-priv-id-rsa`
+Specify the path for finding SSH keys. This defaults to `~/.ssh/id_rsa`. To
+never use this method of authentication, set this to the empty string.
+
+####`--cconns`
+The maximum number of concurrent remote ssh connections to run. This defaults
+to `0`, which means unlimited.
+
+####`--no-caching`
+Don't allow remote caching of the remote execution binary. This will require
+the binary to be copied over for every remote execution, but it limits the
+likelihood that there is leftover information from the configuration process.
+
+####`--prefix <path>`
+Specify a path to a custom working directory prefix. This directory will get
+created if it does not exist. This usually defaults to `/var/lib/mgmt/`. This
+can't be combined with the `--tmp-prefix` option. It can be combined with the
+`--allow-tmp-prefix` option.
+
+####`--tmp-prefix`
+If this option is specified, a temporary prefix will be used instead of the
+default prefix. This can't be combined with the `--prefix` option.
+
+####`--allow-tmp-prefix`
+If this option is specified, we will attempt to fall back to a temporary prefix
+if the primary prefix couldn't be created. This is useful for avoiding failures
+in environments where the primary prefix may or may not be available, but you'd
+like to try. The canonical example is when running `mgmt` with `--remote` there
+might be a cached copy of the binary in the primary prefix, but in case there's
+no binary available continue working in a temporary directory to avoid failure.
+
 ##Examples
 For example configurations, please consult the [examples/](https://github.com/purpleidea/mgmt/tree/master/examples) directory in the git
 source repository. It is available from:
 
 [https://github.com/purpleidea/mgmt/tree/master/examples](https://github.com/purpleidea/mgmt/tree/master/examples)
 
+### Systemd:
+See [`misc/mgmt.service`](misc/mgmt.service) for a sample systemd unit file.
+This unit file is part of the RPM.
+
+To specify your custom options for `mgmt` on a systemd distro:
+
+```bash
+sudo mkdir -p /etc/systemd/system/mgmt.service.d/
+
+cat > /etc/systemd/system/mgmt.service.d/env.conf <<EOF
+# Environment variables:
+MGMT_SEEDS=http://127.0.0.1:2379
+MGMT_CONVERGED_TIMEOUT=-1
+MGMT_MAX_RUNTIME=0
+
+# Other CLI options if necessary.
+#OPTS="--max-runtime=0"
+EOF
+
+sudo systemctl daemon-reload
+```
+
 ##Development
 
 This is a project that I started in my free time in 2013. Development is driven

Makefile

@@ -16,28 +16,29 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 SHELL = /bin/bash
-.PHONY: all version program path deps run race build clean test format docs rpmbuild rpm srpm spec tar upload upload-sources upload-srpms upload-rpms copr
+.PHONY: all version program path deps run race generate build clean test gofmt yamlfmt format docs rpmbuild mkdirs rpm srpm spec tar upload upload-sources upload-srpms upload-rpms copr
 .SILENT: clean
 
-SVERSION := $(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --dirty --always)
+SVERSION := $(or $(SVERSION),$(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --dirty --always))
-VERSION := $(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --abbrev=0)
+VERSION := $(or $(VERSION),$(shell git describe --match '[0-9]*\.[0-9]*\.[0-9]*' --tags --abbrev=0))
-PROGRAM := $(shell basename --suffix=-$(VERSION) $(notdir $(CURDIR)))
+PROGRAM := $(shell echo $(notdir $(CURDIR)) | cut -f1 -d"-")
+OLDGOLANG := $(shell go version | grep -E 'go1.3|go1.4')
 ifeq ($(VERSION),$(SVERSION))
 	RELEASE = 1
 else
 	RELEASE = untagged
 endif
-ARCH = $(shell arch)
+ARCH = $(uname -m)
-SPEC = rpmbuild/SPECS/mgmt.spec
+SPEC = rpmbuild/SPECS/$(PROGRAM).spec
-SOURCE = rpmbuild/SOURCES/mgmt-$(VERSION).tar.bz2
+SOURCE = rpmbuild/SOURCES/$(PROGRAM)-$(VERSION).tar.bz2
-SRPM = rpmbuild/SRPMS/mgmt-$(VERSION)-$(RELEASE).src.rpm
+SRPM = rpmbuild/SRPMS/$(PROGRAM)-$(VERSION)-$(RELEASE).src.rpm
-SRPM_BASE = mgmt-$(VERSION)-$(RELEASE).src.rpm
+SRPM_BASE = $(PROGRAM)-$(VERSION)-$(RELEASE).src.rpm
-RPM = rpmbuild/RPMS/mgmt-$(VERSION)-$(RELEASE).$(ARCH).rpm
+RPM = rpmbuild/RPMS/$(PROGRAM)-$(VERSION)-$(RELEASE).$(ARCH).rpm
 USERNAME := $(shell cat ~/.config/copr 2>/dev/null | grep username | awk -F '=' '{print $$2}' | tr -d ' ')
 SERVER = 'dl.fedoraproject.org'
-REMOTE_PATH = 'pub/alt/$(USERNAME)/mgmt'
+REMOTE_PATH = 'pub/alt/$(USERNAME)/$(PROGRAM)'
 
-all: docs
+all: docs $(PROGRAM).static
 
 # show the current version
 version:
@@ -53,39 +54,55 @@ deps:
 	./misc/make-deps.sh
 
 run:
-	find -maxdepth 1 -type f -name '*.go' -not -name '*_test.go' | xargs go run -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION)"
+	find . -maxdepth 1 -type f -name '*.go' -not -name '*_test.go' | xargs go run -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION)"
 
 # include race flag
 race:
-	find -maxdepth 1 -type f -name '*.go' -not -name '*_test.go' | xargs go run -race -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION)"
+	find . -maxdepth 1 -type f -name '*.go' -not -name '*_test.go' | xargs go run -race -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION)"
 
-build: mgmt
+generate:
-
-mgmt: main.go
-	@echo "Building: $(PROGRAM), version: $(SVERSION)."
 	go generate
-	# avoid equals sign in old golang versions eg in: -X foo=bar
+
-	if go version | grep -qE 'go1.3|go1.4'; then \
+build: $(PROGRAM)
-		go build -ldflags "-X main.program $(PROGRAM) -X main.version $(SVERSION)" -o mgmt; \
+
-	else \
+$(PROGRAM): main.go
-		go build -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION)" -o mgmt; \
+	@echo "Building: $(PROGRAM), version: $(SVERSION)..."
-	fi
+ifneq ($(OLDGOLANG),)
+	@# avoid equals sign in old golang versions eg in: -X foo=bar
+	time go build -ldflags "-X main.program $(PROGRAM) -X main.version $(SVERSION)" -o $(PROGRAM);
+else
+	time go build -ldflags "-X main.program=$(PROGRAM) -X main.version=$(SVERSION)" -o $(PROGRAM);
+endif
+
+$(PROGRAM).static: main.go
+	@echo "Building: $(PROGRAM).static, version: $(SVERSION)..."
+	go generate
+ifneq ($(OLDGOLANG),)
+	@# avoid equals sign in old golang versions eg in: -X foo=bar
+	go build -a -installsuffix cgo -tags netgo -ldflags '-extldflags "-static" -X main.program $(PROGRAM) -X main.version $(SVERSION)' -o $(PROGRAM).static;
+else
+	go build -a -installsuffix cgo -tags netgo -ldflags '-extldflags "-static" -X main.program=$(PROGRAM) -X main.version=$(SVERSION)' -o $(PROGRAM).static;
+endif
 
 clean:
-	[ ! -e mgmt ] || rm mgmt
+	[ ! -e $(PROGRAM) ] || rm $(PROGRAM)
-	rm -f *_stringer.go	# generated by `go generate`
+	#rm -f *_stringer.go	# generated by `go generate`
 
 test:
 	./test.sh
 
-format:
+gofmt:
-	find -type f -name '*.go' -not -path './old/*' -not -path './tmp/*' -exec gofmt -w {} \;
+	find . -maxdepth 3 -type f -name '*.go' -not -path './old/*' -not -path './tmp/*' -exec gofmt -w {} \;
-	find -type f -name '*.yaml' -not -path './old/*' -not -path './tmp/*' -not -path './omv.yaml' -exec ruby -e "require 'yaml'; x=YAML.load_file('{}').to_yaml.each_line.map(&:rstrip).join(10.chr)+10.chr; File.open('{}', 'w').write x" \;
 
-docs: mgmt-documentation.pdf
+yamlfmt:
+	find . -type f -name '*.yaml' -not -path './old/*' -not -path './tmp/*' -not -path './omv.yaml' -exec ruby -e "require 'yaml'; x=YAML.load_file('{}').to_yaml.each_line.map(&:rstrip).join(10.chr)+10.chr; File.open('{}', 'w').write x" \;
 
-mgmt-documentation.pdf: DOCUMENTATION.md
+format: gofmt yamlfmt
-	pandoc DOCUMENTATION.md -o 'mgmt-documentation.pdf'
+
+docs: $(PROGRAM)-documentation.pdf
+
+$(PROGRAM)-documentation.pdf: DOCUMENTATION.md
+	pandoc DOCUMENTATION.md -o '$(PROGRAM)-documentation.pdf'
 
 #
 #	build aliases
@@ -116,21 +133,21 @@ upload: upload-sources upload-srpms upload-rpms
 $(RPM): $(SPEC) $(SOURCE)
 	@echo Running rpmbuild -bb...
 	rpmbuild --define '_topdir $(shell pwd)/rpmbuild' -bb $(SPEC) && \
-	mv rpmbuild/RPMS/$(ARCH)/mgmt-$(VERSION)-$(RELEASE).*.rpm $(RPM)
+	mv rpmbuild/RPMS/$(ARCH)/$(PROGRAM)-$(VERSION)-$(RELEASE).*.rpm $(RPM)
 
 $(SRPM): $(SPEC) $(SOURCE)
 	@echo Running rpmbuild -bs...
 	rpmbuild --define '_topdir $(shell pwd)/rpmbuild' -bs $(SPEC)
 	# renaming is not needed because we aren't using the dist variable
-	#mv rpmbuild/SRPMS/mgmt-$(VERSION)-$(RELEASE).*.src.rpm $(SRPM)
+	#mv rpmbuild/SRPMS/$(PROGRAM)-$(VERSION)-$(RELEASE).*.src.rpm $(SRPM)
 
 #
 #	spec
 #
-$(SPEC): rpmbuild/ mgmt.spec.in
+$(SPEC): rpmbuild/ spec.in
 	@echo Running templater...
-	#cat mgmt.spec.in > $(SPEC)
+	#cat spec.in > $(SPEC)
-	sed -e s/__VERSION__/$(VERSION)/ -e s/__RELEASE__/$(RELEASE)/ < mgmt.spec.in > $(SPEC)
+	sed -e s/__PROGRAM__/$(PROGRAM)/ -e s/__VERSION__/$(VERSION)/ -e s/__RELEASE__/$(RELEASE)/ < spec.in > $(SPEC)
 	# append a changelog to the .spec file
 	git log --format="* %cd %aN <%aE>%n- (%h) %s%d%n" --date=local | sed -r 's/[0-9]+:[0-9]+:[0-9]+ //' >> $(SPEC)
 
@@ -140,7 +157,7 @@ $(SPEC): rpmbuild/ mgmt.spec.in
 $(SOURCE): rpmbuild/
 	@echo Running git archive...
 	# use HEAD if tag doesn't exist yet, so that development is easier...
-	git archive --prefix=mgmt-$(VERSION)/ -o $(SOURCE) $(VERSION) 2> /dev/null || (echo 'Warning: $(VERSION) does not exist. Using HEAD instead.' && git archive --prefix=mgmt-$(VERSION)/ -o $(SOURCE) HEAD)
+	git archive --prefix=$(PROGRAM)-$(VERSION)/ -o $(SOURCE) $(VERSION) 2> /dev/null || (echo 'Warning: $(VERSION) does not exist. Using HEAD instead.' && git archive --prefix=$(PROGRAM)-$(VERSION)/ -o $(SOURCE) HEAD)
 	# TODO: if git archive had a --submodules flag this would easier!
 	@echo Running git archive submodules...
 	# i thought i would need --ignore-zeros, but it doesn't seem necessary!
@@ -149,14 +166,14 @@ $(SOURCE): rpmbuild/
 		temp="$${temp#\'}"; \
 		path=$$temp; \
 		[ "$$path" = "" ] && continue; \
-		(cd $$path && git archive --prefix=mgmt-$(VERSION)/$$path/ HEAD > $$p/rpmbuild/tmp.tar && tar --concatenate --file=$$p/$(SOURCE) $$p/rpmbuild/tmp.tar && rm $$p/rpmbuild/tmp.tar); \
+		(cd $$path && git archive --prefix=$(PROGRAM)-$(VERSION)/$$path/ HEAD > $$p/rpmbuild/tmp.tar && tar --concatenate --file=$$p/$(SOURCE) $$p/rpmbuild/tmp.tar && rm $$p/rpmbuild/tmp.tar); \
 	done
 
 # TODO: ensure that each sub directory exists
 rpmbuild/:
 	mkdir -p rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
 
-rpmbuild:
+mkdirs:
 	mkdir -p rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
 
 #

README.md

@@ -1,5 +1,6 @@
 # *mgmt*: This is: mgmt!
 
+[![Go Report Card](https://goreportcard.com/badge/github.com/purpleidea/mgmt)](https://goreportcard.com/report/github.com/purpleidea/mgmt)
 [![Build Status](https://secure.travis-ci.org/purpleidea/mgmt.png?branch=master)](http://travis-ci.org/purpleidea/mgmt)
 [![Documentation](https://img.shields.io/docs/markdown.png)](DOCUMENTATION.md)
 [![IRC](https://img.shields.io/irc/%23mgmtconfig.png)](https://webchat.freenode.net/?channels=#mgmtconfig)
@@ -15,9 +16,9 @@ Please join the [#mgmtconfig](https://webchat.freenode.net/?channels=#mgmtconfig
 If you have a well phrased question that might benefit others, consider asking it by sending a patch to the documentation [FAQ](https://github.com/purpleidea/mgmt/blob/master/DOCUMENTATION.md#usage-and-frequently-asked-questions) section. I'll merge your question, and a patch with the answer!
 
 ## Quick start:
-* Either get the golang dependencies on your own, or run `make deps` if you're comfortable with how we install them.
+* Clone the repository recursively, eg: `git clone --recursive https://github.com/purpleidea/mgmt/`.
-* Run `make build` to get a fresh built `mgmt` binary.
+* Get the remaining golang dependencies on your own, or run `make deps` if you're comfortable with how we install them.
-* Run `cd $(mktemp --tmpdir -d tmp.XXX) && etcd` to get etcd running. The `mgmt` software will do this automatically for you in the future.
+* Run `make build` to get a freshly built `mgmt` binary.
 * Run `time ./mgmt run --file examples/graph0.yaml --converged-timeout=1` to try out a very simple example!
 * To run continuously in the default mode of operation, omit the `--converged-timeout` option.
 * Have fun hacking on our future technology!
@@ -31,10 +32,12 @@ Please see: [DOCUMENTATION.md](DOCUMENTATION.md) or [PDF](https://pdfdoc-purplei
 ## Roadmap:
 Please see: [TODO.md](TODO.md) for a list of upcoming work and TODO items.
 Please get involved by working on one of these items or by suggesting something else!
+Feel free to grab one of the straightforward [#mgmtlove](https://github.com/purpleidea/mgmt/labels/mgmtlove) issues if you're a first time contributor to the project or if you're unsure about what to hack on!
 
 ## Bugs:
 Please set the `DEBUG` constant in [main.go](https://github.com/purpleidea/mgmt/blob/master/main.go) to `true`, and post the logs when you report the [issue](https://github.com/purpleidea/mgmt/issues).
 Bonus points if you provide a [shell](https://github.com/purpleidea/mgmt/tree/master/test/shell) or [OMV](https://github.com/purpleidea/mgmt/tree/master/test/omv) reproducible test case.
+Feel free to read my article on [debugging golang programs](https://ttboj.wordpress.com/2016/02/15/debugging-golang-programs/).
 
 ## Dependencies:
 * golang 1.4 or higher (required, available in most distros)
@@ -46,6 +49,7 @@ Bonus points if you provide a [shell](https://github.com/purpleidea/mgmt/tree/ma
         go get github.com/codegangsta/cli
         go get github.com/coreos/go-systemd/dbus
         go get github.com/coreos/go-systemd/util
+        go get github.com/coreos/pkg/capnslog
 
 * stringer (required for building), available as a package on some platforms, otherwise via `go get`
 
@@ -58,8 +62,23 @@ Bonus points if you provide a [shell](https://github.com/purpleidea/mgmt/tree/ma
 We'd love to have your patches! Please send them by email, or as a pull request.
 
 ## On the web:
-* Introductory blog post: [https://ttboj.wordpress.com/2016/01/18/next-generation-configuration-mgmt/](https://ttboj.wordpress.com/2016/01/18/next-generation-configuration-mgmt/)
+* James Shubin; blog: [Next generation configuration mgmt](https://ttboj.wordpress.com/2016/01/18/next-generation-configuration-mgmt/)
-* Julian Dunn at Cfgmgmtcamp 2016 [https://www.youtube.com/watch?v=kfF9IATUask&t=1949&html5=1](https://www.youtube.com/watch?v=kfF9IATUask&t=1949&html5=1)
+* James Shubin; video: [Introductory recording from DevConf.cz 2016](https://www.youtube.com/watch?v=GVhpPF0j-iE&html5=1)
+* James Shubin; video: [Introductory recording from CfgMgmtCamp.eu 2016](https://www.youtube.com/watch?v=fNeooSiIRnA&html5=1)
+* Julian Dunn; video: [On mgmt at CfgMgmtCamp.eu 2016](https://www.youtube.com/watch?v=kfF9IATUask&t=1949&html5=1)
+* Walter Heck; slides: [On mgmt at CfgMgmtCamp.eu 2016](http://www.slideshare.net/olindata/configuration-management-time-for-a-4th-generation/3)
+* Marco Marongiu; blog: [On mgmt](http://syslog.me/2016/02/15/leap-or-die/)
+* Felix Frank; blog: [From Catalog To Mgmt (on puppet to mgmt "transpiling")](https://ffrank.github.io/features/2016/02/18/from-catalog-to-mgmt/)
+* James Shubin; blog: [Automatic edges in mgmt (...and the pkg resource)](https://ttboj.wordpress.com/2016/03/14/automatic-edges-in-mgmt/)
+* James Shubin; blog: [Automatic grouping in mgmt](https://ttboj.wordpress.com/2016/03/30/automatic-grouping-in-mgmt/)
+* John Arundel; tweet: [“Puppet’s days are numbered.”](https://twitter.com/bitfield/status/732157519142002688)
+* Felix Frank; blog: [Puppet, Meet Mgmt (on puppet to mgmt internals)](https://ffrank.github.io/features/2016/06/12/puppet,-meet-mgmt/)
+* Felix Frank; blog: [Puppet Powered Mgmt (puppet to mgmt tl;dr)](https://ffrank.github.io/features/2016/06/19/puppet-powered-mgmt/)
+* James Shubin; blog: [Automatic clustering in mgmt](https://ttboj.wordpress.com/2016/06/20/automatic-clustering-in-mgmt/)
+* James Shubin; video: [Recording from CoreOSFest 2016](https://www.youtube.com/watch?v=KVmDCUA42wc&html5=1)
+* James Shubin; video: [Recording from DebConf16](http://meetings-archive.debian.net/pub/debian-meetings/2016/debconf16/Next_Generation_Config_Mgmt.webm) ([Slides](https://annex.debconf.org//debconf-share/debconf16/slides/15-next-generation-config-mgmt.pdf))
+* Felix Frank; blog: [Edging It All In (puppet and mgmt edges)](https://ffrank.github.io/features/2016/07/12/edging-it-all-in/)
+* Felix Frank; blog: [Translating All The Things (puppet to mgmt translation warnings)](https://ffrank.github.io/features/2016/08/19/translating-all-the-things/)
 
 ##
 

THANKS

@@ -9,10 +9,16 @@ Chris Wright - For encouraging me to continue work on my prototype.
 
 Daniel Riek - For supporting and sheltering this project from bureaucracy.
 
+Diego Ongaro - For good chats, particularly around distributed systems.
+
+Felix Frank - For taking a difficult problem and building an inspiring solution.
+
 Ira Cooper - For having an algorithmic design discussion with me.
 
 Jeff Darcy - For some algorithm recommendations, and NACKing my TopoSort idea!
 
 Red Hat, inc. - For paying my salary, thus financially supporting my hacking.
 
+Samuel Gélineau - For help with programming language theory and design.
+
 And many others...

TODO.md

@@ -3,11 +3,12 @@ If you're looking for something to do, look here!
 Let us know if you're working on one of the items.
 
 ## Package resource
-- [ ] base type [bug](https://github.com/purpleidea/mgmt/issues/11)
+- [ ] getfiles support on debian [bug](https://github.com/hughsie/PackageKit/issues/118)
+- [ ] directory info on fedora [bug](https://github.com/hughsie/PackageKit/issues/117)
 - [ ] dnf blocker [bug](https://github.com/hughsie/PackageKit/issues/110)
 - [ ] install signal blocker [bug](https://github.com/hughsie/PackageKit/issues/109)
 
-## File resource
+## File resource [bug](https://github.com/purpleidea/mgmt/issues/13) [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove)
 - [ ] ability to make/delete folders
 - [ ] recursive argument (can recursively watch/modify contents)
 - [ ] force argument (can cause switch from file <-> folder)
@@ -17,13 +18,15 @@ Let us know if you're working on one of the items.
 - [ ] base resource improvements
 
 ## Timer resource
-- [ ] base resource
+- [ ] base resource [bug](https://github.com/purpleidea/mgmt/issues/15) [:heart:](https://github.com/purpleidea/mgmt/labels/mgmtlove)
 - [ ] reset on recompile
 - [ ] increment algorithm (linear, exponential, etc...)
 
+## Virt (libvirt) resource
+- [ ] base resource [bug](https://github.com/purpleidea/mgmt/issues/25)
+
 ## Etcd improvements
 - [ ] embedded etcd master
-- [ ] capnslog fixes [bug](https://github.com/coreos/etcd/issues/4115)
 
 ## Language improvements
 - [ ] language design
@@ -35,9 +38,6 @@ Let us know if you're working on one of the items.
 
 ## Other
 - [ ] better error/retry handling
-- [ ] resource grouping
-- [ ] automatic dependency adding (eg: packagekit file dependencies)
-- [ ] rpm package target in Makefile
 - [ ] deb package target in Makefile
 - [ ] reproducible builds
 - [ ] add your suggestions!

config.go

@@ -19,19 +19,21 @@ package main
 
 import (
 	"errors"
+	"fmt"
 	"gopkg.in/yaml.v2"
 	"io/ioutil"
 	"log"
+	"reflect"
 	"strings"
 )
 
-type collectorTypeConfig struct {
+type collectorResConfig struct {
-	Type    string `yaml:"type"`
+	Kind    string `yaml:"kind"`
 	Pattern string `yaml:"pattern"` // XXX: Not Implemented
 }
 
 type vertexConfig struct {
-	Type string `yaml:"type"`
+	Kind string `yaml:"kind"`
 	Name string `yaml:"name"`
 }
 
@@ -41,19 +43,25 @@ type edgeConfig struct {
 	To   vertexConfig `yaml:"to"`
 }
 
+// GraphConfig is the data structure that describes a single graph to run.
 type GraphConfig struct {
-	Graph string `yaml:"graph"`
+	Graph     string `yaml:"graph"`
-	Types struct {
+	Resources struct {
-		Noop    []NoopType    `yaml:"noop"`
+		Noop  []*NoopRes  `yaml:"noop"`
-		File    []FileType    `yaml:"file"`
+		Pkg   []*PkgRes   `yaml:"pkg"`
-		Service []ServiceType `yaml:"service"`
+		File  []*FileRes  `yaml:"file"`
-		Exec    []ExecType    `yaml:"exec"`
+		Svc   []*SvcRes   `yaml:"svc"`
-	} `yaml:"types"`
+		Exec  []*ExecRes  `yaml:"exec"`
-	Collector []collectorTypeConfig `yaml:"collect"`
+		Timer []*TimerRes `yaml:"timer"`
-	Edges     []edgeConfig          `yaml:"edges"`
+	} `yaml:"resources"`
-	Comment   string                `yaml:"comment"`
+	Collector []collectorResConfig `yaml:"collect"`
+	Edges     []edgeConfig         `yaml:"edges"`
+	Comment   string               `yaml:"comment"`
+	Hostname  string               `yaml:"hostname"` // uuid for the host
+	Remote    string               `yaml:"remote"`
 }
 
+// Parse parses a data stream into the graph structure.
 func (c *GraphConfig) Parse(data []byte) error {
 	if err := yaml.Unmarshal(data, c); err != nil {
 		return err
@@ -64,166 +72,495 @@ func (c *GraphConfig) Parse(data []byte) error {
 	return nil
 }
 
+// ParseConfigFromFile takes a filename and returns the graph config structure.
 func ParseConfigFromFile(filename string) *GraphConfig {
 	data, err := ioutil.ReadFile(filename)
 	if err != nil {
-		log.Printf("Error: Config: ParseConfigFromFile: File: %v", err)
+		log.Printf("Config: Error: ParseConfigFromFile: File: %v", err)
 		return nil
 	}
 
 	var config GraphConfig
 	if err := config.Parse(data); err != nil {
-		log.Printf("Error: Config: ParseConfigFromFile: Parse: %v", err)
+		log.Printf("Config: Error: ParseConfigFromFile: Parse: %v", err)
 		return nil
 	}
 
 	return &config
 }
 
-// XXX: we need to fix this function so that it either fails without modifying
+// NewGraphFromConfig returns a new graph from existing input, such as from the
-// the graph, passes successfully and modifies it, or basically panics i guess
+// existing graph, and a GraphConfig struct.
-// this way an invalid compilation can leave the old graph running, and we we
+func (g *Graph) NewGraphFromConfig(config *GraphConfig, embdEtcd *EmbdEtcd, noop bool) (*Graph, error) {
-// don't modify a partial graph. so we really need to validate, and then perform
+	if config.Hostname == "" {
-// whatever actions are necessary
+		return nil, fmt.Errorf("Config: Error: Hostname can't be empty!")
-// finding some way to do this on a copy of the graph, and then do a graph diff
+	}
-// and merge the new data into the old graph would be more appropriate, in
-// particular if we can ensure the graph merge can't fail. As for the putting
-// of stuff into etcd, we should probably store the operations to complete in
-// the new graph, and keep retrying until it succeeds, thus blocking any new
-// etcd operations until that time.
-func UpdateGraphFromConfig(config *GraphConfig, hostname string, g *Graph, etcdO *EtcdWObject) bool {
 
-	var NoopMap = make(map[string]*Vertex)
+	var graph *Graph // new graph to return
-	var FileMap = make(map[string]*Vertex)
+	if g == nil {    // FIXME: how can we check for an empty graph?
-	var ServiceMap = make(map[string]*Vertex)
+		graph = NewGraph("Graph") // give graph a default name
-	var ExecMap = make(map[string]*Vertex)
+	} else {
+		graph = g.Copy() // same vertices, since they're pointers!
+	}
 
 	var lookup = make(map[string]map[string]*Vertex)
-	lookup["noop"] = NoopMap
-	lookup["file"] = FileMap
-	lookup["service"] = ServiceMap
-	lookup["exec"] = ExecMap
 
 	//log.Printf("%+v", config) // debug
 
-	g.SetName(config.Graph) // set graph name
+	// TODO: if defined (somehow)...
+	graph.SetName(config.Graph) // set graph name
 
-	var keep []*Vertex // list of vertex which are the same in new graph
+	var keep []*Vertex  // list of vertex which are the same in new graph
-
+	var resources []Res // list of resources to export
-	for _, t := range config.Types.Noop {
+	// use reflection to avoid duplicating code... better options welcome!
-		obj := NewNoopType(t.Name)
+	value := reflect.Indirect(reflect.ValueOf(config.Resources))
-		v := g.GetVertexMatch(obj)
+	vtype := value.Type()
-		if v == nil { // no match found
+	for i := 0; i < vtype.NumField(); i++ { // number of fields in struct
-			v = NewVertex(obj)
+		name := vtype.Field(i).Name // string of field name
-			g.AddVertex(v) // call standalone in case not part of an edge
+		field := value.FieldByName(name)
+		iface := field.Interface() // interface type of value
+		slice := reflect.ValueOf(iface)
+		// XXX: should we just drop these everywhere and have the kind strings be all lowercase?
+		kind := FirstToUpper(name)
+		if DEBUG {
+			log.Printf("Config: Processing: %v...", kind)
 		}
-		NoopMap[obj.Name] = v  // used for constructing edges
+		for j := 0; j < slice.Len(); j++ { // loop through resources of same kind
-		keep = append(keep, v) // append
+			x := slice.Index(j).Interface()
-	}
+			res, ok := x.(Res) // convert to Res type
-
+			if !ok {
-	for _, t := range config.Types.File {
+				return nil, fmt.Errorf("Config: Error: Can't convert: %v of type: %T to Res.", x, x)
-		// XXX: should we export based on a @@ prefix, or a metaparam
-		// like exported => true || exported => (host pattern)||(other pattern?)
-		if strings.HasPrefix(t.Name, "@@") { // exported resource
-			// add to etcd storage...
-			t.Name = t.Name[2:] //slice off @@
-			if !etcdO.EtcdPut(hostname, t.Name, "file", t) {
-				log.Printf("Problem exporting file resource %v.", t.Name)
-				continue
 			}
-		} else {
+			if noop {
-			obj := NewFileType(t.Name, t.Path, t.Dirname, t.Basename, t.Content, t.State)
+				res.Meta().Noop = noop
-			v := g.GetVertexMatch(obj)
+			}
-			if v == nil { // no match found
+			if _, exists := lookup[kind]; !exists {
-				v = NewVertex(obj)
+				lookup[kind] = make(map[string]*Vertex)
-				g.AddVertex(v) // call standalone in case not part of an edge
+			}
+			// XXX: should we export based on a @@ prefix, or a metaparam
+			// like exported => true || exported => (host pattern)||(other pattern?)
+			if !strings.HasPrefix(res.GetName(), "@@") { // not exported resource
+				// XXX: we don't have a way of knowing if any of the
+				// metaparams are undefined, and as a result to set the
+				// defaults that we want! I hate the go yaml parser!!!
+				v := graph.GetVertexMatch(res)
+				if v == nil { // no match found
+					res.Init()
+					v = NewVertex(res)
+					graph.AddVertex(v) // call standalone in case not part of an edge
+				}
+				lookup[kind][res.GetName()] = v // used for constructing edges
+				keep = append(keep, v)          // append
+
+			} else if !noop { // do not export any resources if noop
+				// store for addition to etcd storage...
+				res.SetName(res.GetName()[2:]) //slice off @@
+				res.setKind(kind)              // cheap init
+				resources = append(resources, res)
 			}
-			FileMap[obj.Name] = v  // used for constructing edges
-			keep = append(keep, v) // append
 		}
 	}
-
+	// store in etcd
-	for _, t := range config.Types.Service {
+	if err := EtcdSetResources(embdEtcd, config.Hostname, resources); err != nil {
-		obj := NewServiceType(t.Name, t.State, t.Startup)
+		return nil, fmt.Errorf("Config: Could not export resources: %v", err)
-		v := g.GetVertexMatch(obj)
-		if v == nil { // no match found
-			v = NewVertex(obj)
-			g.AddVertex(v) // call standalone in case not part of an edge
-		}
-		ServiceMap[obj.Name] = v // used for constructing edges
-		keep = append(keep, v)   // append
 	}
 
-	for _, t := range config.Types.Exec {
+	// lookup from etcd
-		obj := NewExecType(t.Name, t.Cmd, t.Shell, t.Timeout, t.WatchCmd, t.WatchShell, t.IfCmd, t.IfShell, t.PollInt, t.State)
+	var hostnameFilter []string // empty to get from everyone
-		v := g.GetVertexMatch(obj)
+	kindFilter := []string{}
-		if v == nil { // no match found
+	for _, t := range config.Collector {
-			v = NewVertex(obj)
+		// XXX: should we just drop these everywhere and have the kind strings be all lowercase?
-			g.AddVertex(v) // call standalone in case not part of an edge
+		kind := FirstToUpper(t.Kind)
-		}
+		kindFilter = append(kindFilter, kind)
-		ExecMap[obj.Name] = v  // used for constructing edges
-		keep = append(keep, v) // append
 	}
-
-	// lookup from etcd graph
 	// do all the graph look ups in one single step, so that if the etcd
 	// database changes, we don't have a partial state of affairs...
-	nodes, ok := etcdO.EtcdGet()
+	if len(kindFilter) > 0 { // if kindFilter is empty, don't need to do lookups!
-	if ok {
+		var err error
+		resources, err = EtcdGetResources(embdEtcd, hostnameFilter, kindFilter)
+		if err != nil {
+			return nil, fmt.Errorf("Config: Could not collect resources: %v", err)
+		}
+	}
+	for _, res := range resources {
+		matched := false
+		// see if we find a collect pattern that matches
 		for _, t := range config.Collector {
-			// XXX: use t.Type and optionally t.Pattern to collect from etcd storage
+			// XXX: should we just drop these everywhere and have the kind strings be all lowercase?
-			log.Printf("Collect: %v; Pattern: %v", t.Type, t.Pattern)
+			kind := FirstToUpper(t.Kind)
-
+			// use t.Kind and optionally t.Pattern to collect from etcd storage
-			for _, x := range etcdO.EtcdGetProcess(nodes, "file") {
+			log.Printf("Collect: %v; Pattern: %v", kind, t.Pattern)
-				var obj *FileType
-				if B64ToObj(x, &obj) != true {
-					log.Printf("Collect: File: %v not collected!", x)
-					continue
-				}
-				if t.Pattern != "" { // XXX: currently the pattern for files can only override the Dirname variable :P
-					obj.Dirname = t.Pattern
-				}
-
-				log.Printf("Collect: File: %v collected!", obj.GetName())
-
-				// XXX: similar to file add code:
-				v := g.GetVertexMatch(obj)
-				if v == nil { // no match found
-					obj.Init() // initialize go channels or things won't work!!!
-					v = NewVertex(obj)
-					g.AddVertex(v) // call standalone in case not part of an edge
-				}
-				FileMap[obj.GetName()] = v // used for constructing edges
-				keep = append(keep, v)     // append
 
+			// XXX: expand to more complex pattern matching here...
+			if res.Kind() != kind {
+				continue
 			}
 
+			if matched {
+				// we've already matched this resource, should we match again?
+				log.Printf("Config: Warning: Matching %v[%v] again!", kind, res.GetName())
+			}
+			matched = true
+
+			// collect resources but add the noop metaparam
+			if noop {
+				res.Meta().Noop = noop
+			}
+
+			if t.Pattern != "" { // XXX: simplistic for now
+				res.CollectPattern(t.Pattern) // res.Dirname = t.Pattern
+			}
+
+			log.Printf("Collect: %v[%v]: collected!", kind, res.GetName())
+
+			// XXX: similar to other resource add code:
+			if _, exists := lookup[kind]; !exists {
+				lookup[kind] = make(map[string]*Vertex)
+			}
+			v := graph.GetVertexMatch(res)
+			if v == nil { // no match found
+				res.Init() // initialize go channels or things won't work!!!
+				v = NewVertex(res)
+				graph.AddVertex(v) // call standalone in case not part of an edge
+			}
+			lookup[kind][res.GetName()] = v // used for constructing edges
+			keep = append(keep, v)          // append
+
+			//break // let's see if another resource even matches
 		}
 	}
 
 	// get rid of any vertices we shouldn't "keep" (that aren't in new graph)
-	for _, v := range g.GetVertices() {
+	for _, v := range graph.GetVertices() {
-		if !HasVertex(v, keep) {
+		if !VertexContains(v, keep) {
 			// wait for exit before starting new graph!
-			v.Type.SendEvent(eventExit, true, false)
+			v.SendEvent(eventExit, true, false)
-			g.DeleteVertex(v)
+			graph.DeleteVertex(v)
 		}
 	}
 
 	for _, e := range config.Edges {
-		if _, ok := lookup[e.From.Type]; !ok {
+		if _, ok := lookup[FirstToUpper(e.From.Kind)]; !ok {
-			return false
+			return nil, fmt.Errorf("Can't find 'from' resource!")
 		}
-		if _, ok := lookup[e.To.Type]; !ok {
+		if _, ok := lookup[FirstToUpper(e.To.Kind)]; !ok {
-			return false
+			return nil, fmt.Errorf("Can't find 'to' resource!")
 		}
-		if _, ok := lookup[e.From.Type][e.From.Name]; !ok {
+		if _, ok := lookup[FirstToUpper(e.From.Kind)][e.From.Name]; !ok {
-			return false
+			return nil, fmt.Errorf("Can't find 'from' name!")
 		}
-		if _, ok := lookup[e.To.Type][e.To.Name]; !ok {
+		if _, ok := lookup[FirstToUpper(e.To.Kind)][e.To.Name]; !ok {
-			return false
+			return nil, fmt.Errorf("Can't find 'to' name!")
 		}
-		g.AddEdge(lookup[e.From.Type][e.From.Name], lookup[e.To.Type][e.To.Name], NewEdge(e.Name))
+		graph.AddEdge(lookup[FirstToUpper(e.From.Kind)][e.From.Name], lookup[FirstToUpper(e.To.Kind)][e.To.Name], NewEdge(e.Name))
+	}
+
+	return graph, nil
+}
+
+// add edges to the vertex in a graph based on if it matches a uuid list
+func (g *Graph) addEdgesByMatchingUUIDS(v *Vertex, uuids []ResUUID) []bool {
+	// search for edges and see what matches!
+	var result []bool
+
+	// loop through each uuid, and see if it matches any vertex
+	for _, uuid := range uuids {
+		var found = false
+		// uuid is a ResUUID object
+		for _, vv := range g.GetVertices() { // search
+			if v == vv { // skip self
+				continue
+			}
+			if DEBUG {
+				log.Printf("Compile: AutoEdge: Match: %v[%v] with UUID: %v[%v]", vv.Kind(), vv.GetName(), uuid.Kind(), uuid.GetName())
+			}
+			// we must match to an effective UUID for the resource,
+			// that is to say, the name value of a res is a helpful
+			// handle, but it is not necessarily a unique identity!
+			// remember, resources can return multiple UUID's each!
+			if UUIDExistsInUUIDs(uuid, vv.GetUUIDs()) {
+				// add edge from: vv -> v
+				if uuid.Reversed() {
+					txt := fmt.Sprintf("AutoEdge: %v[%v] -> %v[%v]", vv.Kind(), vv.GetName(), v.Kind(), v.GetName())
+					log.Printf("Compile: Adding %v", txt)
+					g.AddEdge(vv, v, NewEdge(txt))
+				} else { // edges go the "normal" way, eg: pkg resource
+					txt := fmt.Sprintf("AutoEdge: %v[%v] -> %v[%v]", v.Kind(), v.GetName(), vv.Kind(), vv.GetName())
+					log.Printf("Compile: Adding %v", txt)
+					g.AddEdge(v, vv, NewEdge(txt))
+				}
+				found = true
+				break
+			}
+		}
+		result = append(result, found)
+	}
+	return result
+}
+
+// AutoEdges adds the automatic edges to the graph.
+func (g *Graph) AutoEdges() {
+	log.Println("Compile: Adding AutoEdges...")
+	for _, v := range g.GetVertices() { // for each vertexes autoedges
+		if !v.Meta().AutoEdge { // is the metaparam true?
+			continue
+		}
+		autoEdgeObj := v.AutoEdges()
+		if autoEdgeObj == nil {
+			log.Printf("%v[%v]: Config: No auto edges were found!", v.Kind(), v.GetName())
+			continue // next vertex
+		}
+
+		for { // while the autoEdgeObj has more uuids to add...
+			uuids := autoEdgeObj.Next() // get some!
+			if uuids == nil {
+				log.Printf("%v[%v]: Config: The auto edge list is empty!", v.Kind(), v.GetName())
+				break // inner loop
+			}
+			if DEBUG {
+				log.Println("Compile: AutoEdge: UUIDS:")
+				for i, u := range uuids {
+					log.Printf("Compile: AutoEdge: UUID%d: %v", i, u)
+				}
+			}
+
+			// match and add edges
+			result := g.addEdgesByMatchingUUIDS(v, uuids)
+
+			// report back, and find out if we should continue
+			if !autoEdgeObj.Test(result) {
+				break
+			}
+		}
+	}
+}
+
+// AutoGrouper is the required interface to implement for an autogroup algorithm
+type AutoGrouper interface {
+	// listed in the order these are typically called in...
+	name() string                                  // friendly identifier
+	init(*Graph) error                             // only call once
+	vertexNext() (*Vertex, *Vertex, error)         // mostly algorithmic
+	vertexCmp(*Vertex, *Vertex) error              // can we merge these ?
+	vertexMerge(*Vertex, *Vertex) (*Vertex, error) // vertex merge fn to use
+	edgeMerge(*Edge, *Edge) *Edge                  // edge merge fn to use
+	vertexTest(bool) (bool, error)                 // call until false
+}
+
+// baseGrouper is the base type for implementing the AutoGrouper interface
+type baseGrouper struct {
+	graph    *Graph    // store a pointer to the graph
+	vertices []*Vertex // cached list of vertices
+	i        int
+	j        int
+	done     bool
+}
+
+// name provides a friendly name for the logs to see
+func (ag *baseGrouper) name() string {
+	return "baseGrouper"
+}
+
+// init is called only once and before using other AutoGrouper interface methods
+// the name method is the only exception: call it any time without side effects!
+func (ag *baseGrouper) init(g *Graph) error {
+	if ag.graph != nil {
+		return fmt.Errorf("The init method has already been called!")
+	}
+	ag.graph = g                               // pointer
+	ag.vertices = ag.graph.GetVerticesSorted() // cache in deterministic order!
+	ag.i = 0
+	ag.j = 0
+	if len(ag.vertices) == 0 { // empty graph
+		ag.done = true
+		return nil
+	}
+	return nil
+}
+
+// vertexNext is a simple iterator that loops through vertex (pair) combinations
+// an intelligent algorithm would selectively offer only valid pairs of vertices
+// these should satisfy logical grouping requirements for the autogroup designs!
+// the desired algorithms can override, but keep this method as a base iterator!
+func (ag *baseGrouper) vertexNext() (v1, v2 *Vertex, err error) {
+	// this does a for v... { for w... { return v, w }} but stepwise!
+	l := len(ag.vertices)
+	if ag.i < l {
+		v1 = ag.vertices[ag.i]
+	}
+	if ag.j < l {
+		v2 = ag.vertices[ag.j]
+	}
+
+	// in case the vertex was deleted
+	if !ag.graph.HasVertex(v1) {
+		v1 = nil
+	}
+	if !ag.graph.HasVertex(v2) {
+		v2 = nil
+	}
+
+	// two nested loops...
+	if ag.j < l {
+		ag.j++
+	}
+	if ag.j == l {
+		ag.j = 0
+		if ag.i < l {
+			ag.i++
+		}
+		if ag.i == l {
+			ag.done = true
+		}
+	}
+
+	return
+}
+
+func (ag *baseGrouper) vertexCmp(v1, v2 *Vertex) error {
+	if v1 == nil || v2 == nil {
+		return fmt.Errorf("Vertex is nil!")
+	}
+	if v1 == v2 { // skip yourself
+		return fmt.Errorf("Vertices are the same!")
+	}
+	if v1.Kind() != v2.Kind() { // we must group similar kinds
+		// TODO: maybe future resources won't need this limitation?
+		return fmt.Errorf("The two resources aren't the same kind!")
+	}
+	// someone doesn't want to group!
+	if !v1.Meta().AutoGroup || !v2.Meta().AutoGroup {
+		return fmt.Errorf("One of the autogroup flags is false!")
+	}
+	if v1.Res.IsGrouped() { // already grouped!
+		return fmt.Errorf("Already grouped!")
+	}
+	if len(v2.Res.GetGroup()) > 0 { // already has children grouped!
+		return fmt.Errorf("Already has groups!")
+	}
+	if !v1.Res.GroupCmp(v2.Res) { // resource groupcmp failed!
+		return fmt.Errorf("The GroupCmp failed!")
+	}
+	return nil // success
+}
+
+func (ag *baseGrouper) vertexMerge(v1, v2 *Vertex) (v *Vertex, err error) {
+	// NOTE: it's important to use w.Res instead of w, b/c
+	// the w by itself is the *Vertex obj, not the *Res obj
+	// which is contained within it! They both satisfy the
+	// Res interface, which is why both will compile! :(
+	err = v1.Res.GroupRes(v2.Res) // GroupRes skips stupid groupings
+	return                        // success or fail, and no need to merge the actual vertices!
+}
+
+func (ag *baseGrouper) edgeMerge(e1, e2 *Edge) *Edge {
+	return e1 // noop
+}
+
+// vertexTest processes the results of the grouping for the algorithm to know
+// return an error if something went horribly wrong, and bool false to stop
+func (ag *baseGrouper) vertexTest(b bool) (bool, error) {
+	// NOTE: this particular baseGrouper version doesn't track what happens
+	// because since we iterate over every pair, we don't care which merge!
+	if ag.done {
+		return false, nil
+	}
+	return true, nil
+}
+
+// TODO: this algorithm may not be correct in all cases. replace if needed!
+type nonReachabilityGrouper struct {
+	baseGrouper // "inherit" what we want, and reimplement the rest
+}
+
+func (ag *nonReachabilityGrouper) name() string {
+	return "nonReachabilityGrouper"
+}
+
+// this algorithm relies on the observation that if there's a path from a to b,
+// then they *can't* be merged (b/c of the existing dependency) so therefore we
+// merge anything that *doesn't* satisfy this condition or that of the reverse!
+func (ag *nonReachabilityGrouper) vertexNext() (v1, v2 *Vertex, err error) {
+	for {
+		v1, v2, err = ag.baseGrouper.vertexNext() // get all iterable pairs
+		if err != nil {
+			log.Fatalf("Error running autoGroup(vertexNext): %v", err)
+		}
+
+		if v1 != v2 { // ignore self cmp early (perf optimization)
+			// if NOT reachable, they're viable...
+			out1 := ag.graph.Reachability(v1, v2)
+			out2 := ag.graph.Reachability(v2, v1)
+			if len(out1) == 0 && len(out2) == 0 {
+				return // return v1 and v2, they're viable
+			}
+		}
+
+		// if we got here, it means we're skipping over this candidate!
+		if ok, err := ag.baseGrouper.vertexTest(false); err != nil {
+			log.Fatalf("Error running autoGroup(vertexTest): %v", err)
+		} else if !ok {
+			return nil, nil, nil // done!
+		}
+
+		// the vertexTest passed, so loop and try with a new pair...
+	}
+}
+
+// autoGroup is the mechanical auto group "runner" that runs the interface spec
+func (g *Graph) autoGroup(ag AutoGrouper) chan string {
+	strch := make(chan string) // output log messages here
+	go func(strch chan string) {
+		strch <- fmt.Sprintf("Compile: Grouping: Algorithm: %v...", ag.name())
+		if err := ag.init(g); err != nil {
+			log.Fatalf("Error running autoGroup(init): %v", err)
+		}
+
+		for {
+			var v, w *Vertex
+			v, w, err := ag.vertexNext() // get pair to compare
+			if err != nil {
+				log.Fatalf("Error running autoGroup(vertexNext): %v", err)
+			}
+			merged := false
+			// save names since they change during the runs
+			vStr := fmt.Sprintf("%s", v) // valid even if it is nil
+			wStr := fmt.Sprintf("%s", w)
+
+			if err := ag.vertexCmp(v, w); err != nil { // cmp ?
+				if DEBUG {
+					strch <- fmt.Sprintf("Compile: Grouping: !GroupCmp for: %s into %s", wStr, vStr)
+				}
+
+				// remove grouped vertex and merge edges (res is safe)
+			} else if err := g.VertexMerge(v, w, ag.vertexMerge, ag.edgeMerge); err != nil { // merge...
+				strch <- fmt.Sprintf("Compile: Grouping: !VertexMerge for: %s into %s", wStr, vStr)
+
+			} else { // success!
+				strch <- fmt.Sprintf("Compile: Grouping: Success for: %s into %s", wStr, vStr)
+				merged = true // woo
+			}
+
+			// did these get used?
+			if ok, err := ag.vertexTest(merged); err != nil {
+				log.Fatalf("Error running autoGroup(vertexTest): %v", err)
+			} else if !ok {
+				break // done!
+			}
+		}
+
+		close(strch)
+		return
+	}(strch) // call function
+	return strch
+}
+
+// AutoGroup runs the auto grouping on the graph and prints out log messages
+func (g *Graph) AutoGroup() {
+	// receive log messages from channel...
+	// this allows test cases to avoid printing them when they're unwanted!
+	// TODO: this algorithm may not be correct in all cases. replace if needed!
+	for str := range g.autoGroup(&nonReachabilityGrouper{}) {
+		log.Println(str)
 	}
-	return true
 }

configwatch.go

@@ -24,10 +24,75 @@ import (
 	"math"
 	"path"
 	"strings"
+	"sync"
 	"syscall"
 )
 
-// XXX: it would be great if we could reuse code between this and the file type
+// ConfigWatcher returns events on a channel anytime one of its files events.
+type ConfigWatcher struct {
+	ch        chan string
+	wg        sync.WaitGroup
+	closechan chan struct{}
+}
+
+// NewConfigWatcher creates a new ConfigWatcher struct.
+func NewConfigWatcher() *ConfigWatcher {
+	return &ConfigWatcher{
+		ch:        make(chan string),
+		closechan: make(chan struct{}),
+	}
+}
+
+// The Add method adds a new file path to watch for events on.
+func (obj *ConfigWatcher) Add(file ...string) {
+	if len(file) == 0 {
+		return
+	}
+	if len(file) > 1 {
+		for _, f := range file { // add all the files...
+			obj.Add(f) // recurse
+		}
+		return
+	}
+	// otherwise, add the one file passed in...
+	obj.wg.Add(1)
+	go func() {
+		defer obj.wg.Done()
+		ch := ConfigWatch(file[0])
+		for {
+			select {
+			case <-ch:
+				obj.ch <- file[0]
+				continue
+			case <-obj.closechan:
+				return
+			}
+		}
+	}()
+}
+
+// Events returns a channel to listen on for file events. It closes when it is
+// emptied after the Close() method is called. You can test for closure with the
+// f, more := <-obj.Events() pattern.
+func (obj *ConfigWatcher) Events() chan string {
+	return obj.ch
+}
+
+// Close shuts down the ConfigWatcher object. It closes the Events channel after
+// all the currently pending events have been emptied.
+func (obj *ConfigWatcher) Close() {
+	if obj.ch == nil {
+		return
+	}
+	close(obj.closechan)
+	obj.wg.Wait() // wait until everyone is done sending on obj.ch
+	//obj.ch <- "" // send finished message
+	close(obj.ch)
+	obj.ch = nil
+}
+
+// ConfigWatch writes on the channel everytime an event is seen for the path.
+// XXX: it would be great if we could reuse code between this and the file resource
 // XXX: patch this to submit it as part of go-fsnotify if they're interested...
 func ConfigWatch(file string) chan bool {
 	ch := make(chan bool)
@@ -51,8 +116,9 @@ func ConfigWatch(file string) chan bool {
 			if current == "" { // the empty string top is the root dir ("/")
 				current = "/"
 			}
-			log.Printf("Watching: %v", current) // attempting to watch...
+			if DEBUG {
-
+				log.Printf("Watching: %v", current) // attempting to watch...
+			}
 			// initialize in the loop so that we can reset on rm-ed handles
 			err = watcher.Add(current)
 			if err != nil {
@@ -61,7 +127,7 @@ func ConfigWatch(file string) chan bool {
 				} else if err == syscall.ENOSPC {
 					// XXX: occasionally: no space left on device,
 					// XXX: probably due to lack of inotify watches
-					log.Printf("Lack of watches for config(%v) error: %+v", file, err.Error) // 0x408da0
+					log.Printf("Out of inotify watches for config(%v)", file)
 					log.Fatal(err)
 				} else {
 					log.Printf("Unknown config(%v) error:", file)
@@ -97,7 +163,10 @@ func ConfigWatch(file string) chan bool {
 				// if we have what we wanted, awesome, send an event...
 				if event.Name == safename {
 					//log.Println("Event!")
-					send = true
+					// TODO: filter out some of the events, is Write a sufficient minimum?
+					if event.Op&fsnotify.Write == fsnotify.Write {
+						send = true
+					}
 
 					// file removed, move the watch upwards
 					if deltaDepth >= 0 && (event.Op&fsnotify.Remove == fsnotify.Remove) {
@@ -138,7 +207,7 @@ func ConfigWatch(file string) chan bool {
 				}
 
 			case err := <-watcher.Errors:
-				log.Println("error:", err)
+				log.Printf("error: %v", err)
 				log.Fatal(err)
 
 			}

converger.go

@@ -0,0 +1,379 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"fmt"
+	"sync"
+	"time"
+)
+
+// TODO: we could make a new function that masks out the state of certain
+// UUID's, but at the moment the new Timer code has obsoleted the need...
+
+// Converger is the general interface for implementing a convergence watcher
+type Converger interface { // TODO: need a better name
+	Register() ConvergerUUID
+	IsConverged(ConvergerUUID) bool         // is the UUID converged ?
+	SetConverged(ConvergerUUID, bool) error // set the converged state of the UUID
+	Unregister(ConvergerUUID)
+	Start()
+	Pause()
+	Loop(bool)
+	ConvergedTimer(ConvergerUUID) <-chan time.Time
+	Status() map[uint64]bool
+	Timeout() int                // returns the timeout that this was created with
+	SetStateFn(func(bool) error) // sets the stateFn
+}
+
+// ConvergerUUID is the interface resources can use to notify with if converged
+// you'll need to use part of the Converger interface to Register initially too
+type ConvergerUUID interface {
+	ID() uint64   // get Id
+	Name() string // get a friendly name
+	SetName(string)
+	IsValid() bool // has Id been initialized ?
+	InvalidateID() // set Id to nil
+	IsConverged() bool
+	SetConverged(bool) error
+	Unregister()
+	ConvergedTimer() <-chan time.Time
+	StartTimer() (func() error, error) // cancellable is the same as StopTimer()
+	ResetTimer() error                 // resets counter to zero
+	StopTimer() error
+}
+
+// converger is an implementation of the Converger interface
+type converger struct {
+	timeout   int              // must be zero (instant) or greater seconds to run
+	stateFn   func(bool) error // run on converged state changes with state bool
+	converged bool             // did we converge (state changes of this run Fn)
+	channel   chan struct{}    // signal here to run an isConverged check
+	control   chan bool        // control channel for start/pause
+	mutex     sync.RWMutex     // used for controlling access to status and lastid
+	lastid    uint64
+	status    map[uint64]bool
+}
+
+// convergerUUID is an implementation of the ConvergerUUID interface
+type convergerUUID struct {
+	converger Converger
+	id        uint64
+	name      string // user defined, friendly name
+	mutex     sync.Mutex
+	timer     chan struct{}
+	running   bool // is the above timer running?
+}
+
+// NewConverger builds a new converger struct
+func NewConverger(timeout int, stateFn func(bool) error) *converger {
+	return &converger{
+		timeout: timeout,
+		stateFn: stateFn,
+		channel: make(chan struct{}),
+		control: make(chan bool),
+		lastid:  0,
+		status:  make(map[uint64]bool),
+	}
+}
+
+// Register assigns a ConvergerUUID to the caller
+func (obj *converger) Register() ConvergerUUID {
+	obj.mutex.Lock()
+	defer obj.mutex.Unlock()
+	obj.lastid++
+	obj.status[obj.lastid] = false // initialize as not converged
+	return &convergerUUID{
+		converger: obj,
+		id:        obj.lastid,
+		name:      fmt.Sprintf("%d", obj.lastid), // some default
+		timer:     nil,
+		running:   false,
+	}
+}
+
+// IsConverged gets the converged status of a uuid
+func (obj *converger) IsConverged(uuid ConvergerUUID) bool {
+	if !uuid.IsValid() {
+		panic(fmt.Sprintf("Id of ConvergerUUID(%s) is nil!", uuid.Name()))
+	}
+	obj.mutex.RLock()
+	isConverged, found := obj.status[uuid.ID()] // lookup
+	obj.mutex.RUnlock()
+	if !found {
+		panic("Id of ConvergerUUID is unregistered!")
+	}
+	return isConverged
+}
+
+// SetConverged updates the converger with the converged state of the UUID
+func (obj *converger) SetConverged(uuid ConvergerUUID, isConverged bool) error {
+	if !uuid.IsValid() {
+		return fmt.Errorf("Id of ConvergerUUID(%s) is nil!", uuid.Name())
+	}
+	obj.mutex.Lock()
+	if _, found := obj.status[uuid.ID()]; !found {
+		panic("Id of ConvergerUUID is unregistered!")
+	}
+	obj.status[uuid.ID()] = isConverged // set
+	obj.mutex.Unlock()                  // unlock *before* poke or deadlock!
+	if isConverged != obj.converged {   // only poke if it would be helpful
+		// run in a go routine so that we never block... just queue up!
+		// this allows us to send events, even if we haven't started...
+		go func() { obj.channel <- struct{}{} }()
+	}
+	return nil
+}
+
+// isConverged returns true if *every* registered uuid has converged
+func (obj *converger) isConverged() bool {
+	obj.mutex.RLock() // take a read lock
+	defer obj.mutex.RUnlock()
+	for _, v := range obj.status {
+		if !v { // everyone must be converged for this to be true
+			return false
+		}
+	}
+	return true
+}
+
+// Unregister dissociates the ConvergedUUID from the converged checking
+func (obj *converger) Unregister(uuid ConvergerUUID) {
+	if !uuid.IsValid() {
+		panic(fmt.Sprintf("Id of ConvergerUUID(%s) is nil!", uuid.Name()))
+	}
+	obj.mutex.Lock()
+	uuid.StopTimer() // ignore any errors
+	delete(obj.status, uuid.ID())
+	obj.mutex.Unlock()
+	uuid.InvalidateID()
+}
+
+// Start causes a Converger object to start or resume running
+func (obj *converger) Start() {
+	obj.control <- true
+}
+
+// Pause causes a Converger object to stop running temporarily
+func (obj *converger) Pause() { // FIXME: add a sync ACK on pause before return
+	obj.control <- false
+}
+
+// Loop is the main loop for a Converger object; it usually runs in a goroutine
+// TODO: we could eventually have each resource tell us as soon as it converges
+// and then keep track of the time delays here, to avoid callers needing select
+// NOTE: when we have very short timeouts, if we start before all the resources
+// have joined the map, then it might appears as if we converged before we did!
+func (obj *converger) Loop(startPaused bool) {
+	if obj.control == nil {
+		panic("Converger not initialized correctly")
+	}
+	if startPaused { // start paused without racing
+		select {
+		case e := <-obj.control:
+			if !e {
+				panic("Converger expected true!")
+			}
+		}
+	}
+	for {
+		select {
+		case e := <-obj.control: // expecting "false" which means pause!
+			if e {
+				panic("Converger expected false!")
+			}
+			// now i'm paused...
+			select {
+			case e := <-obj.control:
+				if !e {
+					panic("Converger expected true!")
+				}
+				// restart
+				// kick once to refresh the check...
+				go func() { obj.channel <- struct{}{} }()
+				continue
+			}
+
+		case <-obj.channel:
+			if !obj.isConverged() {
+				if obj.converged { // we're doing a state change
+					if obj.stateFn != nil {
+						// call an arbitrary function
+						if err := obj.stateFn(false); err != nil {
+							// FIXME: what to do on error ?
+						}
+					}
+				}
+				obj.converged = false
+				continue
+			}
+
+			// we have converged!
+			if obj.timeout >= 0 { // only run if timeout is valid
+				if !obj.converged { // we're doing a state change
+					if obj.stateFn != nil {
+						// call an arbitrary function
+						if err := obj.stateFn(true); err != nil {
+							// FIXME: what to do on error ?
+						}
+					}
+				}
+			}
+			obj.converged = true
+			// loop and wait again...
+		}
+	}
+}
+
+// ConvergedTimer adds a timeout to a select call and blocks until then
+// TODO: this means we could eventually have per resource converged timeouts
+func (obj *converger) ConvergedTimer(uuid ConvergerUUID) <-chan time.Time {
+	// be clever: if i'm already converged, this timeout should block which
+	// avoids unnecessary new signals being sent! this avoids fast loops if
+	// we have a low timeout, or in particular a timeout == 0
+	if uuid.IsConverged() {
+		// blocks the case statement in select forever!
+		return TimeAfterOrBlock(-1)
+	}
+	return TimeAfterOrBlock(obj.timeout)
+}
+
+// Status returns a map of the converged status of each UUID.
+func (obj *converger) Status() map[uint64]bool {
+	status := make(map[uint64]bool)
+	obj.mutex.RLock() // take a read lock
+	defer obj.mutex.RUnlock()
+	for k, v := range obj.status { // make a copy to avoid the mutex
+		status[k] = v
+	}
+	return status
+}
+
+// Timeout returns the timeout in seconds that converger was created with. This
+// is useful to avoid passing in the timeout value separately when you're
+// already passing in the Converger struct.
+func (obj *converger) Timeout() int {
+	return obj.timeout
+}
+
+// SetStateFn sets the state function to be run on change of converged state.
+func (obj *converger) SetStateFn(stateFn func(bool) error) {
+	obj.stateFn = stateFn
+}
+
+// Id returns the unique id of this UUID object
+func (obj *convergerUUID) ID() uint64 {
+	return obj.id
+}
+
+// Name returns a user defined name for the specific convergerUUID.
+func (obj *convergerUUID) Name() string {
+	return obj.name
+}
+
+// SetName sets a user defined name for the specific convergerUUID.
+func (obj *convergerUUID) SetName(name string) {
+	obj.name = name
+}
+
+// IsValid tells us if the id is valid or has already been destroyed
+func (obj *convergerUUID) IsValid() bool {
+	return obj.id != 0 // an id of 0 is invalid
+}
+
+// InvalidateID marks the id as no longer valid
+func (obj *convergerUUID) InvalidateID() {
+	obj.id = 0 // an id of 0 is invalid
+}
+
+// IsConverged is a helper function to the regular IsConverged method
+func (obj *convergerUUID) IsConverged() bool {
+	return obj.converger.IsConverged(obj)
+}
+
+// SetConverged is a helper function to the regular SetConverged notification
+func (obj *convergerUUID) SetConverged(isConverged bool) error {
+	return obj.converger.SetConverged(obj, isConverged)
+}
+
+// Unregister is a helper function to unregister myself
+func (obj *convergerUUID) Unregister() {
+	obj.converger.Unregister(obj)
+}
+
+// ConvergedTimer is a helper around the regular ConvergedTimer method
+func (obj *convergerUUID) ConvergedTimer() <-chan time.Time {
+	return obj.converger.ConvergedTimer(obj)
+}
+
+// StartTimer runs an invisible timer that automatically converges on timeout.
+func (obj *convergerUUID) StartTimer() (func() error, error) {
+	obj.mutex.Lock()
+	if !obj.running {
+		obj.timer = make(chan struct{})
+		obj.running = true
+	} else {
+		obj.mutex.Unlock()
+		return obj.StopTimer, fmt.Errorf("Timer already started!")
+	}
+	obj.mutex.Unlock()
+	go func() {
+		for {
+			select {
+			case _, ok := <-obj.timer: // reset signal channel
+				if !ok { // channel is closed
+					return // false to exit
+				}
+				obj.SetConverged(false)
+
+			case <-obj.ConvergedTimer():
+				obj.SetConverged(true) // converged!
+				select {
+				case _, ok := <-obj.timer: // reset signal channel
+					if !ok { // channel is closed
+						return // false to exit
+					}
+				}
+			}
+		}
+	}()
+	return obj.StopTimer, nil
+}
+
+// ResetTimer resets the counter to zero if using a StartTimer internally.
+func (obj *convergerUUID) ResetTimer() error {
+	obj.mutex.Lock()
+	defer obj.mutex.Unlock()
+	if obj.running {
+		obj.timer <- struct{}{} // send the reset message
+		return nil
+	}
+	return fmt.Errorf("Timer hasn't been started!")
+}
+
+// StopTimer stops the running timer permanently until a StartTimer is run.
+func (obj *convergerUUID) StopTimer() error {
+	obj.mutex.Lock()
+	defer obj.mutex.Unlock()
+	if !obj.running {
+		return fmt.Errorf("Timer isn't running!")
+	}
+	close(obj.timer)
+	obj.running = false
+	return nil
+}

docker/Dockerfile

@@ -0,0 +1,22 @@
+FROM golang:1.6.2
+
+MAINTAINER Michał Czeraszkiewicz <contact@czerasz.com>
+
+# Set the reset cache variable
+# Read more here: http://czerasz.com/2014/11/13/docker-tip-and-tricks/#use-refreshedat-variable-for-better-cache-control
+ENV REFRESHED_AT 2016-05-10
+
+# Update the package list to be able to use required packages
+RUN apt-get update
+
+# Change the working directory
+WORKDIR /go/src/mgmt
+
+# Copy all the files to the working directory
+COPY . /go/src/mgmt
+
+# Install dependencies
+RUN make deps
+
+# Build the binary
+RUN make build

docker/Dockerfile.development

@@ -0,0 +1,31 @@
+FROM golang:1.6.2
+
+MAINTAINER Michał Czeraszkiewicz <contact@czerasz.com>
+
+# Set the reset cache variable
+# Read more here: http://czerasz.com/2014/11/13/docker-tip-and-tricks/#use-refreshedat-variable-for-better-cache-control
+ENV REFRESHED_AT 2016-05-14
+
+RUN apt-get update
+
+# Setup User to match Host User
+# Give the nre user superuser permissions
+ARG USER_ID=1000
+ARG GROUP_ID=1000
+ARG USER_NAME=mgmt
+ARG GROUP_NAME=$USER_NAME
+RUN groupadd --gid $GROUP_ID $GROUP_NAME && \
+	useradd --create-home --home /home/$USER_NAME --uid ${USER_ID} --gid $GROUP_NAME --groups sudo $USER_NAME && \
+	echo "$USER_NAME ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
+
+# Copy all the files to the working directory
+COPY . /home/$USER_NAME/mgmt
+
+# Change working directory
+WORKDIR /home/$USER_NAME/mgmt
+
+# Install dependencies
+RUN make deps
+
+# Change user
+USER ${USER_NAME}

docker/scripts/build

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+script_directory="$( cd "$( dirname "$0" )" && pwd )"
+project_directory=$script_directory/../..
+
+# Specify the Docker image name
+image_name='purpleidea/mgmt'
+
+# Build the image which contains the compiled binary
+docker build -t $image_name \
+	--file=$project_directory/docker/Dockerfile $project_directory
+
+# Remove the container if it already exists
+docker rm -f mgmt-export 2> /dev/null
+
+# Start the container in background so we can "copy out" the binary
+docker run -d --name=mgmt-export $image_name bash -c 'while true; sleep 1000; done'
+
+# Remove the current binary
+rm $project_directory/mgmt 2> /dev/null
+
+# Get the binary from the container
+docker cp mgmt-export:/go/src/mgmt/mgmt $project_directory/mgmt
+
+# Remove the container
+docker rm -f mgmt-export 2> /dev/null

docker/scripts/build-development

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+# Stop on any error
+set -e
+
+script_directory="$( cd "$( dirname "$0" )" && pwd )"
+project_directory=$script_directory/../..
+
+# Specify the Docker image name
+image_name='purpleidea/mgmt:development'
+
+# Build the image
+docker build -t $image_name \
+	--file=$project_directory/docker/Dockerfile.development $project_directory

docker/scripts/run-development

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+# Stop on any error
+set -e
+
+script_directory="$( cd "$( dirname "$0" )" && pwd )"
+project_directory=$script_directory/../..
+
+# Specify the Docker image name
+image_name='purpleidea/mgmt:development'
+
+# Run container in development mode
+docker run --rm --name=mgm_development --user=mgmt \
+	-v $project_directory:/home/mgmt/mgmt \
+	-it $image_name bash

event.go

@@ -21,35 +21,76 @@ package main
 type eventName int
 
 const (
-	eventExit eventName = iota
+	eventNil eventName = iota
+	eventExit
 	eventStart
 	eventPause
 	eventPoke
 	eventBackPoke
 )
 
+// Resp is a channel to be used for boolean responses.
+type Resp chan bool
+
+// Event is the main struct that stores event information and responses.
 type Event struct {
 	Name eventName
-	Resp chan bool // channel to send an ack response on, nil to skip
+	Resp Resp // channel to send an ack response on, nil to skip
 	//Wg   *sync.WaitGroup // receiver barrier to Wait() for everyone else on
 	Msg      string // some words for fun
 	Activity bool   // did something interesting happen?
 }
 
-// send a single acknowledgement on the channel if one was requested
+// ACK sends a single acknowledgement on the channel if one was requested.
 func (event *Event) ACK() {
 	if event.Resp != nil { // if they've requested an ACK
-		event.Resp <- true // send ACK
+		event.Resp.ACK()
 	}
 }
 
+// NACK sends a negative acknowledgement message on the channel if one was requested.
 func (event *Event) NACK() {
-	if event.Resp != nil { // if they've requested an ACK
+	if event.Resp != nil { // if they've requested a NACK
-		event.Resp <- false // send NACK
+		event.Resp.NACK()
 	}
 }
 
-// get the activity value
+// NewResp is just a helper to return the right type of response channel.
+func NewResp() Resp {
+	resp := make(chan bool)
+	return resp
+}
+
+// ACK sends a true value to resp.
+func (resp Resp) ACK() {
+	if resp != nil {
+		resp <- true
+	}
+}
+
+// NACK sends a false value to resp.
+func (resp Resp) NACK() {
+	if resp != nil {
+		resp <- false
+	}
+}
+
+// Wait waits for any response from a Resp channel and returns it.
+func (resp Resp) Wait() bool {
+	return <-resp
+}
+
+// ACKWait waits for a +ive Ack from a Resp channel.
+func (resp Resp) ACKWait() {
+	for {
+		// wait until true value
+		if resp.Wait() {
+			return
+		}
+	}
+}
+
+// GetActivity returns the activity value.
 func (event *Event) GetActivity() bool {
 	return event.Activity
 }

examples/autoedges1.yaml

@@ -0,0 +1,19 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1
+    meta:
+      autoedge: true
+    path: "/tmp/foo/bar/f1"
+    content: |
+      i am f1
+    state: exists
+  - name: file2
+    meta:
+      autoedge: true
+    path: "/tmp/foo/"
+    content: |
+      i am f2
+    state: exists
+edges: []

examples/autoedges2.yaml

@@ -0,0 +1,24 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1
+    meta:
+      autoedge: true
+    path: "/etc/drbd.conf"
+    content: |
+      # this is an mgmt test
+    state: exists
+  - name: file2
+    meta:
+      autoedge: true
+    path: "/tmp/foo/"
+    content: |
+      i am f2
+    state: exists
+  pkg:
+  - name: drbd-utils
+    meta:
+      autoedge: true
+    state: installed
+edges: []

examples/autoedges3.yaml

@@ -0,0 +1,29 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: drbd-utils
+    meta:
+      autoedge: true
+    state: installed
+  file:
+  - name: file1
+    meta:
+      autoedge: true
+    path: "/etc/drbd.conf"
+    content: |
+      # this is an mgmt test
+    state: exists
+  - name: file2
+    meta:
+      autoedge: true
+    path: "/etc/drbd.d/"
+    content: |
+      i am a directory
+    state: exists
+  svc:
+  - name: drbd
+    meta:
+      autoedge: true
+    state: stopped
+edges: []

examples/autogroup1.yaml

@@ -0,0 +1,21 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: drbd-utils
+    meta:
+      autogroup: false
+    state: installed
+  - name: powertop
+    meta:
+      autogroup: true
+    state: installed
+  - name: sl
+    meta:
+      autogroup: true
+    state: installed
+  - name: cowsay
+    meta:
+      autogroup: true
+    state: installed
+edges: []

examples/autogroup2.yaml

@@ -0,0 +1,17 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: powertop
+    meta:
+      autogroup: true
+    state: installed
+  - name: sl
+    meta:
+      autogroup: true
+    state: installed
+  - name: cowsay
+    meta:
+      autogroup: true
+    state: installed
+edges: []

examples/etcd1a.yaml

@@ -0,0 +1,18 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1a
+    path: "/tmp/mgmtA/f1a"
+    content: |
+      i am f1
+    state: exists
+  - name: "@@file2a"
+    path: "/tmp/mgmtA/f2a"
+    content: |
+      i am f2, exported from host A
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtA/"
+edges: []

examples/etcd1b.yaml

@@ -0,0 +1,18 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1b
+    path: "/tmp/mgmtB/f1b"
+    content: |
+      i am f1
+    state: exists
+  - name: "@@file2b"
+    path: "/tmp/mgmtB/f2b"
+    content: |
+      i am f2, exported from host B
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtB/"
+edges: []

examples/etcd1c.yaml

@@ -0,0 +1,18 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1c
+    path: "/tmp/mgmtC/f1c"
+    content: |
+      i am f1
+    state: exists
+  - name: "@@file2c"
+    path: "/tmp/mgmtC/f2c"
+    content: |
+      i am f2, exported from host C
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtC/"
+edges: []

examples/etcd1d.yaml

@@ -0,0 +1,18 @@
+---
+graph: mygraph
+resources:
+  file:
+  - name: file1d
+    path: "/tmp/mgmtD/f1d"
+    content: |
+      i am f1
+    state: exists
+  - name: "@@file2d"
+    path: "/tmp/mgmtD/f2d"
+    content: |
+      i am f2, exported from host D
+    state: exists
+collect:
+- kind: file
+  pattern: "/tmp/mgmtD/"
+edges: []

examples/exec1.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   exec:
   - name: exec1
     cmd: sleep 10s
@@ -45,15 +45,15 @@ types:
 edges:
 - name: e1
   from:
-    type: exec
+    kind: exec
     name: exec1
   to:
-    type: exec
+    kind: exec
     name: exec2
 - name: e2
   from:
-    type: exec
+    kind: exec
     name: exec2
   to:
-    type: exec
+    kind: exec
     name: exec3

examples/exec1a.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   exec:
   - name: exec1
     cmd: sleep 10s
@@ -25,8 +25,8 @@ types:
 edges:
 - name: e1
   from:
-    type: exec
+    kind: exec
     name: exec1
   to:
-    type: exec
+    kind: exec
     name: exec2

examples/exec1b.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   exec:
   - name: exec1
     cmd: sleep 10s
@@ -25,8 +25,8 @@ types:
 edges:
 - name: e1
   from:
-    type: exec
+    kind: exec
     name: exec1
   to:
-    type: exec
+    kind: exec
     name: exec2

examples/exec1c.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   exec:
   - name: exec1
     cmd: echo hello from exec1
@@ -25,8 +25,8 @@ types:
 edges:
 - name: e1
   from:
-    type: exec
+    kind: exec
     name: exec1
   to:
-    type: exec
+    kind: exec
     name: exec2

examples/exec1d.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   exec:
   - name: exec1
     cmd: echo hello from exec1

examples/exec2.yaml

@@ -0,0 +1,83 @@
+---
+graph: mygraph
+resources:
+  exec:
+  - name: exec1
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec3
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec4
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec5
+    cmd: sleep 15s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec2
+- name: e2
+  from:
+    kind: exec
+    name: exec1
+  to:
+    kind: exec
+    name: exec3
+- name: e3
+  from:
+    kind: exec
+    name: exec2
+  to:
+    kind: exec
+    name: exec4
+- name: e4
+  from:
+    kind: exec
+    name: exec3
+  to:
+    kind: exec
+    name: exec4

examples/exec3.yaml

@@ -0,0 +1,59 @@
+---
+graph: parallel
+resources:
+  exec:
+  - name: pkg10
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: svc10
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec10
+    cmd: sleep 10s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: pkg15
+    cmd: sleep 15s
+    shell: ''
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: exec
+    name: pkg10
+  to:
+    kind: exec
+    name: svc10
+- name: e2
+  from:
+    kind: exec
+    name: svc10
+  to:
+    kind: exec
+    name: exec10

examples/file1.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   noop:
   - name: noop1
   file:
@@ -27,15 +27,15 @@ types:
 edges:
 - name: e1
   from:
-    type: file
+    kind: file
     name: file1
   to:
-    type: file
+    kind: file
     name: file2
 - name: e2
   from:
-    type: file
+    kind: file
     name: file2
   to:
-    type: file
+    kind: file
     name: file3

examples/graph0.yaml

@@ -1,7 +1,7 @@
 ---
 graph: mygraph
 comment: hello world example
-types:
+resources:
   noop:
   - name: noop1
   file:
@@ -13,8 +13,8 @@ types:
 edges:
 - name: e1
   from:
-    type: noop
+    kind: noop
     name: noop1
   to:
-    type: file
+    kind: file
     name: file1

examples/graph10.yaml

@@ -1,7 +1,7 @@
 ---
 graph: mygraph
 comment: simple exec fan in to fan out example to demonstrate optimization
-types:
+resources:
   exec:
   - name: exec1
     cmd: sleep 10s
@@ -86,43 +86,43 @@ types:
 edges:
 - name: e1
   from:
-    type: exec
+    kind: exec
     name: exec1
   to:
-    type: exec
+    kind: exec
     name: exec4
 - name: e2
   from:
-    type: exec
+    kind: exec
     name: exec2
   to:
-    type: exec
+    kind: exec
     name: exec4
 - name: e3
   from:
-    type: exec
+    kind: exec
     name: exec3
   to:
-    type: exec
+    kind: exec
     name: exec4
 - name: e4
   from:
-    type: exec
+    kind: exec
     name: exec4
   to:
-    type: exec
+    kind: exec
     name: exec5
 - name: e5
   from:
-    type: exec
+    kind: exec
     name: exec4
   to:
-    type: exec
+    kind: exec
     name: exec6
 - name: e6
   from:
-    type: exec
+    kind: exec
     name: exec4
   to:
-    type: exec
+    kind: exec
     name: exec7

examples/graph1a.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   file:
   - name: file1
     path: "/tmp/mgmt/f1"
@@ -15,8 +15,8 @@ types:
 edges:
 - name: e1
   from:
-    type: file
+    kind: file
     name: file1
   to:
-    type: file
+    kind: file
     name: file2

examples/graph1b.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   file:
   - name: file2
     path: "/tmp/mgmt/f2"
@@ -15,8 +15,8 @@ types:
 edges:
 - name: e2
   from:
-    type: file
+    kind: file
     name: file2
   to:
-    type: file
+    kind: file
     name: file3

examples/graph3a.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   file:
   - name: file1a
     path: "/tmp/mgmtA/f1a"
@@ -23,6 +23,6 @@ types:
       i am f4, exported from host A
     state: exists
 collect:
-- type: file
+- kind: file
   pattern: "/tmp/mgmtA/"
 edges: []

examples/graph3b.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   file:
   - name: file1b
     path: "/tmp/mgmtB/f1b"
@@ -23,6 +23,6 @@ types:
       i am f4, exported from host B
     state: exists
 collect:
-- type: file
+- kind: file
   pattern: "/tmp/mgmtB/"
 edges: []

examples/graph3c.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   file:
   - name: file1c
     path: "/tmp/mgmtC/f1c"
@@ -23,6 +23,6 @@ types:
       i am f4, exported from host C
     state: exists
 collect:
-- type: file
+- kind: file
   pattern: "/tmp/mgmtC/"
 edges: []

examples/graph4.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   file:
   - name: file1
     path: "/tmp/mgmt/f1"
@@ -13,6 +13,6 @@ types:
       i am f3, exported from host A
     state: exists
 collect:
-- type: file
+- kind: file
   pattern: ''
 edges:

examples/graph5.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   file:
   - name: file1
     path: "/tmp/mgmt/f1"
@@ -8,6 +8,6 @@ types:
       i am f1
     state: exists
 collect:
-- type: file
+- kind: file
   pattern: ''
 edges:

examples/graph6.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   noop:
   - name: noop1
 edges:

examples/graph7.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   noop:
   - name: noop1
   exec:

examples/graph9.yaml

@@ -1,7 +1,7 @@
 ---
 graph: mygraph
 comment: simple exec fan in example to demonstrate optimization
-types:
+resources:
   exec:
   - name: exec1
     cmd: sleep 10s
@@ -56,22 +56,22 @@ types:
 edges:
 - name: e1
   from:
-    type: exec
+    kind: exec
     name: exec1
   to:
-    type: exec
+    kind: exec
     name: exec5
 - name: e2
   from:
-    type: exec
+    kind: exec
     name: exec2
   to:
-    type: exec
+    kind: exec
     name: exec5
 - name: e3
   from:
-    type: exec
+    kind: exec
     name: exec3
   to:
-    type: exec
+    kind: exec
     name: exec5

examples/noop1.yaml

@@ -0,0 +1,24 @@
+---
+graph: mygraph
+comment: noop example
+resources:
+  noop:
+  - name: noop1
+    meta:
+      noop: true
+  file:
+  - name: file1
+    path: "/tmp/mgmt-hello-noop"
+    content: |
+      hello world from @purpleidea
+    state: exists
+    meta:
+      noop: true
+edges:
+- name: e1
+  from:
+    kind: noop
+    name: noop1
+  to:
+    kind: file
+    name: file1

examples/pkg1.yaml

@@ -0,0 +1,7 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: powertop
+    state: installed
+edges: []

examples/pkg2.yaml

@@ -0,0 +1,7 @@
+---
+graph: mygraph
+resources:
+  pkg:
+  - name: powertop
+    state: uninstalled
+edges: []

examples/remote1.yaml

@@ -0,0 +1,23 @@
+---
+graph: mygraph
+comment: remote noop example
+resources:
+  noop:
+  - name: noop1
+    meta:
+      noop: true
+  file:
+  - name: file1
+    path: "/tmp/mgmt-remote-hello"
+    content: |
+      hello world from @purpleidea
+    state: exists
+edges:
+- name: e1
+  from:
+    kind: noop
+    name: noop1
+  to:
+    kind: file
+    name: file1
+remote: "ssh://root:password@hostname:22"

examples/svc1.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   noop:
   - name: noop1
   file:
@@ -9,22 +9,22 @@ types:
     content: |
       i am f1
     state: exists
-  service:
+  svc:
   - name: purpleidea
     state: running
     startup: enabled
 edges:
 - name: e1
   from:
-    type: noop
+    kind: noop
     name: noop1
   to:
-    type: file
+    kind: file
     name: file1
 - name: e2
   from:
-    type: file
+    kind: file
     name: file1
   to:
-    type: service
+    kind: svc
     name: purpleidea

examples/timer1.yaml

@@ -0,0 +1,25 @@
+---
+graph: mygraph
+comment: timer example
+resources:
+  timer:
+  - name: timer1
+    interval: 30
+  exec:
+  - name: exec1
+    cmd: echo hello world
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: timer
+    name: timer1
+  to:
+    kind: exec
+    name: exec1

examples/timer2.yaml

@@ -0,0 +1,43 @@
+---
+graph: mygraph
+comment: example of multiple timers
+resources:
+  timer:
+  - name: timer1
+    interval: 30
+  - name: timer2
+    interval: 60
+  exec:
+  - name: exec1
+    cmd: echo hello world 30
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+  - name: exec2
+    cmd: echo hello world 60
+    timeout: 0
+    watchcmd: ''
+    watchshell: ''
+    ifcmd: ''
+    ifshell: ''
+    pollint: 0
+    state: present
+edges:
+- name: e1
+  from:
+    kind: timer
+    name: timer1
+  to:
+    kind: exec
+    name: exec1
+- name: e2
+  from:
+    kind: timer
+    name: timer2
+  to:
+    kind: exec
+    name: exec2

exec.go

@@ -20,13 +20,20 @@ package main
 import (
 	"bufio"
 	"bytes"
+	"encoding/gob"
+	"errors"
 	"log"
 	"os/exec"
 	"strings"
 )
 
-type ExecType struct {
+func init() {
-	BaseType   `yaml:",inline"`
+	gob.Register(&ExecRes{})
+}
+
+// ExecRes is an exec resource for running commands.
+type ExecRes struct {
+	BaseRes    `yaml:",inline"`
 	State      string `yaml:"state"`      // state: exists/present?, absent, (undefined?)
 	Cmd        string `yaml:"cmd"`        // the command to run
 	Shell      string `yaml:"shell"`      // the (optional) shell to use to run the cmd
@@ -38,13 +45,11 @@ type ExecType struct {
 	PollInt    int    `yaml:"pollint"`    // the poll interval for the ifcmd
 }
 
-func NewExecType(name, cmd, shell string, timeout int, watchcmd, watchshell, ifcmd, ifshell string, pollint int, state string) *ExecType {
+// NewExecRes is a constructor for this resource. It also calls Init() for you.
-	// FIXME if path = nil, path = name ...
+func NewExecRes(name, cmd, shell string, timeout int, watchcmd, watchshell, ifcmd, ifshell string, pollint int, state string) *ExecRes {
-	return &ExecType{
+	obj := &ExecRes{
-		BaseType: BaseType{
+		BaseRes: BaseRes{
-			Name:   name,
+			Name: name,
-			events: make(chan Event),
-			vertex: nil,
 		},
 		Cmd:        cmd,
 		Shell:      shell,
@@ -56,15 +61,19 @@ func NewExecType(name, cmd, shell string, timeout int, watchcmd, watchshell, ifc
 		PollInt:    pollint,
 		State:      state,
 	}
+	obj.Init()
+	return obj
 }
 
-func (obj *ExecType) GetType() string {
+// Init runs some startup code for this resource.
-	return "Exec"
+func (obj *ExecRes) Init() {
+	obj.BaseRes.kind = "Exec"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
 }
 
 // validate if the params passed in are valid data
 // FIXME: where should this get called ?
-func (obj *ExecType) Validate() bool {
+func (obj *ExecRes) Validate() bool {
 	if obj.Cmd == "" { // this is the only thing that is really required
 		return false
 	}
@@ -78,7 +87,7 @@ func (obj *ExecType) Validate() bool {
 }
 
 // wraps the scanner output in a channel
-func (obj *ExecType) BufioChanScanner(scanner *bufio.Scanner) (chan string, chan error) {
+func (obj *ExecRes) BufioChanScanner(scanner *bufio.Scanner) (chan string, chan error) {
 	ch, errch := make(chan string), make(chan error)
 	go func() {
 		for scanner.Scan() {
@@ -95,18 +104,19 @@ func (obj *ExecType) BufioChanScanner(scanner *bufio.Scanner) (chan string, chan
 	return ch, errch
 }
 
-// Exec watcher
+// Watch is the primary listener for this resource and it outputs events.
-func (obj *ExecType) Watch() {
+func (obj *ExecRes) Watch(processChan chan Event) {
 	if obj.IsWatching() {
 		return
 	}
 	obj.SetWatching(true)
 	defer obj.SetWatching(false)
+	cuuid := obj.converger.Register()
+	defer cuuid.Unregister()
 
 	var send = false // send event?
 	var exit = false
 	bufioch, errch := make(chan string), make(chan error)
-	//vertex := obj.GetVertex()         // stored with SetVertex
 
 	if obj.WatchCmd != "" {
 		var cmdName string
@@ -118,7 +128,7 @@ func (obj *ExecType) Watch() {
 			cmdName = split[0]
 			//d, _ := os.Getwd() // TODO: how does this ever error ?
 			//cmdName = path.Join(d, cmdName)
-			cmdArgs = split[1:len(split)]
+			cmdArgs = split[1:]
 		} else {
 			cmdName = obj.Shell // usually bash, or sh
 			cmdArgs = []string{"-c", obj.WatchCmd}
@@ -128,7 +138,7 @@ func (obj *ExecType) Watch() {
 
 		cmdReader, err := cmd.StdoutPipe()
 		if err != nil {
-			log.Printf("%v[%v]: Error creating StdoutPipe for Cmd: %v", obj.GetType(), obj.GetName(), err)
+			log.Printf("%v[%v]: Error creating StdoutPipe for Cmd: %v", obj.Kind(), obj.GetName(), err)
 			log.Fatal(err) // XXX: how should we handle errors?
 		}
 		scanner := bufio.NewScanner(cmdReader)
@@ -140,7 +150,7 @@ func (obj *ExecType) Watch() {
 			cmd.Process.Kill() // TODO: is this necessary?
 		}()
 		if err := cmd.Start(); err != nil {
-			log.Printf("%v[%v]: Error starting Cmd: %v", obj.GetType(), obj.GetName(), err)
+			log.Printf("%v[%v]: Error starting Cmd: %v", obj.Kind(), obj.GetName(), err)
 			log.Fatal(err) // XXX: how should we handle errors?
 		}
 
@@ -148,37 +158,36 @@ func (obj *ExecType) Watch() {
 	}
 
 	for {
-		obj.SetState(typeWatching) // reset
+		obj.SetState(resStateWatching) // reset
 		select {
 		case text := <-bufioch:
-			obj.SetConvergedState(typeConvergedNil)
+			cuuid.SetConverged(false)
 			// each time we get a line of output, we loop!
-			log.Printf("%v[%v]: Watch output: %s", obj.GetType(), obj.GetName(), text)
+			log.Printf("%v[%v]: Watch output: %s", obj.Kind(), obj.GetName(), text)
 			if text != "" {
 				send = true
 			}
 
 		case err := <-errch:
-			obj.SetConvergedState(typeConvergedNil) // XXX ?
+			cuuid.SetConverged(false) // XXX ?
-			if err == nil {                         // EOF
+			if err == nil {           // EOF
 				// FIXME: add an "if watch command ends/crashes"
 				// restart or generate error option
-				log.Printf("%v[%v]: Reached EOF", obj.GetType(), obj.GetName())
+				log.Printf("%v[%v]: Reached EOF", obj.Kind(), obj.GetName())
 				return
 			}
-			log.Printf("%v[%v]: Error reading input?: %v", obj.GetType(), obj.GetName(), err)
+			log.Printf("%v[%v]: Error reading input?: %v", obj.Kind(), obj.GetName(), err)
 			log.Fatal(err)
 			// XXX: how should we handle errors?
 
 		case event := <-obj.events:
-			obj.SetConvergedState(typeConvergedNil)
+			cuuid.SetConverged(false)
 			if exit, send = obj.ReadEvent(&event); exit {
 				return // exit
 			}
 
-		case _ = <-TimeAfterOrBlock(obj.ctimeout):
+		case <-cuuid.ConvergedTimer():
-			obj.SetConvergedState(typeConvergedTimeout)
+			cuuid.SetConverged(true) // converged!
-			obj.converged <- true
 			continue
 		}
 
@@ -187,19 +196,24 @@ func (obj *ExecType) Watch() {
 			send = false
 			// it is okay to invalidate the clean state on poke too
 			obj.isStateOK = false // something made state dirty
-			Process(obj)          // XXX: rename this function
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
 		}
 	}
 }
 
+// CheckApply checks the resource state and applies the resource if the bool
+// input is true. It returns error info and if the state check passed or not.
 // TODO: expand the IfCmd to be a list of commands
-func (obj *ExecType) StateOK() bool {
+func (obj *ExecRes) CheckApply(apply bool) (checkok bool, err error) {
+	log.Printf("%v[%v]: CheckApply(%t)", obj.Kind(), obj.GetName(), apply)
 
 	// if there is a watch command, but no if command, run based on state
-	if b := obj.isStateOK; obj.WatchCmd != "" && obj.IfCmd == "" {
+	if obj.WatchCmd != "" && obj.IfCmd == "" {
-		obj.isStateOK = true // reset
+		if obj.isStateOK {
-		//if !obj.isStateOK { obj.isStateOK = true; return false }
+			return true, nil
-		return b
+		}
 
 		// if there is no watcher, but there is an onlyif check, run it to see
 	} else if obj.IfCmd != "" { // && obj.WatchCmd == ""
@@ -221,28 +235,32 @@ func (obj *ExecType) StateOK() bool {
 			cmdName = split[0]
 			//d, _ := os.Getwd() // TODO: how does this ever error ?
 			//cmdName = path.Join(d, cmdName)
-			cmdArgs = split[1:len(split)]
+			cmdArgs = split[1:]
 		} else {
 			cmdName = obj.IfShell // usually bash, or sh
 			cmdArgs = []string{"-c", obj.IfCmd}
 		}
-		err := exec.Command(cmdName, cmdArgs...).Run()
+		err = exec.Command(cmdName, cmdArgs...).Run()
 		if err != nil {
 			// TODO: check exit value
-			return true // don't run
+			return true, nil // don't run
 		}
-		return false // just run
 
 		// if there is no watcher and no onlyif check, assume we should run
 	} else { // if obj.WatchCmd == "" && obj.IfCmd == "" {
-		b := obj.isStateOK
+		// just run if state is dirty
-		obj.isStateOK = true
+		if obj.isStateOK {
-		return b // just run if state is dirty
+			return true, nil
+		}
 	}
-}
 
-func (obj *ExecType) Apply() bool {
+	// state is not okay, no work done, exit, but without error
-	log.Printf("%v[%v]: Apply", obj.GetType(), obj.GetName())
+	if !apply {
+		return false, nil
+	}
+
+	// apply portion
+	log.Printf("%v[%v]: Apply", obj.Kind(), obj.GetName())
 	var cmdName string
 	var cmdArgs []string
 	if obj.Shell == "" {
@@ -253,7 +271,7 @@ func (obj *ExecType) Apply() bool {
 		cmdName = split[0]
 		//d, _ := os.Getwd() // TODO: how does this ever error ?
 		//cmdName = path.Join(d, cmdName)
-		cmdArgs = split[1:len(split)]
+		cmdArgs = split[1:]
 	} else {
 		cmdName = obj.Shell // usually bash, or sh
 		cmdArgs = []string{"-c", obj.Cmd}
@@ -263,9 +281,9 @@ func (obj *ExecType) Apply() bool {
 	var out bytes.Buffer
 	cmd.Stdout = &out
 
-	if err := cmd.Start(); err != nil {
+	if err = cmd.Start(); err != nil {
-		log.Printf("%v[%v]: Error starting Cmd: %v", obj.GetType(), obj.GetName(), err)
+		log.Printf("%v[%v]: Error starting Cmd: %v", obj.Kind(), obj.GetName(), err)
-		return false
+		return false, err
 	}
 
 	timeout := obj.Timeout
@@ -276,16 +294,16 @@ func (obj *ExecType) Apply() bool {
 	go func() { done <- cmd.Wait() }()
 
 	select {
-	case err := <-done:
+	case err = <-done:
 		if err != nil {
-			log.Printf("%v[%v]: Error waiting for Cmd: %v", obj.GetType(), obj.GetName(), err)
+			log.Printf("%v[%v]: Error waiting for Cmd: %v", obj.Kind(), obj.GetName(), err)
-			return false
+			return false, err
 		}
 
 	case <-TimeAfterOrBlock(timeout):
-		log.Printf("%v[%v]: Timeout waiting for Cmd", obj.GetType(), obj.GetName())
+		log.Printf("%v[%v]: Timeout waiting for Cmd", obj.Kind(), obj.GetName())
 		//cmd.Process.Kill() // TODO: is this necessary?
-		return false
+		return false, errors.New("Timeout waiting for Cmd!")
 	}
 
 	// TODO: if we printed the stdout while the command is running, this
@@ -298,38 +316,120 @@ func (obj *ExecType) Apply() bool {
 		log.Printf(out.String())
 	}
 	// XXX: return based on exit value!!
+
+	// the state tracking is for exec resources that can't "detect" their
+	// state, and assume it's invalid when the Watch() function triggers.
+	// if we apply state successfully, we should reset it here so that we
+	// know that we have applied since the state was set not ok by event!
+	obj.isStateOK = true // reset
+	return false, nil    // success
+}
+
+// ExecUUID is the UUID struct for ExecRes.
+type ExecUUID struct {
+	BaseUUID
+	Cmd   string
+	IfCmd string
+	// TODO: add more elements here
+}
+
+// if and only if they are equivalent, return true
+// if they are not equivalent, return false
+func (obj *ExecUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*ExecUUID)
+	if !ok {
+		return false
+	}
+	if obj.Cmd != res.Cmd {
+		return false
+	}
+	// TODO: add more checks here
+	//if obj.Shell != res.Shell {
+	//	return false
+	//}
+	//if obj.Timeout != res.Timeout {
+	//	return false
+	//}
+	//if obj.WatchCmd != res.WatchCmd {
+	//	return false
+	//}
+	//if obj.WatchShell != res.WatchShell {
+	//	return false
+	//}
+	if obj.IfCmd != res.IfCmd {
+		return false
+	}
+	//if obj.PollInt != res.PollInt {
+	//	return false
+	//}
+	//if obj.State != res.State {
+	//	return false
+	//}
 	return true
 }
 
-func (obj *ExecType) Compare(typ Type) bool {
+// The AutoEdges method returns the AutoEdges. In this case none are used.
-	switch typ.(type) {
+func (obj *ExecRes) AutoEdges() AutoEdge {
-	case *ExecType:
+	// TODO: parse as many exec params to look for auto edges, for example
-		typ := typ.(*ExecType)
+	// the path of the binary in the Cmd variable might be from in a pkg
-		if obj.Name != typ.Name {
+	return nil
+}
+
+// GetUUIDs includes all params to make a unique identification of this object.
+// Most resources only return one, although some resources can return multiple.
+func (obj *ExecRes) GetUUIDs() []ResUUID {
+	x := &ExecUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		Cmd:      obj.Cmd,
+		IfCmd:    obj.IfCmd,
+		// TODO: add more params here
+	}
+	return []ResUUID{x}
+}
+
+// GroupCmp returns whether two resources can be grouped together or not.
+func (obj *ExecRes) GroupCmp(r Res) bool {
+	_, ok := r.(*ExecRes)
+	if !ok {
+		return false
+	}
+	return false // not possible atm
+}
+
+// Compare two resources and return if they are equivalent.
+func (obj *ExecRes) Compare(res Res) bool {
+	switch res.(type) {
+	case *ExecRes:
+		res := res.(*ExecRes)
+		if !obj.BaseRes.Compare(res) { // call base Compare
 			return false
 		}
-		if obj.Cmd != typ.Cmd {
+
+		if obj.Name != res.Name {
 			return false
 		}
-		if obj.Shell != typ.Shell {
+		if obj.Cmd != res.Cmd {
 			return false
 		}
-		if obj.Timeout != typ.Timeout {
+		if obj.Shell != res.Shell {
 			return false
 		}
-		if obj.WatchCmd != typ.WatchCmd {
+		if obj.Timeout != res.Timeout {
 			return false
 		}
-		if obj.WatchShell != typ.WatchShell {
+		if obj.WatchCmd != res.WatchCmd {
 			return false
 		}
-		if obj.IfCmd != typ.IfCmd {
+		if obj.WatchShell != res.WatchShell {
 			return false
 		}
-		if obj.PollInt != typ.PollInt {
+		if obj.IfCmd != res.IfCmd {
 			return false
 		}
-		if obj.State != typ.State {
+		if obj.PollInt != res.PollInt {
+			return false
+		}
+		if obj.State != res.State {
 			return false
 		}
 	default:

file.go

@@ -22,6 +22,7 @@ import (
 	"encoding/hex"
 	"gopkg.in/fsnotify.v1"
 	//"github.com/go-fsnotify/fsnotify" // git master of "gopkg.in/fsnotify.v1"
+	"encoding/gob"
 	"io"
 	"log"
 	"math"
@@ -31,8 +32,13 @@ import (
 	"syscall"
 )
 
-type FileType struct {
+func init() {
-	BaseType  `yaml:",inline"`
+	gob.Register(&FileRes{})
+}
+
+// FileRes is a file and directory resource.
+type FileRes struct {
+	BaseRes   `yaml:",inline"`
 	Path      string `yaml:"path"` // path variable (should default to name)
 	Dirname   string `yaml:"dirname"`
 	Basename  string `yaml:"basename"`
@@ -41,13 +47,12 @@ type FileType struct {
 	sha256sum string
 }
 
-func NewFileType(name, path, dirname, basename, content, state string) *FileType {
+// NewFileRes is a constructor for this resource. It also calls Init() for you.
+func NewFileRes(name, path, dirname, basename, content, state string) *FileRes {
 	// FIXME if path = nil, path = name ...
-	return &FileType{
+	obj := &FileRes{
-		BaseType: BaseType{
+		BaseRes: BaseRes{
-			Name:   name,
+			Name: name,
-			events: make(chan Event),
-			vertex: nil,
 		},
 		Path:      path,
 		Dirname:   dirname,
@@ -56,14 +61,34 @@ func NewFileType(name, path, dirname, basename, content, state string) *FileType
 		State:     state,
 		sha256sum: "",
 	}
+	obj.Init()
+	return obj
 }
 
-func (obj *FileType) GetType() string {
+// Init runs some startup code for this resource.
-	return "File"
+func (obj *FileRes) Init() {
+	obj.BaseRes.kind = "File"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+}
+
+// GetPath returns the actual path to use for this resource. It computes this
+// after analysis of the path, dirname and basename values.
+func (obj *FileRes) GetPath() string {
+	d := Dirname(obj.Path)
+	b := Basename(obj.Path)
+	if !obj.Validate() || (obj.Dirname == "" && obj.Basename == "") {
+		return obj.Path
+	} else if obj.Dirname == "" {
+		return d + obj.Basename
+	} else if obj.Basename == "" {
+		return obj.Dirname + b
+	} else { // if obj.dirname != "" && obj.basename != "" {
+		return obj.Dirname + obj.Basename
+	}
 }
 
 // validate if the params passed in are valid data
-func (obj *FileType) Validate() bool {
+func (obj *FileRes) Validate() bool {
 	if obj.Dirname != "" {
 		// must end with /
 		if obj.Dirname[len(obj.Dirname)-1:] != "/" {
@@ -79,34 +104,22 @@ func (obj *FileType) Validate() bool {
 	return true
 }
 
-func (obj *FileType) GetPath() string {
+// Watch is the primary listener for this resource and it outputs events.
-	d := Dirname(obj.Path)
+// This one is a file watcher for files and directories.
-	b := Basename(obj.Path)
+// Modify with caution, it is probably important to write some test cases first!
-	if !obj.Validate() || (obj.Dirname == "" && obj.Basename == "") {
-		return obj.Path
-	} else if obj.Dirname == "" {
-		return d + obj.Basename
-	} else if obj.Basename == "" {
-		return obj.Dirname + b
-	} else { // if obj.dirname != "" && obj.basename != "" {
-		return obj.Dirname + obj.Basename
-	}
-}
-
-// File watcher for files and directories
-// Modify with caution, probably important to write some test cases first!
 // obj.GetPath(): file or directory
-func (obj *FileType) Watch() {
+func (obj *FileRes) Watch(processChan chan Event) {
 	if obj.IsWatching() {
 		return
 	}
 	obj.SetWatching(true)
 	defer obj.SetWatching(false)
+	cuuid := obj.converger.Register()
+	defer cuuid.Unregister()
 
 	//var recursive bool = false
 	//var isdir = (obj.GetPath()[len(obj.GetPath())-1:] == "/") // dirs have trailing slashes
 	//log.Printf("IsDirectory: %v", isdir)
-	//vertex := obj.GetVertex()         // stored with SetVertex
 	var safename = path.Clean(obj.GetPath()) // no trailing slash
 
 	watcher, err := fsnotify.NewWatcher()
@@ -142,7 +155,7 @@ func (obj *FileType) Watch() {
 			} else if err == syscall.ENOSPC {
 				// XXX: occasionally: no space left on device,
 				// XXX: probably due to lack of inotify watches
-				log.Printf("Lack of watches for file[%v] error: %+v", obj.Name, err.Error) // 0x408da0
+				log.Printf("%v[%v]: Out of inotify watches!", obj.Kind(), obj.GetName())
 				log.Fatal(err)
 			} else {
 				log.Printf("Unknown file[%v] error:", obj.Name)
@@ -152,13 +165,13 @@ func (obj *FileType) Watch() {
 			continue
 		}
 
-		obj.SetState(typeWatching) // reset
+		obj.SetState(resStateWatching) // reset
 		select {
 		case event := <-watcher.Events:
 			if DEBUG {
 				log.Printf("File[%v]: Watch(%v), Event(%v): %v", obj.GetName(), current, event.Name, event.Op)
 			}
-			obj.SetConvergedState(typeConvergedNil) // XXX: technically i can detect if the event is erroneous or not first
+			cuuid.SetConverged(false) // XXX: technically i can detect if the event is erroneous or not first
 			// the deeper you go, the bigger the deltaDepth is...
 			// this is the difference between what we're watching,
 			// and the event... doesn't mean we can't watch deeper
@@ -228,21 +241,20 @@ func (obj *FileType) Watch() {
 			}
 
 		case err := <-watcher.Errors:
-			obj.SetConvergedState(typeConvergedNil) // XXX ?
+			cuuid.SetConverged(false) // XXX ?
-			log.Println("error:", err)
+			log.Printf("error: %v", err)
 			log.Fatal(err)
 			//obj.events <- fmt.Sprintf("file: %v", "error") // XXX: how should we handle errors?
 
 		case event := <-obj.events:
-			obj.SetConvergedState(typeConvergedNil)
+			cuuid.SetConverged(false)
 			if exit, send = obj.ReadEvent(&event); exit {
 				return // exit
 			}
 			//dirty = false // these events don't invalidate state
 
-		case _ = <-TimeAfterOrBlock(obj.ctimeout):
+		case <-cuuid.ConvergedTimer():
-			obj.SetConvergedState(typeConvergedTimeout)
+			cuuid.SetConverged(true) // converged!
-			obj.converged <- true
 			continue
 		}
 
@@ -254,12 +266,16 @@ func (obj *FileType) Watch() {
 				dirty = false
 				obj.isStateOK = false // something made state dirty
 			}
-			Process(obj) // XXX: rename this function
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
 		}
 	}
 }
 
-func (obj *FileType) HashSHA256fromContent() string {
+// HashSHA256fromContent computes the hash of the file contents and returns it.
+// It also caches the value if it can.
+func (obj *FileRes) HashSHA256fromContent() string {
 	if obj.sha256sum != "" { // return if already computed
 		return obj.sha256sum
 	}
@@ -270,138 +286,234 @@ func (obj *FileType) HashSHA256fromContent() string {
 	return obj.sha256sum
 }
 
-// FIXME: add the obj.CleanState() calls all over the true returns!
+// FileHashSHA256Check computes the hash of the actual file and compares it to
-func (obj *FileType) StateOK() bool {
+// the computed hash of the resources file contents.
-	if obj.isStateOK { // cache the state
+func (obj *FileRes) FileHashSHA256Check() (bool, error) {
-		return true
+	if PathIsDir(obj.GetPath()) { // assert
+		log.Fatal("This should only be called on a File resource.")
 	}
-
+	// run a diff, and return true if it needs changing
-	if _, err := os.Stat(obj.GetPath()); os.IsNotExist(err) {
-		// no such file or directory
-		if obj.State == "absent" {
-			return obj.CleanState() // missing file should be missing, phew :)
-		} else {
-			// state invalid, skip expensive checksums
-			return false
-		}
-	}
-
-	// TODO: add file mode check here...
-
-	if PathIsDir(obj.GetPath()) {
-		return obj.StateOKDir()
-	} else {
-		return obj.StateOKFile()
-	}
-}
-
-func (obj *FileType) StateOKFile() bool {
-	if PathIsDir(obj.GetPath()) {
-		log.Fatal("This should only be called on a File type.")
-	}
-
-	// run a diff, and return true if needs changing
-
 	hash := sha256.New()
-
 	f, err := os.Open(obj.GetPath())
 	if err != nil {
-		//log.Fatal(err)
+		if e, ok := err.(*os.PathError); ok && (e.Err.(syscall.Errno) == syscall.ENOENT) {
-		return false
+			return false, nil // no "error", file is just absent
+		}
+		return false, err
 	}
 	defer f.Close()
-
 	if _, err := io.Copy(hash, f); err != nil {
-		//log.Fatal(err)
+		return false, err
-		return false
 	}
-
 	sha256sum := hex.EncodeToString(hash.Sum(nil))
 	//log.Printf("sha256sum: %v", sha256sum)
-
 	if obj.HashSHA256fromContent() == sha256sum {
-		return true
+		return true, nil
 	}
-
+	return false, nil
-	return false
 }
 
-func (obj *FileType) StateOKDir() bool {
+// FileApply writes the resource file contents out to the correct path. This
-	if !PathIsDir(obj.GetPath()) {
+// implementation doesn't try to be particularly clever in any way.
-		log.Fatal("This should only be called on a Dir type.")
+func (obj *FileRes) FileApply() error {
-	}
-
-	// XXX: not implemented
-	log.Fatal("Not implemented!")
-	return false
-}
-
-func (obj *FileType) Apply() bool {
-	log.Printf("%v[%v]: Apply", obj.GetType(), obj.GetName())
-
 	if PathIsDir(obj.GetPath()) {
-		return obj.ApplyDir()
+		log.Fatal("This should only be called on a File resource.")
-	} else {
-		return obj.ApplyFile()
-	}
-}
-
-func (obj *FileType) ApplyFile() bool {
-
-	if PathIsDir(obj.GetPath()) {
-		log.Fatal("This should only be called on a File type.")
 	}
 
 	if obj.State == "absent" {
 		log.Printf("About to remove: %v", obj.GetPath())
 		err := os.Remove(obj.GetPath())
-		if err != nil {
+		return err // either nil or not, for success or failure
-			return false
-		}
-		return true
 	}
 
-	//log.Println("writing: " + filename)
 	f, err := os.Create(obj.GetPath())
 	if err != nil {
-		log.Println("error:", err)
+		return nil
-		return false
 	}
 	defer f.Close()
 
 	_, err = io.WriteString(f, obj.Content)
 	if err != nil {
-		log.Println("error:", err)
+		return err
+	}
+
+	return nil // success
+}
+
+// CheckApply checks the resource state and applies the resource if the bool
+// input is true. It returns error info and if the state check passed or not.
+func (obj *FileRes) CheckApply(apply bool) (checkok bool, err error) {
+	log.Printf("%v[%v]: CheckApply(%t)", obj.Kind(), obj.GetName(), apply)
+
+	if obj.isStateOK { // cache the state
+		return true, nil
+	}
+
+	if _, err = os.Stat(obj.GetPath()); os.IsNotExist(err) {
+		// no such file or directory
+		if obj.State == "absent" {
+			// missing file should be missing, phew :)
+			obj.isStateOK = true
+			return true, nil
+		}
+	}
+	err = nil // reset
+
+	// FIXME: add file mode check here...
+
+	if PathIsDir(obj.GetPath()) {
+		log.Fatal("Not implemented!") // XXX
+	} else {
+		ok, err := obj.FileHashSHA256Check()
+		if err != nil {
+			return false, err
+		}
+		if ok {
+			obj.isStateOK = true
+			return true, nil
+		}
+		// if no err, but !ok, then we continue on...
+	}
+
+	// state is not okay, no work done, exit, but without error
+	if !apply {
+		return false, nil
+	}
+
+	// apply portion
+	log.Printf("%v[%v]: Apply", obj.Kind(), obj.GetName())
+	if PathIsDir(obj.GetPath()) {
+		log.Fatal("Not implemented!") // XXX
+	} else {
+		err = obj.FileApply()
+		if err != nil {
+			return false, err
+		}
+	}
+
+	obj.isStateOK = true
+	return false, nil // success
+}
+
+// FileUUID is the UUID struct for FileRes.
+type FileUUID struct {
+	BaseUUID
+	path string
+}
+
+// if and only if they are equivalent, return true
+// if they are not equivalent, return false
+func (obj *FileUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*FileUUID)
+	if !ok {
 		return false
 	}
-
+	return obj.path == res.path
-	return true
 }
 
-func (obj *FileType) ApplyDir() bool {
+// FileResAutoEdges holds the state of the auto edge generator.
-	if !PathIsDir(obj.GetPath()) {
+type FileResAutoEdges struct {
-		log.Fatal("This should only be called on a Dir type.")
+	data    []ResUUID
+	pointer int
+	found   bool
+}
+
+// Next returns the next automatic edge.
+func (obj *FileResAutoEdges) Next() []ResUUID {
+	if obj.found {
+		log.Fatal("Shouldn't be called anymore!")
 	}
-
+	if len(obj.data) == 0 { // check length for rare scenarios
-	// XXX: not implemented
+		return nil
-	log.Fatal("Not implemented!")
+	}
-	return true
+	value := obj.data[obj.pointer]
+	obj.pointer++
+	return []ResUUID{value} // we return one, even though api supports N
 }
 
-func (obj *FileType) Compare(typ Type) bool {
+// Test gets results of the earlier Next() call, & returns if we should continue!
-	switch typ.(type) {
+func (obj *FileResAutoEdges) Test(input []bool) bool {
-	case *FileType:
+	// if there aren't any more remaining
-		typ := typ.(*FileType)
+	if len(obj.data) <= obj.pointer {
-		if obj.Name != typ.Name {
+		return false
+	}
+	if obj.found { // already found, done!
+		return false
+	}
+	if len(input) != 1 { // in case we get given bad data
+		log.Fatal("Expecting a single value!")
+	}
+	if input[0] { // if a match is found, we're done!
+		obj.found = true // no more to find!
+		return false
+	}
+	return true // keep going
+}
+
+// AutoEdges generates a simple linear sequence of each parent directory from
+// the bottom up!
+func (obj *FileRes) AutoEdges() AutoEdge {
+	var data []ResUUID                             // store linear result chain here...
+	values := PathSplitFullReversed(obj.GetPath()) // build it
+	_, values = values[0], values[1:]              // get rid of first value which is me!
+	for _, x := range values {
+		var reversed = true // cheat by passing a pointer
+		data = append(data, &FileUUID{
+			BaseUUID: BaseUUID{
+				name:     obj.GetName(),
+				kind:     obj.Kind(),
+				reversed: &reversed,
+			},
+			path: x, // what matters
+		}) // build list
+	}
+	return &FileResAutoEdges{
+		data:    data,
+		pointer: 0,
+		found:   false,
+	}
+}
+
+// GetUUIDs includes all params to make a unique identification of this object.
+// Most resources only return one, although some resources can return multiple.
+func (obj *FileRes) GetUUIDs() []ResUUID {
+	x := &FileUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		path:     obj.GetPath(),
+	}
+	return []ResUUID{x}
+}
+
+// GroupCmp returns whether two resources can be grouped together or not.
+func (obj *FileRes) GroupCmp(r Res) bool {
+	_, ok := r.(*FileRes)
+	if !ok {
+		return false
+	}
+	// TODO: we might be able to group directory children into a single
+	// recursive watcher in the future, thus saving fanotify watches
+	return false // not possible atm
+}
+
+// Compare two resources and return if they are equivalent.
+func (obj *FileRes) Compare(res Res) bool {
+	switch res.(type) {
+	case *FileRes:
+		res := res.(*FileRes)
+		if !obj.BaseRes.Compare(res) { // call base Compare
 			return false
 		}
-		if obj.GetPath() != typ.Path {
+
+		if obj.Name != res.Name {
 			return false
 		}
-		if obj.Content != typ.Content {
+		if obj.GetPath() != res.Path {
 			return false
 		}
-		if obj.State != typ.State {
+		if obj.Content != res.Content {
+			return false
+		}
+		if obj.State != res.State {
 			return false
 		}
 	default:
@@ -409,3 +521,9 @@ func (obj *FileType) Compare(typ Type) bool {
 	}
 	return true
 }
+
+// CollectPattern applies the pattern for collection resources.
+func (obj *FileRes) CollectPattern(pattern string) {
+	// XXX: currently the pattern for files can only override the Dirname variable :P
+	obj.Dirname = pattern // XXX: simplistic for now
+}

gopath/.gitignore

@@ -0,0 +1,2 @@
+bin/
+pkg/

gopath/src

@@ -0,0 +1 @@
+../vendor

main.go

@@ -18,24 +18,31 @@
 package main
 
 import (
-	"github.com/codegangsta/cli"
+	"fmt"
+	etcdtypes "github.com/coreos/etcd/pkg/types"
+	"github.com/coreos/pkg/capnslog"
+	"github.com/urfave/cli"
+	"io/ioutil"
 	"log"
 	"os"
 	"os/signal"
 	"sync"
 	"syscall"
 	"time"
-	//etcd_context "github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 )
 
 // set at compile time
 var (
 	program string
 	version string
+	prefix  = fmt.Sprintf("/var/lib/%s/", program)
 )
 
+// variables controlling verbosity
 const (
-	DEBUG = false
+	DEBUG   = false // add additional log messages
+	TRACE   = false // add execution flow log messages
+	VERBOSE = false // add extra log message output
 )
 
 // signal handler
@@ -57,14 +64,114 @@ func waitForSignal(exit chan bool) {
 	}
 }
 
-func run(c *cli.Context) {
+// run is the main run target.
+func run(c *cli.Context) error {
 	var start = time.Now().UnixNano()
-	var wg sync.WaitGroup
-	exit := make(chan bool)      // exit signal
-	converged := make(chan bool) // converged signal
 	log.Printf("This is: %v, version: %v", program, version)
 	log.Printf("Main: Start: %v", start)
-	G := NewGraph("Graph") // give graph a default name
+
+	hostname, _ := os.Hostname()
+	// allow passing in the hostname, instead of using --hostname
+	if c.IsSet("file") {
+		if config := ParseConfigFromFile(c.String("file")); config != nil {
+			if h := config.Hostname; h != "" {
+				hostname = h
+			}
+		}
+	}
+	if c.IsSet("hostname") { // override by cli
+		if h := c.String("hostname"); h != "" {
+			hostname = h
+		}
+	}
+	noop := c.Bool("noop")
+
+	seeds, err := etcdtypes.NewURLs(
+		FlattenListWithSplit(c.StringSlice("seeds"), []string{",", ";", " "}),
+	)
+	if err != nil && len(c.StringSlice("seeds")) > 0 {
+		log.Printf("Main: Error: seeds didn't parse correctly!")
+		return cli.NewExitError("", 1)
+	}
+	clientURLs, err := etcdtypes.NewURLs(
+		FlattenListWithSplit(c.StringSlice("client-urls"), []string{",", ";", " "}),
+	)
+	if err != nil && len(c.StringSlice("client-urls")) > 0 {
+		log.Printf("Main: Error: clientURLs didn't parse correctly!")
+		return cli.NewExitError("", 1)
+	}
+	serverURLs, err := etcdtypes.NewURLs(
+		FlattenListWithSplit(c.StringSlice("server-urls"), []string{",", ";", " "}),
+	)
+	if err != nil && len(c.StringSlice("server-urls")) > 0 {
+		log.Printf("Main: Error: serverURLs didn't parse correctly!")
+		return cli.NewExitError("", 1)
+	}
+
+	idealClusterSize := uint16(c.Int("ideal-cluster-size"))
+	if idealClusterSize < 1 {
+		log.Printf("Main: Error: idealClusterSize should be at least one!")
+		return cli.NewExitError("", 1)
+	}
+
+	if c.IsSet("file") && c.IsSet("puppet") {
+		log.Println("Main: Error: the --file and --puppet parameters cannot be used together!")
+		return cli.NewExitError("", 1)
+	}
+
+	if c.Bool("no-server") && len(c.StringSlice("remote")) > 0 {
+		// TODO: in this case, we won't be able to tunnel stuff back to
+		// here, so if we're okay with every remote graph running in an
+		// isolated mode, then this is okay. Improve on this if there's
+		// someone who really wants to be able to do this.
+		log.Println("Main: Error: the --no-server and --remote parameters cannot be used together!")
+		return cli.NewExitError("", 1)
+	}
+
+	cConns := uint16(c.Int("cconns"))
+	if cConns < 0 {
+		log.Printf("Main: Error: --cconns should be at least zero!")
+		return cli.NewExitError("", 1)
+	}
+
+	if c.IsSet("converged-timeout") && cConns > 0 && len(c.StringSlice("remote")) > c.Int("cconns") {
+		log.Printf("Main: Error: combining --converged-timeout with more remotes than available connections will never converge!")
+		return cli.NewExitError("", 1)
+	}
+
+	depth := uint16(c.Int("depth"))
+	if depth < 0 { // user should not be using this argument manually
+		log.Printf("Main: Error: negative values for --depth are not permitted!")
+		return cli.NewExitError("", 1)
+	}
+
+	if c.IsSet("prefix") && c.Bool("tmp-prefix") {
+		log.Println("Main: Error: combining --prefix and the request for a tmp prefix is illogical!")
+		return cli.NewExitError("", 1)
+	}
+	if s := c.String("prefix"); c.IsSet("prefix") && s != "" {
+		prefix = s
+	}
+
+	// make sure the working directory prefix exists
+	if c.Bool("tmp-prefix") || os.MkdirAll(prefix, 0770) != nil {
+		if c.Bool("tmp-prefix") || c.Bool("allow-tmp-prefix") {
+			if prefix, err = ioutil.TempDir("", program+"-"); err != nil {
+				log.Printf("Main: Error: Can't create temporary prefix!")
+				return cli.NewExitError("", 1)
+			}
+			log.Println("Main: Warning: Working prefix directory is temporary!")
+
+		} else {
+			log.Printf("Main: Error: Can't create prefix!")
+			return cli.NewExitError("", 1)
+		}
+	}
+	log.Printf("Main: Working prefix is: %s", prefix)
+
+	var wg sync.WaitGroup
+	exit := make(chan bool) // exit signal
+	var G, fullGraph *Graph
 
 	// exit after `max-runtime` seconds for no reason at all...
 	if i := c.Int("max-runtime"); i > 0 {
@@ -74,78 +181,141 @@ func run(c *cli.Context) {
 		}()
 	}
 
-	// initial etcd peer endpoint
+	// setup converger
-	seed := c.String("seed")
+	converger := NewConverger(
-	if seed == "" {
+		c.Int("converged-timeout"),
-		// XXX: start up etcd server, others will join me!
+		nil, // stateFn gets added in by EmbdEtcd
-		seed = "http://127.0.0.1:2379" // thus we use the local server!
+	)
+	go converger.Loop(true) // main loop for converger, true to start paused
+
+	// embedded etcd
+	if len(seeds) == 0 {
+		log.Printf("Main: Seeds: No seeds specified!")
+	} else {
+		log.Printf("Main: Seeds(%v): %v", len(seeds), seeds)
 	}
-	// then, connect to `seed` as a client
+	EmbdEtcd := NewEmbdEtcd(
-
+		hostname,
-	// FIXME: validate seed, or wait for it to fail in etcd init?
+		seeds,
-
+		clientURLs,
-	// etcd
+		serverURLs,
-	etcdO := &EtcdWObject{
+		c.Bool("no-server"),
-		seed:      seed,
+		idealClusterSize,
-		ctimeout:  c.Int("converged-timeout"),
+		prefix,
-		converged: converged,
+		converger,
+	)
+	if EmbdEtcd == nil {
+		// TODO: verify EmbdEtcd is not nil below...
+		log.Printf("Main: Etcd: Creation failed!")
+		exit <- true
+	} else if err := EmbdEtcd.Startup(); err != nil { // startup (returns when etcd main loop is running)
+		log.Printf("Main: Etcd: Startup failed: %v", err)
+		exit <- true
+	}
+	convergerStateFn := func(b bool) error {
+		// exit if we are using the converged-timeout and we are the
+		// root node. otherwise, if we are a child node in a remote
+		// execution hierarchy, we should only notify our converged
+		// state and wait for the parent to trigger the exit.
+		if depth == 0 && c.Int("converged-timeout") >= 0 {
+			if b {
+				log.Printf("Converged for %d seconds, exiting!", c.Int("converged-timeout"))
+				exit <- true // trigger an exit!
+			}
+			return nil
+		}
+		// send our individual state into etcd for others to see
+		return EtcdSetHostnameConverged(EmbdEtcd, hostname, b) // TODO: what should happen on error?
+	}
+	if EmbdEtcd != nil {
+		converger.SetStateFn(convergerStateFn)
 	}
 
-	hostname := c.String("hostname")
+	exitchan := make(chan Event) // exit event
-	if hostname == "" {
-		hostname, _ = os.Hostname() // etcd watch key // XXX: this is not the correct key name this is the set key name... WOOPS
-	}
 	go func() {
 		startchan := make(chan struct{}) // start signal
 		go func() { startchan <- struct{}{} }()
 		file := c.String("file")
-		configchan := make(chan bool)
+		var configchan chan bool
-		if !c.Bool("no-watch") {
+		var puppetchan <-chan time.Time
+		if !c.Bool("no-watch") && c.IsSet("file") {
 			configchan = ConfigWatch(file)
+		} else if c.IsSet("puppet") {
+			interval := PuppetInterval(c.String("puppet-conf"))
+			puppetchan = time.Tick(time.Duration(interval) * time.Second)
 		}
-		log.Printf("Etcd: Starting...")
+		log.Println("Etcd: Starting...")
-		etcdchan := etcdO.EtcdWatch()
+		etcdchan := EtcdWatch(EmbdEtcd)
 		first := true // first loop or not
 		for {
+			log.Println("Main: Waiting...")
 			select {
-			case _ = <-startchan: // kick the loop once at start
+			case <-startchan: // kick the loop once at start
 				// pass
-			case msg := <-etcdchan:
+
-				switch msg {
+			case b := <-etcdchan:
-				// some types of messages we ignore...
+				if !b { // ignore the message
-				case etcdFoo, etcdBar:
 					continue
-				// while others passthrough and cause a compile!
-				case etcdStart, etcdEvent:
-					// pass
-				default:
-					log.Fatal("Etcd: Unhandled message: ", msg)
 				}
+				// everything else passes through to cause a compile!
+
+			case <-puppetchan:
+				// nothing, just go on
+
 			case msg := <-configchan:
 				if c.Bool("no-watch") || !msg {
 					continue // not ready to read config
 				}
-				//case compile_event: XXX
+			// XXX: case compile_event: ...
+			// ...
+			case msg := <-exitchan:
+				msg.ACK()
+				return
 			}
 
-			config := ParseConfigFromFile(file)
+			var config *GraphConfig
+			if c.IsSet("file") {
+				config = ParseConfigFromFile(file)
+			} else if c.IsSet("puppet") {
+				config = ParseConfigFromPuppet(c.String("puppet"), c.String("puppet-conf"))
+			}
 			if config == nil {
-				log.Printf("Config parse failure")
+				log.Printf("Config: Parse failure")
 				continue
 			}
 
+			if config.Hostname != "" && config.Hostname != hostname {
+				log.Printf("Config: Hostname changed, ignoring config!")
+				continue
+			}
+			config.Hostname = hostname // set it in case it was ""
+
 			// run graph vertex LOCK...
-			if !first { // XXX: we can flatten this check out I think
+			if !first { // TODO: we can flatten this check out I think
-				log.Printf("State: %v -> %v", G.SetState(graphPausing), G.GetState())
+				converger.Pause() // FIXME: add sync wait?
-				G.Pause() // sync
+				G.Pause()         // sync
-				log.Printf("State: %v -> %v", G.SetState(graphPaused), G.GetState())
 			}
 
-			// build the graph from a config file
+			// build graph from yaml file on events (eg: from etcd)
-			// build the graph on events (eg: from etcd)
+			// we need the vertices to be paused to work on them
-			if !UpdateGraphFromConfig(config, hostname, G, etcdO) {
+			if newFullgraph, err := fullGraph.NewGraphFromConfig(config, EmbdEtcd, noop); err == nil { // keep references to all original elements
-				log.Fatal("Config: We borked the graph.") // XXX
+				fullGraph = newFullgraph
+			} else {
+				log.Printf("Config: Error making new graph from config: %v", err)
+				// unpause!
+				if !first {
+					G.Start(&wg, first) // sync
+					converger.Start()   // after G.Start()
+				}
+				continue
 			}
+
+			G = fullGraph.Copy() // copy to active graph
+			// XXX: do etcd transaction out here...
+			G.AutoEdges() // add autoedges; modifies the graph
+			G.AutoGroup() // run autogroup; modifies the graph
+			// TODO: do we want to do a transitive reduction?
+
 			log.Printf("Graph: %v", G) // show graph
 			err := G.ExecGraphviz(c.String("graphviz-filter"), c.String("graphviz"))
 			if err != nil {
@@ -153,52 +323,77 @@ func run(c *cli.Context) {
 			} else {
 				log.Printf("Graphviz: Successfully generated graph!")
 			}
-			G.SetVertex()
+			G.AssociateData(converger)
-			G.SetConvergedCallback(c.Int("converged-timeout"), converged)
 			// G.Start(...) needs to be synchronous or wait,
 			// because if half of the nodes are started and
 			// some are not ready yet and the EtcdWatch
 			// loops, we'll cause G.Pause(...) before we
 			// even got going, thus causing nil pointer errors
-			log.Printf("State: %v -> %v", G.SetState(graphStarting), G.GetState())
 			G.Start(&wg, first) // sync
-			log.Printf("State: %v -> %v", G.SetState(graphStarted), G.GetState())
+			converger.Start()   // after G.Start()
 			first = false
 		}
 	}()
 
-	if i := c.Int("converged-timeout"); i >= 0 {
+	configWatcher := NewConfigWatcher()
-		go func() {
+	events := configWatcher.Events()
-		ConvergedLoop:
+	if !c.Bool("no-watch") {
-			for {
+		configWatcher.Add(c.StringSlice("remote")...) // add all the files...
-				<-converged // when anyone says they have converged
+	} else {
-
+		events = nil // signal that no-watch is true
-				if etcdO.GetConvergedState() != etcdConvergedTimeout {
-					continue
-				}
-				for v := range G.GetVerticesChan() {
-					if v.Type.GetConvergedState() != typeConvergedTimeout {
-						continue ConvergedLoop
-					}
-				}
-
-				// if all have converged, exit
-				log.Printf("Converged for %d seconds, exiting!", i)
-				exit <- true
-				for {
-					<-converged
-				} // unblock/drain
-				//return
-			}
-		}()
 	}
 
+	// initialize the add watcher, which calls the f callback on map changes
+	convergerCb := func(f func(map[string]bool) error) (func(), error) {
+		return EtcdAddHostnameConvergedWatcher(EmbdEtcd, f)
+	}
+
+	// build remotes struct for remote ssh
+	remotes := NewRemotes(
+		EmbdEtcd.LocalhostClientURLs().StringSlice(),
+		[]string{DefaultClientURL},
+		noop,
+		c.StringSlice("remote"), // list of files
+		events,                  // watch for file changes
+		cConns,
+		c.Bool("allow-interactive"),
+		c.String("ssh-priv-id-rsa"),
+		!c.Bool("no-caching"),
+		depth,
+		prefix,
+		converger,
+		convergerCb,
+	)
+
+	// TODO: is there any benefit to running the remotes above in the loop?
+	// wait for etcd to be running before we remote in, which we do above!
+	go remotes.Run()
+
+	if !c.IsSet("file") && !c.IsSet("puppet") {
+		converger.Start() // better start this for empty graphs
+	}
 	log.Println("Main: Running...")
 
 	waitForSignal(exit) // pass in exit channel to watch
 
+	log.Println("Destroy...")
+
+	configWatcher.Close() // stop sending file changes to remotes
+	remotes.Exit()        // tell all the remote connections to shutdown; waits!
+
 	G.Exit() // tell all the children to exit
 
+	// tell inner main loop to exit
+	resp := NewResp()
+	go func() { exitchan <- Event{eventExit, resp, "", false} }()
+
+	// cleanup etcd main loop last so it can process everything first
+	if err := EmbdEtcd.Destroy(); err != nil { // shutdown and cleanup etcd
+		log.Printf("Etcd exited poorly with: %v", err)
+	}
+
+	resp.ACKWait() // let inner main loop finish cleanly just in case
+
 	if DEBUG {
 		log.Printf("Graph: %v", G)
 	}
@@ -207,13 +402,26 @@ func run(c *cli.Context) {
 
 	// TODO: wait for each vertex to exit...
 	log.Println("Goodbye!")
+	return nil
 }
 
 func main() {
-	//if DEBUG {
+	var flags int
-	log.SetFlags(log.LstdFlags | log.Lshortfile)
+	if DEBUG || true { // TODO: remove || true
-	//}
+		flags = log.LstdFlags | log.Lshortfile
-	log.SetFlags(log.Flags() - log.Ldate) // remove the date for now
+	}
+	flags = (flags - log.Ldate) // remove the date for now
+	log.SetFlags(flags)
+
+	// un-hijack from capnslog...
+	log.SetOutput(os.Stderr)
+	if VERBOSE {
+		capnslog.SetFormatter(capnslog.NewLogFormatter(os.Stderr, "(etcd) ", flags))
+	} else {
+		capnslog.SetFormatter(capnslog.NewNilFormatter())
+	}
+
+	// test for sanity
 	if program == "" || version == "" {
 		log.Fatal("Program was not compiled correctly. Please see Makefile.")
 	}
@@ -231,9 +439,10 @@ func main() {
 			Action:  run,
 			Flags: []cli.Flag{
 				cli.StringFlag{
-					Name:  "file, f",
+					Name:   "file, f",
-					Value: "",
+					Value:  "",
-					Usage: "graph definition to run",
+					Usage:  "graph definition to run",
+					EnvVar: "MGMT_FILE",
 				},
 				cli.BoolFlag{
 					Name:  "no-watch",
@@ -261,20 +470,105 @@ func main() {
 					Usage: "hostname to use",
 				},
 				// if empty, it will startup a new server
+				cli.StringSliceFlag{
+					Name:   "seeds, s",
+					Value:  &cli.StringSlice{}, // empty slice
+					Usage:  "default etc client endpoint",
+					EnvVar: "MGMT_SEEDS",
+				},
+				// port 2379 and 4001 are common
+				cli.StringSliceFlag{
+					Name:   "client-urls",
+					Value:  &cli.StringSlice{},
+					Usage:  "list of URLs to listen on for client traffic",
+					EnvVar: "MGMT_CLIENT_URLS",
+				},
+				// port 2380 and 7001 are common
+				cli.StringSliceFlag{
+					Name:   "server-urls, peer-urls",
+					Value:  &cli.StringSlice{},
+					Usage:  "list of URLs to listen on for server (peer) traffic",
+					EnvVar: "MGMT_SERVER_URLS",
+				},
+				cli.BoolFlag{
+					Name:  "no-server",
+					Usage: "do not let other servers peer with me",
+				},
+				cli.IntFlag{
+					Name:   "ideal-cluster-size",
+					Value:  defaultIdealClusterSize,
+					Usage:  "ideal number of server peers in cluster, only read by initial server",
+					EnvVar: "MGMT_IDEAL_CLUSTER_SIZE",
+				},
+				cli.IntFlag{
+					Name:   "converged-timeout, t",
+					Value:  -1,
+					Usage:  "exit after approximately this many seconds in a converged state",
+					EnvVar: "MGMT_CONVERGED_TIMEOUT",
+				},
+				cli.IntFlag{
+					Name:   "max-runtime",
+					Value:  0,
+					Usage:  "exit after a maximum of approximately this many seconds",
+					EnvVar: "MGMT_MAX_RUNTIME",
+				},
+				cli.BoolFlag{
+					Name:  "noop",
+					Usage: "globally force all resources into no-op mode",
+				},
 				cli.StringFlag{
-					Name:  "seed, s",
+					Name:  "puppet, p",
 					Value: "",
-					Usage: "default etc peer endpoint",
+					Usage: "load graph from puppet, optionally takes a manifest or path to manifest file",
+				},
+				cli.StringFlag{
+					Name:  "puppet-conf",
+					Value: "",
+					Usage: "supply the path to an alternate puppet.conf file to use",
+				},
+				cli.StringSliceFlag{
+					Name:  "remote",
+					Value: &cli.StringSlice{},
+					Usage: "list of remote graph definitions to run",
+				},
+				cli.BoolFlag{
+					Name:  "allow-interactive",
+					Usage: "allow interactive prompting, such as for remote passwords",
+				},
+				cli.StringFlag{
+					Name:   "ssh-priv-id-rsa",
+					Value:  "~/.ssh/id_rsa",
+					Usage:  "default path to ssh key file, set empty to never touch",
+					EnvVar: "MGMT_SSH_PRIV_ID_RSA",
 				},
 				cli.IntFlag{
-					Name:  "converged-timeout, t",
+					Name:   "cconns",
-					Value: -1,
+					Value:  0,
-					Usage: "exit after approximately this many seconds in a converged state",
+					Usage:  "number of maximum concurrent remote ssh connections to run, 0 for unlimited",
+					EnvVar: "MGMT_CCONNS",
+				},
+				cli.BoolFlag{
+					Name:  "no-caching",
+					Usage: "don't allow remote caching of remote execution binary",
 				},
 				cli.IntFlag{
-					Name:  "max-runtime",
+					Name:   "depth",
-					Value: 0,
+					Hidden: true, // internal use only
-					Usage: "exit after a maximum of approximately this many seconds",
+					Value:  0,
+					Usage:  "specify depth in remote hierarchy",
+				},
+				cli.StringFlag{
+					Name:   "prefix",
+					Usage:  "specify a path to the working prefix directory",
+					EnvVar: "MGMT_PREFIX",
+				},
+				cli.BoolFlag{
+					Name:  "tmp-prefix",
+					Usage: "request a pseudo-random, temporary prefix to be used",
+				},
+				cli.BoolFlag{
+					Name:  "allow-tmp-prefix",
+					Usage: "allow creation of a new temporary prefix if main prefix is unavailable",
 				},
 			},
 		},

misc.go

@@ -18,15 +18,155 @@
 package main
 
 import (
-	"bytes"
+	"github.com/godbus/dbus"
-	"encoding/base64"
-	"encoding/gob"
 	"path"
+	"sort"
 	"strings"
 	"time"
 )
 
-// Similar to the GNU dirname command
+// FirstToUpper returns the string with the first character capitalized.
+func FirstToUpper(str string) string {
+	if str == "" {
+		return ""
+	}
+	return strings.ToUpper(str[0:1]) + str[1:]
+}
+
+// StrInList returns true if a string exists inside a list, otherwise false.
+func StrInList(needle string, haystack []string) bool {
+	for _, x := range haystack {
+		if needle == x {
+			return true
+		}
+	}
+	return false
+}
+
+// Uint64KeyFromStrInMap returns true if needle is found in haystack of keys
+// that have uint64 type.
+func Uint64KeyFromStrInMap(needle string, haystack map[uint64]string) (uint64, bool) {
+	for k, v := range haystack {
+		if v == needle {
+			return k, true
+		}
+	}
+	return 0, false
+}
+
+// StrRemoveDuplicatesInList removes any duplicate values in the list.
+// This is a possibly sub-optimal, O(n^2)? implementation.
+func StrRemoveDuplicatesInList(list []string) []string {
+	unique := []string{}
+	for _, x := range list {
+		if !StrInList(x, unique) {
+			unique = append(unique, x)
+		}
+	}
+	return unique
+}
+
+// StrFilterElementsInList removes any of the elements in filter, if they exist
+// in the list.
+func StrFilterElementsInList(filter []string, list []string) []string {
+	result := []string{}
+	for _, x := range list {
+		if !StrInList(x, filter) {
+			result = append(result, x)
+		}
+	}
+	return result
+}
+
+// StrListIntersection removes any of the elements in filter, if they don't
+// exist in the list. This is an in order intersection of two lists.
+func StrListIntersection(list1 []string, list2 []string) []string {
+	result := []string{}
+	for _, x := range list1 {
+		if StrInList(x, list2) {
+			result = append(result, x)
+		}
+	}
+	return result
+}
+
+// ReverseStringList reverses a list of strings.
+func ReverseStringList(in []string) []string {
+	var out []string // empty list
+	l := len(in)
+	for i := range in {
+		out = append(out, in[l-i-1])
+	}
+	return out
+}
+
+// StrMapKeys return the sorted list of string keys in a map with string keys.
+// NOTE: i thought it would be nice for this to use: map[string]interface{} but
+// it turns out that's not allowed. I know we don't have generics, but come on!
+func StrMapKeys(m map[string]string) []string {
+	result := []string{}
+	for k := range m {
+		result = append(result, k)
+	}
+	sort.Strings(result) // deterministic order
+	return result
+}
+
+// StrMapKeysUint64 return the sorted list of string keys in a map with string
+// keys but uint64 values.
+func StrMapKeysUint64(m map[string]uint64) []string {
+	result := []string{}
+	for k := range m {
+		result = append(result, k)
+	}
+	sort.Strings(result) // deterministic order
+	return result
+}
+
+// BoolMapValues returns the sorted list of bool values in a map with string
+// values.
+func BoolMapValues(m map[string]bool) []bool {
+	result := []bool{}
+	for _, v := range m {
+		result = append(result, v)
+	}
+	//sort.Bools(result) // TODO: deterministic order
+	return result
+}
+
+// StrMapValues returns the sorted list of string values in a map with string
+// values.
+func StrMapValues(m map[string]string) []string {
+	result := []string{}
+	for _, v := range m {
+		result = append(result, v)
+	}
+	sort.Strings(result) // deterministic order
+	return result
+}
+
+// StrMapValuesUint64 return the sorted list of string values in a map with
+// string values.
+func StrMapValuesUint64(m map[uint64]string) []string {
+	result := []string{}
+	for _, v := range m {
+		result = append(result, v)
+	}
+	sort.Strings(result) // deterministic order
+	return result
+}
+
+// BoolMapTrue returns true if everyone in the list is true.
+func BoolMapTrue(l []bool) bool {
+	for _, b := range l {
+		if !b {
+			return false
+		}
+	}
+	return true
+}
+
+// Dirname is similar to the GNU dirname command.
 func Dirname(p string) string {
 	if p == "/" {
 		return ""
@@ -35,20 +175,28 @@ func Dirname(p string) string {
 	return d
 }
 
+// Basename is the base of a path string.
 func Basename(p string) string {
 	_, b := path.Split(path.Clean(p))
+	if p == "" {
+		return ""
+	}
 	if p[len(p)-1:] == "/" { // don't loose the tail slash
 		b += "/"
 	}
 	return b
 }
 
-// Split a path into an array of tokens excluding any trailing empty tokens
+// PathSplit splits a path into an array of tokens excluding any trailing empty
+// tokens.
 func PathSplit(p string) []string {
+	if p == "/" { // TODO: can't this all be expressed nicely in one line?
+		return []string{""}
+	}
 	return strings.Split(path.Clean(p), "/")
 }
 
-// Does path string contain the given path prefix in it?
+// HasPathPrefix tells us if a path string contain the given path prefix in it.
 func HasPathPrefix(p, prefix string) bool {
 
 	patharray := PathSplit(p)
@@ -67,7 +215,51 @@ func HasPathPrefix(p, prefix string) bool {
 	return true
 }
 
-// Delta of path prefix, tells you how many path tokens different the prefix is
+// StrInPathPrefixList returns true if the needle is a PathPrefix in the
+// haystack.
+func StrInPathPrefixList(needle string, haystack []string) bool {
+	for _, x := range haystack {
+		if HasPathPrefix(x, needle) {
+			return true
+		}
+	}
+	return false
+}
+
+// RemoveCommonFilePrefixes removes redundant file path prefixes that are under
+// the tree of other files.
+func RemoveCommonFilePrefixes(paths []string) []string {
+	var result = make([]string, len(paths))
+	for i := 0; i < len(paths); i++ { // copy, b/c append can modify the args!!
+		result[i] = paths[i]
+	}
+	// is there a string path which is common everywhere?
+	// if so, remove it, and iterate until nothing common is left
+	// return what's left over, that's the most common superset
+loop:
+	for {
+		if len(result) <= 1 {
+			return result
+		}
+		for i := 0; i < len(result); i++ {
+			var copied = make([]string, len(result))
+			for j := 0; j < len(result); j++ { // copy, b/c append can modify the args!!
+				copied[j] = result[j]
+			}
+			noi := append(copied[:i], copied[i+1:]...) // rm i
+			if StrInPathPrefixList(result[i], noi) {
+				// delete the element common to everyone
+				result = noi
+				continue loop
+			}
+		}
+		break
+	}
+	return result
+}
+
+// PathPrefixDelta returns the delta of the path prefix, which tells you how
+// many path tokens different the prefix is.
 func PathPrefixDelta(p, prefix string) int {
 
 	if !HasPathPrefix(p, prefix) {
@@ -78,45 +270,100 @@ func PathPrefixDelta(p, prefix string) int {
 	return len(patharray) - len(prefixarray)
 }
 
+// PathIsDir returns true if there is a trailing slash.
 func PathIsDir(p string) bool {
 	return p[len(p)-1:] == "/" // a dir has a trailing slash in this context
 }
 
-// encode an object as base 64, serialize and then base64 encode
+// PathSplitFullReversed returns the full list of "dependency" paths for a given
-func ObjToB64(obj interface{}) (string, bool) {
+// path in reverse order.
-	b := bytes.Buffer{}
+func PathSplitFullReversed(p string) []string {
-	e := gob.NewEncoder(&b)
+	var result []string
-	err := e.Encode(obj)
+	split := PathSplit(p)
-	if err != nil {
+	count := len(split)
-		//log.Println("Gob failed to Encode: ", err)
+	var x string
-		return "", false
+	for i := 0; i < count; i++ {
+		x = "/" + path.Join(split[0:i+1]...)
+		if i != 0 && !(i+1 == count && !PathIsDir(p)) {
+			x += "/" // add trailing slash
+		}
+		result = append(result, x)
 	}
-	return base64.StdEncoding.EncodeToString(b.Bytes()), true
+	return ReverseStringList(result)
 }
 
-// TODO: is it possible to somehow generically just return the obj?
+// DirifyFileList adds trailing slashes to any likely dirs in a package manager
-// decode an object into the waiting obj which you pass a reference to
+// fileList if removeDirs is true, otherwise, don't keep the dirs in our output.
-func B64ToObj(str string, obj interface{}) bool {
+func DirifyFileList(fileList []string, removeDirs bool) []string {
-	bb, err := base64.StdEncoding.DecodeString(str)
+	dirs := []string{}
-	if err != nil {
+	for _, file := range fileList {
-		//log.Println("Base64 failed to Decode: ", err)
+		dir, _ := path.Split(file) // dir
-		return false
+		dir = path.Clean(dir)      // clean so cmp is easier
+		if !StrInList(dir, dirs) {
+			dirs = append(dirs, dir)
+		}
 	}
-	b := bytes.NewBuffer(bb)
+
-	d := gob.NewDecoder(b)
+	result := []string{}
-	err = d.Decode(obj)
+	for _, file := range fileList {
-	if err != nil {
+		cleanFile := path.Clean(file)
-		//log.Println("Gob failed to Decode: ", err)
+		if !StrInList(cleanFile, dirs) { // we're not a directory!
-		return false
+			result = append(result, file) // pass through
+		} else if !removeDirs {
+			result = append(result, cleanFile+"/")
+		}
 	}
-	return true
+
+	return result
 }
 
-// special version of time.After that blocks when given a negative integer
+// FlattenListWithSplit flattens a list of input by splitting each element by
-// when used in a case statement, the timer restarts on each select call to it
+// any and all of the strings listed in the split array
+func FlattenListWithSplit(input []string, split []string) []string {
+	if len(split) == 0 { // nothing to split by
+		return input
+	}
+	out := []string{}
+	for _, x := range input {
+		s := []string{}
+		if len(split) == 1 {
+			s = strings.Split(x, split[0]) // split by only string
+		} else {
+			s = []string{x} // initial
+			for i := range split {
+				s = FlattenListWithSplit(s, []string{split[i]}) // recurse
+			}
+		}
+		out = append(out, s...)
+	}
+	return out
+}
+
+// TimeAfterOrBlock is aspecial version of time.After that blocks when given a
+// negative integer. When used in a case statement, the timer restarts on each
+// select call to it.
 func TimeAfterOrBlock(t int) <-chan time.Time {
 	if t < 0 {
 		return make(chan time.Time) // blocks forever
 	}
 	return time.After(time.Duration(t) * time.Second)
 }
+
+// SystemBusPrivateUsable makes using the private bus usable
+// TODO: should be upstream: https://github.com/godbus/dbus/issues/15
+func SystemBusPrivateUsable() (conn *dbus.Conn, err error) {
+	conn, err = dbus.SystemBusPrivate()
+	if err != nil {
+		return nil, err
+	}
+	if err = conn.Auth(nil); err != nil {
+		conn.Close()
+		conn = nil
+		return
+	}
+	if err = conn.Hello(); err != nil {
+		conn.Close()
+		conn = nil
+	}
+	return conn, nil // success
+}

misc/example.conf

@@ -0,0 +1 @@
+# example mgmt configuration file, currently has no options at the moment!

misc/go

@@ -0,0 +1,15 @@
+#!/bin/bash
+# hack around stupid $GOPATH semantics, with ~/bin/go helper
+# thanks to Nilium in #go-nuts for 1/3 of the idea
+[ -z "$GOPATH" ] && echo '$GOPATH is not set!' && exit 1
+GO="$(which -a go | sed -e '2q;d')"	# TODO: pick /usr/bin/go in a better way
+if [ "$1" = "generate" ]; then
+	exec $GO "$@"	# go generate is stupid and gets confused by $GOPATH
+fi
+# the idea is to have $project/gopath/src/ be a symlink to ../vendor but you put
+# all of your vendored things in vendor/ but with this gopath can be per project
+if [ -d "$PWD/vendor/" ] && [ -d "$PWD/gopath/" ] && [ "`readlink $PWD/gopath/src`" = "../vendor" ] ; then
+	GOPATH="$PWD/gopath/:$GOPATH" $GO "$@"
+else
+	$GO "$@"
+fi

misc/make-deps.sh

@@ -1,11 +1,16 @@
 #!/bin/bash
 # setup a simple go environment
+XPWD=`pwd`
+ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && cd .. && pwd )"	# dir!
+cd "${ROOT}" >/dev/null
 
 travis=0
 if env | grep -q '^TRAVIS=true$'; then
 	travis=1
 fi
 
+sudo_command=$(which sudo)
+
 if [ $travis -eq 0 ]; then
 	YUM=`which yum 2>/dev/null`
 	APT=`which apt-get 2>/dev/null`
@@ -15,25 +20,32 @@ if [ $travis -eq 0 ]; then
 	fi
 	if [ ! -z "$YUM" ]; then
 		# some go dependencies are stored in mercurial
-		sudo $YUM install -y golang golang-googlecode-tools-stringer hg
+		$sudo_command $YUM install -y golang golang-googlecode-tools-stringer hg
 
 	fi
 	if [ ! -z "$APT" ]; then
-		sudo $APT install -y golang mercurial
+		$sudo_command $APT update
-
+		$sudo_command $APT install -y golang make gcc packagekit mercurial
+		# one of these two golang tools packages should work on debian
+		$sudo_command $APT install -y golang-golang-x-tools || true
+		$sudo_command $APT install -y golang-go.tools || true
 	fi
 fi
 
-# build etcd
+# if golang is too old, we don't want to fail with an obscure error later
-git clone --recursive https://github.com/coreos/etcd/ && cd etcd
+if go version | grep 'go1\.[0123]\.'; then
-git checkout v2.2.4	# TODO: update to newer versions as needed
+	echo "mgmt requires go1.4 or higher."
-[ -x build ] && ./build
+	exit 1
-mkdir -p ~/bin/
+fi
-cp bin/etcd ~/bin/
-cd -
-rm -rf etcd	# clean up to avoid failing on upstream gofmt errors
 
 go get ./...	# get all the go dependencies
 [ -e "$GOBIN/mgmt" ] && rm -f "$GOBIN/mgmt"	# the `go get` version has no -X
-go get golang.org/x/tools/cmd/vet	# add in `go vet` for travis
+# vet is built-in in go 1.6 - we check for go vet command
+go vet 1> /dev/null 2>&1
+ret=$?
+if [[ $ret != 0 ]]; then
+	go get golang.org/x/tools/cmd/vet      # add in `go vet` for travis
+fi
 go get golang.org/x/tools/cmd/stringer	# for automatic stringer-ing
+go get github.com/golang/lint/golint	# for `golint`-ing
+cd "$XPWD" >/dev/null

misc/mgmt.conf.example

@@ -1 +0,0 @@
-# example mgmt configuration file, currently has not options at the moment!

misc/mgmt.service

@@ -0,0 +1,13 @@
+[Unit]
+Description=Run mgmt configuration management
+Documentation=https://github.com/purpleidea/mgmt/
+After=systemd-networkd.service
+Requires=systemd-networkd.service
+
+[Service]
+ExecStart=/usr/bin/mgmt run ${OPTS}
+RestartSec=5s
+Restart=always
+
+[Install]
+WantedBy=multi-user.target

misc_test.go

@@ -18,7 +18,8 @@
 package main
 
 import (
-	"fmt"
+	"reflect"
+	"sort"
 	"testing"
 )
 
@@ -32,6 +33,10 @@ func TestMiscT1(t *testing.T) {
 		t.Errorf("Result is incorrect.")
 	}
 
+	if Dirname("/foo/") != "/" {
+		t.Errorf("Result is incorrect.")
+	}
+
 	if Dirname("/") != "" { // TODO: should this equal "/" or "" ?
 		t.Errorf("Result is incorrect.")
 	}
@@ -44,15 +49,29 @@ func TestMiscT1(t *testing.T) {
 		t.Errorf("Result is incorrect.")
 	}
 
+	if Basename("/foo/") != "foo/" {
+		t.Errorf("Result is incorrect.")
+	}
+
 	if Basename("/") != "/" { // TODO: should this equal "" or "/" ?
 		t.Errorf("Result is incorrect.")
 	}
 
+	if Basename("") != "" { // TODO: should this equal something different?
+		t.Errorf("Result is incorrect.")
+	}
 }
 
 func TestMiscT2(t *testing.T) {
 
 	// TODO: compare the output with the actual list
+	p0 := "/"
+	r0 := []string{""} // TODO: is this correct?
+	if len(PathSplit(p0)) != len(r0) {
+		t.Errorf("Result should be: %q.", r0)
+		t.Errorf("Result should have a length of: %v.", len(r0))
+	}
+
 	p1 := "/foo/bar/baz"
 	r1 := []string{"", "foo", "bar", "baz"}
 	if len(PathSplit(p1)) != len(r1) {
@@ -92,6 +111,10 @@ func TestMiscT3(t *testing.T) {
 	if HasPathPrefix("/foo/bar/baz/", "/foo/bar/baz/dude") != false {
 		t.Errorf("Result should be false.")
 	}
+
+	if HasPathPrefix("/foo/bar/baz/boo/", "/foo/") != true {
+		t.Errorf("Result should be true.")
+	}
 }
 
 func TestMiscT4(t *testing.T) {
@@ -148,53 +171,642 @@ func TestMiscT5(t *testing.T) {
 	}
 }
 
-func TestMiscT6(t *testing.T) {
+func TestMiscT8(t *testing.T) {
 
-	type foo struct {
+	r0 := []string{"/"}
-		Name  string `yaml:"name"`
+	if fullList0 := PathSplitFullReversed("/"); !reflect.DeepEqual(r0, fullList0) {
-		Type  string `yaml:"type"`
+		t.Errorf("PathSplitFullReversed expected: %v; got: %v.", r0, fullList0)
-		Value int    `yaml:"value"`
 	}
 
-	obj := foo{"dude", "sweet", 42}
+	r1 := []string{"/foo/bar/baz/file", "/foo/bar/baz/", "/foo/bar/", "/foo/", "/"}
-	output, ok := ObjToB64(obj)
+	if fullList1 := PathSplitFullReversed("/foo/bar/baz/file"); !reflect.DeepEqual(r1, fullList1) {
-	if ok != true {
+		t.Errorf("PathSplitFullReversed expected: %v; got: %v.", r1, fullList1)
-		t.Errorf("First result should be true.")
 	}
-	var data foo
+
-	if B64ToObj(output, &data) != true {
+	r2 := []string{"/foo/bar/baz/dir/", "/foo/bar/baz/", "/foo/bar/", "/foo/", "/"}
-		t.Errorf("Second result should be true.")
+	if fullList2 := PathSplitFullReversed("/foo/bar/baz/dir/"); !reflect.DeepEqual(r2, fullList2) {
+		t.Errorf("PathSplitFullReversed expected: %v; got: %v.", r2, fullList2)
 	}
-	// TODO: there is probably a better way to compare these two...
+
-	if fmt.Sprintf("%+v\n", obj) != fmt.Sprintf("%+v\n", data) {
+}
-		t.Errorf("Strings should match.")
+
+func TestMiscT9(t *testing.T) {
+	fileListIn := []string{ // list taken from drbd-utils package
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d",
+		"/usr/share/doc/drbd-utils",
+		"/var/lib/drbd",
+	}
+	sort.Strings(fileListIn)
+
+	fileListOut := []string{ // fixed up manually
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d/",               // added trailing slash
+		"/usr/share/doc/drbd-utils/", // added trailing slash
+		"/var/lib/drbd",              // can't be fixed :(
+	}
+	sort.Strings(fileListOut)
+
+	dirify := DirifyFileList(fileListIn, false) // TODO: test with true
+	sort.Strings(dirify)
+	equals := reflect.DeepEqual(fileListOut, dirify)
+	if a, b := len(fileListOut), len(dirify); a != b {
+		t.Errorf("DirifyFileList counts didn't match: %d != %d", a, b)
+	} else if !equals {
+		t.Error("DirifyFileList did not match expected!")
+		for i := 0; i < len(dirify); i++ {
+			if fileListOut[i] != dirify[i] {
+				t.Errorf("# %d: %v <> %v", i, fileListOut[i], dirify[i])
+			}
+		}
 	}
 }
 
-func TestMiscT7(t *testing.T) {
+func TestMiscT10(t *testing.T) {
+	fileListIn := []string{ // fake package list
+		"/etc/drbd.conf",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d",
+		"/etc/drbd.d/foo",
+		"/var/lib/drbd",
+		"/var/somedir/",
+	}
+	sort.Strings(fileListIn)
 
-	type Foo struct {
+	fileListOut := []string{ // fixed up manually
-		Name  string `yaml:"name"`
+		"/etc/drbd.conf",
-		Type  string `yaml:"type"`
+		"/usr/share/man/man8/drbdsetup.8.gz",
-		Value int    `yaml:"value"`
+		"/etc/drbd.d/", // added trailing slash
+		"/etc/drbd.d/foo",
+		"/var/lib/drbd", // can't be fixed :(
+		"/var/somedir/", // stays the same
 	}
+	sort.Strings(fileListOut)
 
-	type bar struct {
+	dirify := DirifyFileList(fileListIn, false) // TODO: test with true
-		Foo     `yaml:",inline"` // anonymous struct must be public!
+	sort.Strings(dirify)
-		Comment string           `yaml:"comment"`
+	equals := reflect.DeepEqual(fileListOut, dirify)
-	}
+	if a, b := len(fileListOut), len(dirify); a != b {
-
+		t.Errorf("DirifyFileList counts didn't match: %d != %d", a, b)
-	obj := bar{Foo{"dude", "sweet", 42}, "hello world"}
+	} else if !equals {
-	output, ok := ObjToB64(obj)
+		t.Error("DirifyFileList did not match expected!")
-	if ok != true {
+		for i := 0; i < len(dirify); i++ {
-		t.Errorf("First result should be true.")
+			if fileListOut[i] != dirify[i] {
-	}
+				t.Errorf("# %d: %v <> %v", i, fileListOut[i], dirify[i])
-	var data bar
+			}
-	if B64ToObj(output, &data) != true {
+		}
-		t.Errorf("Second result should be true.")
+	}
-	}
+}
-	// TODO: there is probably a better way to compare these two...
+
-	if fmt.Sprintf("%+v\n", obj) != fmt.Sprintf("%+v\n", data) {
+func TestMiscT11(t *testing.T) {
-		t.Errorf("Strings should match.")
+	in1 := []string{"/", "/usr/", "/usr/lib/", "/usr/share/"} // input
+	ex1 := []string{"/usr/lib/", "/usr/share/"}               // expected
+	sort.Strings(ex1)
+	out1 := RemoveCommonFilePrefixes(in1)
+	sort.Strings(out1)
+	if !reflect.DeepEqual(ex1, out1) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex1, out1)
+	}
+
+	in2 := []string{"/", "/usr/"}
+	ex2 := []string{"/usr/"}
+	sort.Strings(ex2)
+	out2 := RemoveCommonFilePrefixes(in2)
+	sort.Strings(out2)
+	if !reflect.DeepEqual(ex2, out2) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex2, out2)
+	}
+
+	in3 := []string{"/"}
+	ex3 := []string{"/"}
+	out3 := RemoveCommonFilePrefixes(in3)
+	if !reflect.DeepEqual(ex3, out3) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex3, out3)
+	}
+
+	in4 := []string{"/usr/bin/foo", "/usr/bin/bar", "/usr/lib/", "/usr/share/"}
+	ex4 := []string{"/usr/bin/foo", "/usr/bin/bar", "/usr/lib/", "/usr/share/"}
+	sort.Strings(ex4)
+	out4 := RemoveCommonFilePrefixes(in4)
+	sort.Strings(out4)
+	if !reflect.DeepEqual(ex4, out4) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex4, out4)
+	}
+
+	in5 := []string{"/usr/bin/foo", "/usr/bin/bar", "/usr/lib/", "/usr/share/", "/usr/bin"}
+	ex5 := []string{"/usr/bin/foo", "/usr/bin/bar", "/usr/lib/", "/usr/share/"}
+	sort.Strings(ex5)
+	out5 := RemoveCommonFilePrefixes(in5)
+	sort.Strings(out5)
+	if !reflect.DeepEqual(ex5, out5) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex5, out5)
+	}
+
+	in6 := []string{"/etc/drbd.d/", "/lib/drbd/", "/usr/lib/drbd/", "/usr/lib/systemd/system/", "/usr/lib/tmpfiles.d/", "/usr/sbin/", "/usr/share/doc/drbd-utils/", "/usr/share/man/man5/", "/usr/share/man/man8/", "/usr/share/doc/", "/var/lib/"}
+	ex6 := []string{"/etc/drbd.d/", "/lib/drbd/", "/usr/lib/drbd/", "/usr/lib/systemd/system/", "/usr/lib/tmpfiles.d/", "/usr/sbin/", "/usr/share/doc/drbd-utils/", "/usr/share/man/man5/", "/usr/share/man/man8/", "/var/lib/"}
+	sort.Strings(ex6)
+	out6 := RemoveCommonFilePrefixes(in6)
+	sort.Strings(out6)
+	if !reflect.DeepEqual(ex6, out6) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex6, out6)
+	}
+
+	in7 := []string{"/etc/", "/lib/", "/usr/lib/", "/usr/lib/systemd/", "/usr/", "/usr/share/doc/", "/usr/share/man/", "/var/"}
+	ex7 := []string{"/etc/", "/lib/", "/usr/lib/systemd/", "/usr/share/doc/", "/usr/share/man/", "/var/"}
+	sort.Strings(ex7)
+	out7 := RemoveCommonFilePrefixes(in7)
+	sort.Strings(out7)
+	if !reflect.DeepEqual(ex7, out7) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex7, out7)
+	}
+
+	in8 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/etc/drbd.d/",
+		"/usr/share/doc/drbd-utils/",
+		"/var/lib/drbd",
+	}
+	ex8 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/var/lib/drbd",
+	}
+	sort.Strings(ex8)
+	out8 := RemoveCommonFilePrefixes(in8)
+	sort.Strings(out8)
+	if !reflect.DeepEqual(ex8, out8) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex8, out8)
+	}
+
+	in9 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/",
+		"/lib/drbd/drbd",
+		"/lib/drbd/",
+		"/lib/drbd/",
+		"/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/drbd/",
+		"/usr/lib/systemd/system/",
+		"/usr/lib/tmpfiles.d/",
+		"/usr/sbin/",
+		"/usr/sbin/",
+		"/usr/share/doc/drbd-utils/",
+		"/usr/share/doc/drbd-utils/",
+		"/usr/share/man/man5/",
+		"/usr/share/man/man5/",
+		"/usr/share/man/man8/",
+		"/usr/share/man/man8/",
+		"/usr/share/man/man8/",
+		"/etc/drbd.d/",
+		"/usr/share/doc/drbd-utils/",
+		"/var/lib/drbd",
+	}
+	ex9 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/",
+		"/lib/drbd/drbd",
+		"/usr/lib/drbd/",
+		"/usr/lib/systemd/system/",
+		"/usr/lib/tmpfiles.d/",
+		"/usr/sbin/",
+		"/usr/share/doc/drbd-utils/",
+		"/usr/share/man/man5/",
+		"/usr/share/man/man8/",
+		"/var/lib/drbd",
+	}
+	sort.Strings(ex9)
+	out9 := RemoveCommonFilePrefixes(in9)
+	sort.Strings(out9)
+	if !reflect.DeepEqual(ex9, out9) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex9, out9)
+	}
+
+	in10 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/",                   // watch me, i'm a dir
+		"/etc/drbd.d/global_common.conf", // and watch me i'm a file!
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/", // watch me, i'm a dir too
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/var/lib/drbd",
+	}
+	ex10 := []string{
+		"/etc/drbd.conf",
+		"/etc/drbd.d/global_common.conf",
+		"/lib/drbd/drbd",
+		"/lib/drbd/drbdadm-83",
+		"/lib/drbd/drbdadm-84",
+		"/lib/drbd/drbdsetup-83",
+		"/lib/drbd/drbdsetup-84",
+		"/usr/lib/drbd/crm-fence-peer.sh",
+		"/usr/lib/drbd/crm-unfence-peer.sh",
+		"/usr/lib/drbd/notify-emergency-reboot.sh",
+		"/usr/lib/drbd/notify-emergency-shutdown.sh",
+		"/usr/lib/drbd/notify-io-error.sh",
+		"/usr/lib/drbd/notify-out-of-sync.sh",
+		"/usr/lib/drbd/notify-pri-lost-after-sb.sh",
+		"/usr/lib/drbd/notify-pri-lost.sh",
+		"/usr/lib/drbd/notify-pri-on-incon-degr.sh",
+		"/usr/lib/drbd/notify-split-brain.sh",
+		"/usr/lib/drbd/notify.sh",
+		"/usr/lib/drbd/outdate-peer.sh",
+		"/usr/lib/drbd/rhcs_fence",
+		"/usr/lib/drbd/snapshot-resync-target-lvm.sh",
+		"/usr/lib/drbd/stonith_admin-fence-peer.sh",
+		"/usr/lib/drbd/unsnapshot-resync-target-lvm.sh",
+		"/usr/lib/systemd/system/drbd.service",
+		"/usr/lib/tmpfiles.d/drbd.conf",
+		"/usr/sbin/drbd-overview",
+		"/usr/sbin/drbdadm",
+		"/usr/sbin/drbdmeta",
+		"/usr/sbin/drbdsetup",
+		"/usr/share/doc/drbd-utils/COPYING",
+		"/usr/share/doc/drbd-utils/ChangeLog",
+		"/usr/share/doc/drbd-utils/README",
+		"/usr/share/doc/drbd-utils/drbd.conf.example",
+		"/usr/share/man/man5/drbd.conf-8.3.5.gz",
+		"/usr/share/man/man5/drbd.conf-8.4.5.gz",
+		"/usr/share/man/man5/drbd.conf-9.0.5.gz",
+		"/usr/share/man/man5/drbd.conf.5.gz",
+		"/usr/share/man/man8/drbd-8.3.8.gz",
+		"/usr/share/man/man8/drbd-8.4.8.gz",
+		"/usr/share/man/man8/drbd-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview-9.0.8.gz",
+		"/usr/share/man/man8/drbd-overview.8.gz",
+		"/usr/share/man/man8/drbd.8.gz",
+		"/usr/share/man/man8/drbdadm-8.3.8.gz",
+		"/usr/share/man/man8/drbdadm-8.4.8.gz",
+		"/usr/share/man/man8/drbdadm-9.0.8.gz",
+		"/usr/share/man/man8/drbdadm.8.gz",
+		"/usr/share/man/man8/drbddisk-8.3.8.gz",
+		"/usr/share/man/man8/drbddisk-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.3.8.gz",
+		"/usr/share/man/man8/drbdmeta-8.4.8.gz",
+		"/usr/share/man/man8/drbdmeta-9.0.8.gz",
+		"/usr/share/man/man8/drbdmeta.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.3.8.gz",
+		"/usr/share/man/man8/drbdsetup-8.4.8.gz",
+		"/usr/share/man/man8/drbdsetup-9.0.8.gz",
+		"/usr/share/man/man8/drbdsetup.8.gz",
+		"/var/lib/drbd",
+	}
+	sort.Strings(ex10)
+	out10 := RemoveCommonFilePrefixes(in10)
+	sort.Strings(out10)
+	if !reflect.DeepEqual(ex10, out10) {
+		t.Errorf("RemoveCommonFilePrefixes expected: %v; got: %v.", ex10, out10)
+		for i := 0; i < len(ex10); i++ {
+			if ex10[i] != out10[i] {
+				t.Errorf("# %d: %v <> %v", i, ex10[i], out10[i])
+			}
+		}
+	}
+}
+
+func TestMiscFlattenListWithSplit1(t *testing.T) {
+	{
+		in := []string{} // input
+		ex := []string{} // expected
+		out := FlattenListWithSplit(in, []string{",", ";", " "})
+		sort.Strings(out)
+		sort.Strings(ex)
+		if !reflect.DeepEqual(ex, out) {
+			t.Errorf("FlattenListWithSplit expected: %v; got: %v.", ex, out)
+		}
+	}
+
+	{
+		in := []string{"hey"} // input
+		ex := []string{"hey"} // expected
+		out := FlattenListWithSplit(in, []string{",", ";", " "})
+		sort.Strings(out)
+		sort.Strings(ex)
+		if !reflect.DeepEqual(ex, out) {
+			t.Errorf("FlattenListWithSplit expected: %v; got: %v.", ex, out)
+		}
+	}
+
+	{
+		in := []string{"a", "b", "c", "d"} // input
+		ex := []string{"a", "b", "c", "d"} // expected
+		out := FlattenListWithSplit(in, []string{",", ";", " "})
+		sort.Strings(out)
+		sort.Strings(ex)
+		if !reflect.DeepEqual(ex, out) {
+			t.Errorf("FlattenListWithSplit expected: %v; got: %v.", ex, out)
+		}
+	}
+
+	{
+		in := []string{"a,b,c,d"}          // input
+		ex := []string{"a", "b", "c", "d"} // expected
+		out := FlattenListWithSplit(in, []string{",", ";", " "})
+		sort.Strings(out)
+		sort.Strings(ex)
+		if !reflect.DeepEqual(ex, out) {
+			t.Errorf("FlattenListWithSplit expected: %v; got: %v.", ex, out)
+		}
+	}
+
+	{
+		in := []string{"a,b;c d"}          // input (mixed)
+		ex := []string{"a", "b", "c", "d"} // expected
+		out := FlattenListWithSplit(in, []string{",", ";", " "})
+		sort.Strings(out)
+		sort.Strings(ex)
+		if !reflect.DeepEqual(ex, out) {
+			t.Errorf("FlattenListWithSplit expected: %v; got: %v.", ex, out)
+		}
+	}
+
+	{
+		in := []string{"a,b,c,d;e,f,g,h;i,j,k,l;m,n,o,p q,r,s,t;u,v,w,x y z"}                                                                            // input (mixed)
+		ex := []string{"a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z"} // expected
+		out := FlattenListWithSplit(in, []string{",", ";", " "})
+		sort.Strings(out)
+		sort.Strings(ex)
+		if !reflect.DeepEqual(ex, out) {
+			t.Errorf("FlattenListWithSplit expected: %v; got: %v.", ex, out)
+		}
 	}
 }

noop.go

@@ -0,0 +1,155 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"encoding/gob"
+	"log"
+)
+
+func init() {
+	gob.Register(&NoopRes{})
+}
+
+// NoopRes is a no-op resource that does nothing.
+type NoopRes struct {
+	BaseRes `yaml:",inline"`
+	Comment string `yaml:"comment"` // extra field for example purposes
+}
+
+// NewNoopRes is a constructor for this resource. It also calls Init() for you.
+func NewNoopRes(name string) *NoopRes {
+	obj := &NoopRes{
+		BaseRes: BaseRes{
+			Name: name,
+		},
+		Comment: "",
+	}
+	obj.Init()
+	return obj
+}
+
+// Init runs some startup code for this resource.
+func (obj *NoopRes) Init() {
+	obj.BaseRes.kind = "Noop"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+}
+
+// validate if the params passed in are valid data
+// FIXME: where should this get called ?
+func (obj *NoopRes) Validate() bool {
+	return true
+}
+
+// Watch is the primary listener for this resource and it outputs events.
+func (obj *NoopRes) Watch(processChan chan Event) {
+	if obj.IsWatching() {
+		return
+	}
+	obj.SetWatching(true)
+	defer obj.SetWatching(false)
+	cuuid := obj.converger.Register()
+	defer cuuid.Unregister()
+
+	var send = false // send event?
+	var exit = false
+	for {
+		obj.SetState(resStateWatching) // reset
+		select {
+		case event := <-obj.events:
+			cuuid.SetConverged(false)
+			// we avoid sending events on unpause
+			if exit, send = obj.ReadEvent(&event); exit {
+				return // exit
+			}
+
+		case <-cuuid.ConvergedTimer():
+			cuuid.SetConverged(true) // converged!
+			continue
+		}
+
+		// do all our event sending all together to avoid duplicate msgs
+		if send {
+			send = false
+			// only do this on certain types of events
+			//obj.isStateOK = false // something made state dirty
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
+		}
+	}
+}
+
+// CheckApply method for Noop resource. Does nothing, returns happy!
+func (obj *NoopRes) CheckApply(apply bool) (checkok bool, err error) {
+	log.Printf("%v[%v]: CheckApply(%t)", obj.Kind(), obj.GetName(), apply)
+	return true, nil // state is always okay
+}
+
+// NoopUUID is the UUID struct for NoopRes.
+type NoopUUID struct {
+	BaseUUID
+	name string
+}
+
+// The AutoEdges method returns the AutoEdges. In this case none are used.
+func (obj *NoopRes) AutoEdges() AutoEdge {
+	return nil
+}
+
+// GetUUIDs includes all params to make a unique identification of this object.
+// Most resources only return one, although some resources can return multiple.
+func (obj *NoopRes) GetUUIDs() []ResUUID {
+	x := &NoopUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		name:     obj.Name,
+	}
+	return []ResUUID{x}
+}
+
+// GroupCmp returns whether two resources can be grouped together or not.
+func (obj *NoopRes) GroupCmp(r Res) bool {
+	_, ok := r.(*NoopRes)
+	if !ok {
+		// NOTE: technically we could group a noop into any other
+		// resource, if that resource knew how to handle it, although,
+		// since the mechanics of inter-kind resource grouping are
+		// tricky, avoid doing this until there's a good reason.
+		return false
+	}
+	return true // noop resources can always be grouped together!
+}
+
+// Compare two resources and return if they are equivalent.
+func (obj *NoopRes) Compare(res Res) bool {
+	switch res.(type) {
+	// we can only compare NoopRes to others of the same resource
+	case *NoopRes:
+		res := res.(*NoopRes)
+		// calling base Compare is unneeded for the noop res
+		//if !obj.BaseRes.Compare(res) { // call base Compare
+		//	return false
+		//}
+		if obj.Name != res.Name {
+			return false
+		}
+	default:
+		return false
+	}
+	return true
+}

packagekit.go

@@ -0,0 +1,923 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+// DOCS: https://www.freedesktop.org/software/PackageKit/gtk-doc/index.html
+
+//package packagekit // TODO
+package main
+
+import (
+	"fmt"
+	"github.com/godbus/dbus"
+	"log"
+	"runtime"
+	"strings"
+)
+
+// global tweaks of verbosity and code path
+const (
+	PK_DEBUG = false
+	PARANOID = false // enable if you see any ghosts
+)
+
+// constants which might need to be tweaked or which contain special dbus strings.
+const (
+	// FIXME: if PkBufferSize is too low, install seems to drop signals
+	PkBufferSize = 1000
+	// TODO: the PkSignalTimeout value might be too low
+	PkSignalPackageTimeout = 60 // 60 seconds, arbitrary
+	PkSignalDestroyTimeout = 15 // 15 seconds, arbitrary
+	PkPath                 = "/org/freedesktop/PackageKit"
+	PkIface                = "org.freedesktop.PackageKit"
+	PkIfaceTransaction     = PkIface + ".Transaction"
+	dbusAddMatch           = "org.freedesktop.DBus.AddMatch"
+)
+
+var (
+	// PkArchMap contains the mapping from PackageKit arch to GOARCH.
+	// GOARCH's: 386, amd64, arm, arm64, mips64, mips64le, ppc64, ppc64le
+	PkArchMap = map[string]string{ // map of PackageKit arch to GOARCH
+		// TODO: add more values
+		// noarch
+		"noarch": "ANY", // special value "ANY" (noarch as seen in Fedora)
+		"all":    "ANY", // special value "ANY" ('all' as seen in Debian)
+		// fedora
+		"x86_64":  "amd64",
+		"aarch64": "arm64",
+		// debian, from: https://www.debian.org/ports/
+		"amd64": "amd64",
+		"arm64": "arm64",
+		"i386":  "386",
+		"i486":  "386",
+		"i586":  "386",
+		"i686":  "386",
+	}
+)
+
+//type enum_filter uint64
+// https://github.com/hughsie/PackageKit/blob/master/lib/packagekit-glib2/pk-enum.c
+const ( //static const PkEnumMatch enum_filter[]
+	PK_FILTER_ENUM_UNKNOWN         uint64 = 1 << iota // "unknown"
+	PK_FILTER_ENUM_NONE                               // "none"
+	PK_FILTER_ENUM_INSTALLED                          // "installed"
+	PK_FILTER_ENUM_NOT_INSTALLED                      // "~installed"
+	PK_FILTER_ENUM_DEVELOPMENT                        // "devel"
+	PK_FILTER_ENUM_NOT_DEVELOPMENT                    // "~devel"
+	PK_FILTER_ENUM_GUI                                // "gui"
+	PK_FILTER_ENUM_NOT_GUI                            // "~gui"
+	PK_FILTER_ENUM_FREE                               // "free"
+	PK_FILTER_ENUM_NOT_FREE                           // "~free"
+	PK_FILTER_ENUM_VISIBLE                            // "visible"
+	PK_FILTER_ENUM_NOT_VISIBLE                        // "~visible"
+	PK_FILTER_ENUM_SUPPORTED                          // "supported"
+	PK_FILTER_ENUM_NOT_SUPPORTED                      // "~supported"
+	PK_FILTER_ENUM_BASENAME                           // "basename"
+	PK_FILTER_ENUM_NOT_BASENAME                       // "~basename"
+	PK_FILTER_ENUM_NEWEST                             // "newest"
+	PK_FILTER_ENUM_NOT_NEWEST                         // "~newest"
+	PK_FILTER_ENUM_ARCH                               // "arch"
+	PK_FILTER_ENUM_NOT_ARCH                           // "~arch"
+	PK_FILTER_ENUM_SOURCE                             // "source"
+	PK_FILTER_ENUM_NOT_SOURCE                         // "~source"
+	PK_FILTER_ENUM_COLLECTIONS                        // "collections"
+	PK_FILTER_ENUM_NOT_COLLECTIONS                    // "~collections"
+	PK_FILTER_ENUM_APPLICATION                        // "application"
+	PK_FILTER_ENUM_NOT_APPLICATION                    // "~application"
+	PK_FILTER_ENUM_DOWNLOADED                         // "downloaded"
+	PK_FILTER_ENUM_NOT_DOWNLOADED                     // "~downloaded"
+)
+
+// constants from packagekit c library.
+const ( //static const PkEnumMatch enum_transaction_flag[]
+	PK_TRANSACTION_FLAG_ENUM_NONE            uint64 = 1 << iota // "none"
+	PK_TRANSACTION_FLAG_ENUM_ONLY_TRUSTED                       // "only-trusted"
+	PK_TRANSACTION_FLAG_ENUM_SIMULATE                           // "simulate"
+	PK_TRANSACTION_FLAG_ENUM_ONLY_DOWNLOAD                      // "only-download"
+	PK_TRANSACTION_FLAG_ENUM_ALLOW_REINSTALL                    // "allow-reinstall"
+	PK_TRANSACTION_FLAG_ENUM_JUST_REINSTALL                     // "just-reinstall"
+	PK_TRANSACTION_FLAG_ENUM_ALLOW_DOWNGRADE                    // "allow-downgrade"
+)
+
+// constants from packagekit c library.
+const ( //typedef enum
+	PK_INFO_ENUM_UNKNOWN uint64 = 1 << iota
+	PK_INFO_ENUM_INSTALLED
+	PK_INFO_ENUM_AVAILABLE
+	PK_INFO_ENUM_LOW
+	PK_INFO_ENUM_ENHANCEMENT
+	PK_INFO_ENUM_NORMAL
+	PK_INFO_ENUM_BUGFIX
+	PK_INFO_ENUM_IMPORTANT
+	PK_INFO_ENUM_SECURITY
+	PK_INFO_ENUM_BLOCKED
+	PK_INFO_ENUM_DOWNLOADING
+	PK_INFO_ENUM_UPDATING
+	PK_INFO_ENUM_INSTALLING
+	PK_INFO_ENUM_REMOVING
+	PK_INFO_ENUM_CLEANUP
+	PK_INFO_ENUM_OBSOLETING
+	PK_INFO_ENUM_COLLECTION_INSTALLED
+	PK_INFO_ENUM_COLLECTION_AVAILABLE
+	PK_INFO_ENUM_FINISHED
+	PK_INFO_ENUM_REINSTALLING
+	PK_INFO_ENUM_DOWNGRADING
+	PK_INFO_ENUM_PREPARING
+	PK_INFO_ENUM_DECOMPRESSING
+	PK_INFO_ENUM_UNTRUSTED
+	PK_INFO_ENUM_TRUSTED
+	PK_INFO_ENUM_UNAVAILABLE
+	PK_INFO_ENUM_LAST
+)
+
+// Conn is a wrapper struct so we can pass bus connection around in the struct.
+type Conn struct {
+	conn *dbus.Conn
+}
+
+// PkPackageIDActionData is a struct that is returned by PackagesToPackageIDs in the map values.
+type PkPackageIDActionData struct {
+	Found     bool
+	Installed bool
+	Version   string
+	PackageID string
+	Newest    bool
+}
+
+// NewBus returns a new bus connection.
+func NewBus() *Conn {
+	// if we share the bus with others, we will get each others messages!!
+	bus, err := SystemBusPrivateUsable() // don't share the bus connection!
+	if err != nil {
+		return nil
+	}
+	return &Conn{
+		conn: bus,
+	}
+}
+
+// GetBus gets the dbus connection object.
+func (bus *Conn) GetBus() *dbus.Conn {
+	return bus.conn
+}
+
+// Close closes the dbus connection object.
+func (bus *Conn) Close() error {
+	return bus.conn.Close()
+}
+
+// internal helper to add signal matches to the bus, should only be called once
+func (bus *Conn) matchSignal(ch chan *dbus.Signal, path dbus.ObjectPath, iface string, signals []string) error {
+	if PK_DEBUG {
+		log.Printf("PackageKit: matchSignal(%v, %v, %v, %v)", ch, path, iface, signals)
+	}
+	// eg: gdbus monitor --system --dest org.freedesktop.PackageKit --object-path /org/freedesktop/PackageKit | grep <signal>
+	var call *dbus.Call
+	// TODO: if we make this call many times, we seem to receive signals
+	// that many times... Maybe this should be an object singleton?
+	obj := bus.GetBus().BusObject()
+	pathStr := fmt.Sprintf("%s", path)
+	if len(signals) == 0 {
+		call = obj.Call(dbusAddMatch, 0, "type='signal',path='"+pathStr+"',interface='"+iface+"'")
+	} else {
+		for _, signal := range signals {
+			call = obj.Call(dbusAddMatch, 0, "type='signal',path='"+pathStr+"',interface='"+iface+"',member='"+signal+"'")
+			if call.Err != nil {
+				break
+			}
+		}
+	}
+	if call.Err != nil {
+		return call.Err
+	}
+	// The caller has to make sure that ch is sufficiently buffered; if a
+	// message arrives when a write to c is not possible, it is discarded!
+	// This can be disastrous if we're waiting for a "Finished" signal!
+	bus.GetBus().Signal(ch)
+	return nil
+}
+
+// WatchChanges gets a signal anytime an event happens.
+func (bus *Conn) WatchChanges() (chan *dbus.Signal, error) {
+	ch := make(chan *dbus.Signal, PkBufferSize)
+	// NOTE: the TransactionListChanged signal fires much more frequently,
+	// but with much less specificity. If we're missing events, report the
+	// issue upstream! The UpdatesChanged signal is what hughsie suggested
+	var signal = "UpdatesChanged"
+	err := bus.matchSignal(ch, PkPath, PkIface, []string{signal})
+	if err != nil {
+		return nil, err
+	}
+	if PARANOID { // TODO: this filtering might not be necessary anymore...
+		// try to handle the filtering inside this function!
+		rch := make(chan *dbus.Signal)
+		go func() {
+		loop:
+			for {
+				select {
+				case event := <-ch:
+					// "A receive from a closed channel returns the
+					// zero value immediately": if i get nil here,
+					// it means the channel was closed by someone!!
+					if event == nil { // shared bus issue?
+						log.Println("PackageKit: Hrm, channel was closed!")
+						break loop // TODO: continue?
+					}
+					// i think this was caused by using the shared
+					// bus, but we might as well leave it in for now
+					if event.Path != PkPath || event.Name != fmt.Sprintf("%s.%s", PkIface, signal) {
+						log.Printf("PackageKit: Woops: Event: %+v", event)
+						continue
+					}
+					rch <- event // forward...
+				}
+			}
+			defer close(ch)
+		}()
+		return rch, nil
+	}
+	return ch, nil
+}
+
+// CreateTransaction creates and returns a transaction path.
+func (bus *Conn) CreateTransaction() (dbus.ObjectPath, error) {
+	if PK_DEBUG {
+		log.Println("PackageKit: CreateTransaction()")
+	}
+	var interfacePath dbus.ObjectPath
+	obj := bus.GetBus().Object(PkIface, PkPath)
+	call := obj.Call(fmt.Sprintf("%s.CreateTransaction", PkIface), 0).Store(&interfacePath)
+	if call != nil {
+		return "", call
+	}
+	if PK_DEBUG {
+		log.Printf("PackageKit: CreateTransaction(): %v", interfacePath)
+	}
+	return interfacePath, nil
+}
+
+// ResolvePackages runs the PackageKit Resolve method and returns the result.
+func (bus *Conn) ResolvePackages(packages []string, filter uint64) ([]string, error) {
+	packageIDs := []string{}
+	ch := make(chan *dbus.Signal, PkBufferSize)   // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction() // emits Destroy on close
+	if err != nil {
+		return []string{}, err
+	}
+
+	// add signal matches for Package and Finished which will always be last
+	var signals = []string{"Package", "Finished", "Error", "Destroy"}
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+	if PK_DEBUG {
+		log.Printf("PackageKit: ResolvePackages(): Object(%v, %v)", PkIface, interfacePath)
+	}
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("Resolve"), 0, filter, packages)
+	if PK_DEBUG {
+		log.Println("PackageKit: ResolvePackages(): Call: Success!")
+	}
+	if call.Err != nil {
+		return []string{}, call.Err
+	}
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+			if PK_DEBUG {
+				log.Printf("PackageKit: ResolvePackages(): Signal: %+v", signal)
+			}
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("Package") {
+				//pkg_int, ok := signal.Body[0].(int)
+				packageID, ok := signal.Body[1].(string)
+				// format is: name;version;arch;data
+				if !ok {
+					continue loop
+				}
+				//comment, ok := signal.Body[2].(string)
+				for _, p := range packageIDs {
+					if packageID == p {
+						continue loop // duplicate!
+					}
+				}
+				packageIDs = append(packageIDs, packageID)
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				return []string{}, fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		}
+	}
+	return packageIDs, nil
+}
+
+// IsInstalledList queries a list of packages to see if they are installed.
+func (bus *Conn) IsInstalledList(packages []string) ([]bool, error) {
+	var filter uint64             // initializes at the "zero" value of 0
+	filter += PK_FILTER_ENUM_ARCH // always search in our arch
+	packageIDs, e := bus.ResolvePackages(packages, filter)
+	if e != nil {
+		return nil, fmt.Errorf("ResolvePackages error: %v", e)
+	}
+
+	var m = make(map[string]int)
+	for _, packageID := range packageIDs {
+		s := strings.Split(packageID, ";")
+		//if len(s) != 4 { continue } // this would be a bug!
+		pkg := s[0]
+		flags := strings.Split(s[3], ":")
+		for _, f := range flags {
+			if f == "installed" {
+				if _, exists := m[pkg]; !exists {
+					m[pkg] = 0
+				}
+				m[pkg]++ // if we see pkg installed, increment
+				break
+			}
+		}
+	}
+
+	var r []bool
+	for _, p := range packages {
+		if value, exists := m[p]; exists {
+			r = append(r, value > 0) // at least 1 means installed
+		} else {
+			r = append(r, false)
+		}
+	}
+	return r, nil
+}
+
+// IsInstalled returns if a package is installed.
+// TODO: this could be optimized by making the resolve call directly
+func (bus *Conn) IsInstalled(pkg string) (bool, error) {
+	p, e := bus.IsInstalledList([]string{pkg})
+	if len(p) != 1 {
+		return false, e
+	}
+	return p[0], nil
+}
+
+// InstallPackages installs a list of packages by packageID.
+func (bus *Conn) InstallPackages(packageIDs []string, transactionFlags uint64) error {
+
+	ch := make(chan *dbus.Signal, PkBufferSize)   // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction() // emits Destroy on close
+	if err != nil {
+		return err
+	}
+
+	var signals = []string{"Package", "ErrorCode", "Finished", "Destroy"} // "ItemProgress", "Status" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("InstallPackages"), 0, transactionFlags, packageIDs)
+	if call.Err != nil {
+		return call.Err
+	}
+	timeout := -1 // disabled initially
+	finished := false
+loop:
+	for {
+		select {
+		case signal := <-ch:
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			} else if signal.Name == FmtTransactionMethod("Package") {
+				// a package was installed...
+				// only start the timer once we're here...
+				timeout = PkSignalPackageTimeout
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				finished = true
+				timeout = PkSignalDestroyTimeout // wait a bit
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				return nil // success
+			} else {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		case <-TimeAfterOrBlock(timeout):
+			if finished {
+				log.Println("PackageKit: Timeout: InstallPackages: Waiting for 'Destroy'")
+				return nil // got tired of waiting for Destroy
+			}
+			return fmt.Errorf("PackageKit: Timeout: InstallPackages: %v", strings.Join(packageIDs, ", "))
+		}
+	}
+}
+
+// RemovePackages removes a list of packages by packageID.
+func (bus *Conn) RemovePackages(packageIDs []string, transactionFlags uint64) error {
+
+	var allowDeps = true                          // TODO: configurable
+	var autoremove = false                        // unsupported on GNU/Linux
+	ch := make(chan *dbus.Signal, PkBufferSize)   // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction() // emits Destroy on close
+	if err != nil {
+		return err
+	}
+
+	var signals = []string{"Package", "ErrorCode", "Finished", "Destroy"} // "ItemProgress", "Status" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("RemovePackages"), 0, transactionFlags, packageIDs, allowDeps, autoremove)
+	if call.Err != nil {
+		return call.Err
+	}
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			} else if signal.Name == FmtTransactionMethod("Package") {
+				// a package was installed...
+				continue loop
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		}
+	}
+	return nil
+}
+
+// UpdatePackages updates a list of packages to versions that are specified.
+func (bus *Conn) UpdatePackages(packageIDs []string, transactionFlags uint64) error {
+	ch := make(chan *dbus.Signal, PkBufferSize) // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction()
+	if err != nil {
+		return err
+	}
+
+	var signals = []string{"Package", "ErrorCode", "Finished", "Destroy"} // "ItemProgress", "Status" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("UpdatePackages"), 0, transactionFlags, packageIDs)
+	if call.Err != nil {
+		return call.Err
+	}
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			} else if signal.Name == FmtTransactionMethod("Package") {
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				return fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		}
+	}
+	return nil
+}
+
+// GetFilesByPackageID gets the list of files that are contained inside a list of packageIDs.
+func (bus *Conn) GetFilesByPackageID(packageIDs []string) (files map[string][]string, err error) {
+	// NOTE: the maximum number of files in an RPM is 52116 in Fedora 23
+	// https://gist.github.com/purpleidea/b98e60dcd449e1ac3b8a
+	ch := make(chan *dbus.Signal, PkBufferSize) // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction()
+	if err != nil {
+		return
+	}
+
+	var signals = []string{"Files", "ErrorCode", "Finished", "Destroy"} // "ItemProgress", "Status" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("GetFiles"), 0, packageIDs)
+	if call.Err != nil {
+		err = call.Err
+		return
+	}
+	files = make(map[string][]string)
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				err = fmt.Errorf("PackageKit: Error: %v", signal.Body)
+				return
+
+				// one signal returned per packageID found...
+			} else if signal.Name == FmtTransactionMethod("Files") {
+				if len(signal.Body) != 2 { // bad data
+					continue loop
+				}
+				var ok bool
+				var key string
+				var fileList []string
+				if key, ok = signal.Body[0].(string); !ok {
+					continue loop
+				}
+				if fileList, ok = signal.Body[1].([]string); !ok {
+					continue loop // failed conversion
+				}
+				files[key] = fileList // build up map
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				err = fmt.Errorf("PackageKit: Error: %v", signal.Body)
+				return
+			}
+		}
+	}
+	return
+}
+
+// GetUpdates gets a list of packages that are installed and which can be updated, mod filter.
+func (bus *Conn) GetUpdates(filter uint64) ([]string, error) {
+	if PK_DEBUG {
+		log.Println("PackageKit: GetUpdates()")
+	}
+	packageIDs := []string{}
+	ch := make(chan *dbus.Signal, PkBufferSize) // we need to buffer :(
+	interfacePath, err := bus.CreateTransaction()
+	if err != nil {
+		return nil, err
+	}
+
+	var signals = []string{"Package", "ErrorCode", "Finished", "Destroy"} // "ItemProgress" ?
+	bus.matchSignal(ch, interfacePath, PkIfaceTransaction, signals)
+
+	obj := bus.GetBus().Object(PkIface, interfacePath) // pass in found transaction path
+	call := obj.Call(FmtTransactionMethod("GetUpdates"), 0, filter)
+	if call.Err != nil {
+		return nil, call.Err
+	}
+loop:
+	for {
+		// FIXME: add a timeout option to error in case signals are dropped!
+		select {
+		case signal := <-ch:
+			if signal.Path != interfacePath {
+				log.Printf("PackageKit: Woops: Signal.Path: %+v", signal.Path)
+				continue loop
+			}
+
+			if signal.Name == FmtTransactionMethod("ErrorCode") {
+				return nil, fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			} else if signal.Name == FmtTransactionMethod("Package") {
+
+				//pkg_int, ok := signal.Body[0].(int)
+				packageID, ok := signal.Body[1].(string)
+				// format is: name;version;arch;data
+				if !ok {
+					continue loop
+				}
+				//comment, ok := signal.Body[2].(string)
+				for _, p := range packageIDs { // optional?
+					if packageID == p {
+						continue loop // duplicate!
+					}
+				}
+				packageIDs = append(packageIDs, packageID)
+			} else if signal.Name == FmtTransactionMethod("Finished") {
+				// TODO: should we wait for the Destroy signal?
+				break loop
+			} else if signal.Name == FmtTransactionMethod("Destroy") {
+				// should already be broken
+				break loop
+			} else {
+				return nil, fmt.Errorf("PackageKit: Error: %v", signal.Body)
+			}
+		}
+	}
+	return packageIDs, nil
+}
+
+// PackagesToPackageIDs is a helper function that *might* be generally useful
+// outside mgmt. The packageMap input has the package names as keys and
+// requested states as values. These states can be: installed, uninstalled,
+// newest or a requested version str.
+func (bus *Conn) PackagesToPackageIDs(packageMap map[string]string, filter uint64) (map[string]*PkPackageIDActionData, error) {
+	count := 0
+	packages := make([]string, len(packageMap))
+	for k := range packageMap { // lol, golang has no hash.keys() function!
+		packages[count] = k
+		count++
+	}
+
+	if !(filter&PK_FILTER_ENUM_ARCH == PK_FILTER_ENUM_ARCH) {
+		filter += PK_FILTER_ENUM_ARCH // always search in our arch
+	}
+
+	if PK_DEBUG {
+		log.Printf("PackageKit: PackagesToPackageIDs(): %v", strings.Join(packages, ", "))
+	}
+	resolved, e := bus.ResolvePackages(packages, filter)
+	if e != nil {
+		return nil, fmt.Errorf("Resolve error: %v", e)
+	}
+
+	found := make([]bool, count) // default false
+	installed := make([]bool, count)
+	version := make([]string, count)
+	usePackageID := make([]string, count)
+	newest := make([]bool, count) // default true
+	for i := range newest {
+		newest[i] = true // assume, for now
+	}
+	var index int
+
+	for _, packageID := range resolved {
+		index = -1
+		//log.Printf("* %v", packageID)
+		// format is: name;version;arch;data
+		s := strings.Split(packageID, ";")
+		//if len(s) != 4 { continue } // this would be a bug!
+		pkg, ver, arch, data := s[0], s[1], s[2], s[3]
+		// we might need to allow some of this, eg: i386 .deb on amd64
+		if !IsMyArch(arch) {
+			continue
+		}
+
+		for i := range packages { // find pkg if it exists
+			if pkg == packages[i] {
+				index = i
+			}
+		}
+		if index == -1 { // can't find what we're looking for
+			continue
+		}
+		state := packageMap[pkg] // lookup the requested state/version
+		if state == "" {
+			return nil, fmt.Errorf("Empty package state for %v", pkg)
+		}
+		found[index] = true
+		stateIsVersion := (state != "installed" && state != "uninstalled" && state != "newest") // must be a ver. string
+
+		if stateIsVersion {
+			if state == ver && ver != "" { // we match what we want...
+				usePackageID[index] = packageID
+			}
+		}
+
+		if FlagInData("installed", data) {
+			installed[index] = true
+			version[index] = ver
+			// state of "uninstalled" matched during CheckApply, and
+			// states of "installed" and "newest" for fileList
+			if !stateIsVersion {
+				usePackageID[index] = packageID // save for later
+			}
+		} else { // not installed...
+			if !stateIsVersion {
+				// if there is more than one result, eg: there
+				// is the old and newest version of a package,
+				// then this section can run more than once...
+				// in that case, don't worry, we'll choose the
+				// right value in the "updates" section below!
+				usePackageID[index] = packageID
+			}
+		}
+	}
+
+	// we can't determine which packages are "newest", without searching
+	// for each one individually, so instead we check if any updates need
+	// to be done, and if so, anything that needs updating isn't newest!
+	// if something isn't installed, we can't verify it with this method
+	// FIXME: https://github.com/hughsie/PackageKit/issues/116
+	updates, e := bus.GetUpdates(filter)
+	if e != nil {
+		return nil, fmt.Errorf("Updates error: %v", e)
+	}
+	for _, packageID := range updates {
+		//log.Printf("* %v", packageID)
+		// format is: name;version;arch;data
+		s := strings.Split(packageID, ";")
+		//if len(s) != 4 { continue } // this would be a bug!
+		pkg, _, _, _ := s[0], s[1], s[2], s[3]
+		for index := range packages { // find pkg if it exists
+			if pkg == packages[index] {
+				state := packageMap[pkg] // lookup
+				newest[index] = false
+				if state == "installed" || state == "newest" {
+					// fix up in case above wasn't correct!
+					usePackageID[index] = packageID
+				}
+				break
+			}
+		}
+	}
+
+	// skip if the "newest" filter was used, otherwise we might need fixing
+	// this check is for packages that need to verify their "newest" status
+	// we need to know this so we can install the correct newest packageID!
+	recursion := make(map[string]*PkPackageIDActionData)
+	if !(filter&PK_FILTER_ENUM_NEWEST == PK_FILTER_ENUM_NEWEST) {
+		checkPackages := []string{}
+		filteredPackageMap := make(map[string]string)
+		for index, pkg := range packages {
+			state := packageMap[pkg]               // lookup the requested state/version
+			if !found[index] || installed[index] { // skip these, they're okay
+				continue
+			}
+			if !(state == "newest" || state == "installed") {
+				continue
+			}
+
+			checkPackages = append(checkPackages, pkg)
+			filteredPackageMap[pkg] = packageMap[pkg] // check me!
+		}
+
+		// we _could_ do a second resolve and then parse like this...
+		//resolved, e := bus.ResolvePackages(..., filter+PK_FILTER_ENUM_NEWEST)
+		// but that's basically what recursion here could do too!
+		if len(checkPackages) > 0 {
+			if PK_DEBUG {
+				log.Printf("PackageKit: PackagesToPackageIDs(): Recurse: %v", strings.Join(checkPackages, ", "))
+			}
+			recursion, e = bus.PackagesToPackageIDs(filteredPackageMap, filter+PK_FILTER_ENUM_NEWEST)
+			if e != nil {
+				return nil, fmt.Errorf("Recursion error: %v", e)
+			}
+		}
+	}
+
+	// fix up and build result format
+	result := make(map[string]*PkPackageIDActionData)
+	for index, pkg := range packages {
+
+		if !found[index] || !installed[index] {
+			newest[index] = false // make the results more logical!
+		}
+
+		// prefer recursion results if present
+		if lookup, ok := recursion[pkg]; ok {
+			result[pkg] = lookup
+		} else {
+			result[pkg] = &PkPackageIDActionData{
+				Found:     found[index],
+				Installed: installed[index],
+				Version:   version[index],
+				PackageID: usePackageID[index],
+				Newest:    newest[index],
+			}
+		}
+	}
+
+	return result, nil
+}
+
+// FilterPackageIDs returns a list of packageIDs which match the set of package names in packages.
+func FilterPackageIDs(m map[string]*PkPackageIDActionData, packages []string) ([]string, error) {
+	result := []string{}
+	for _, k := range packages {
+		obj, ok := m[k] // lookup single package
+		// package doesn't exist, this is an error!
+		if !ok || !obj.Found || obj.PackageID == "" {
+			return nil, fmt.Errorf("Can't find package named '%s'.", k)
+		}
+		result = append(result, obj.PackageID)
+	}
+	return result, nil
+}
+
+// FilterState returns a map of whether each package queried matches the particular state.
+func FilterState(m map[string]*PkPackageIDActionData, packages []string, state string) (result map[string]bool, err error) {
+	result = make(map[string]bool)
+	pkgs := []string{} // bad pkgs that don't have a bool state
+	for _, k := range packages {
+		obj, ok := m[k] // lookup single package
+		// package doesn't exist, this is an error!
+		if !ok || !obj.Found {
+			return nil, fmt.Errorf("Can't find package named '%s'.", k)
+		}
+		var b bool
+		if state == "installed" {
+			b = obj.Installed
+		} else if state == "uninstalled" {
+			b = !obj.Installed
+		} else if state == "newest" {
+			b = obj.Newest
+		} else {
+			// we can't filter "version" state in this function
+			pkgs = append(pkgs, k)
+			continue
+		}
+		result[k] = b // save
+	}
+	if len(pkgs) > 0 {
+		err = fmt.Errorf("Can't filter non-boolean state on: %v!", strings.Join(pkgs, ","))
+	}
+	return result, err
+}
+
+// FilterPackageState returns all packages that are in package and match the specific state.
+func FilterPackageState(m map[string]*PkPackageIDActionData, packages []string, state string) (result []string, err error) {
+	result = []string{}
+	for _, k := range packages {
+		obj, ok := m[k] // lookup single package
+		// package doesn't exist, this is an error!
+		if !ok || !obj.Found {
+			return nil, fmt.Errorf("Can't find package named '%s'.", k)
+		}
+		b := false
+		if state == "installed" && obj.Installed {
+			b = true
+		} else if state == "uninstalled" && !obj.Installed {
+			b = true
+		} else if state == "newest" && obj.Newest {
+			b = true
+		} else if state == obj.Version {
+			b = true
+		}
+		if b {
+			result = append(result, k)
+		}
+	}
+	return result, err
+}
+
+// FlagInData asks whether a flag exists inside the data portion of a packageID field?
+func FlagInData(flag, data string) bool {
+	flags := strings.Split(data, ":")
+	for _, f := range flags {
+		if f == flag {
+			return true
+		}
+	}
+	return false
+}
+
+// FmtTransactionMethod builds the transaction method string properly.
+func FmtTransactionMethod(method string) string {
+	return fmt.Sprintf("%s.%s", PkIfaceTransaction, method)
+}
+
+// IsMyArch determines if a PackageKit architecture matches the current os arch.
+func IsMyArch(arch string) bool {
+	goarch, ok := PkArchMap[arch]
+	if !ok {
+		// if you get this error, please update the PkArchMap const
+		log.Fatalf("PackageKit: Arch '%v', not found!", arch)
+	}
+	if goarch == "ANY" { // special value that corresponds to noarch
+		return true
+	}
+	return goarch == runtime.GOARCH
+}

pgraph.go

@@ -25,6 +25,7 @@ import (
 	"log"
 	"os"
 	"os/exec"
+	"sort"
 	"strconv"
 	"sync"
 	"syscall"
@@ -35,13 +36,14 @@ import (
 type graphState int
 
 const (
-	graphNil graphState = iota
+	graphStateNil graphState = iota
-	graphStarting
+	graphStateStarting
-	graphStarted
+	graphStateStarted
-	graphPausing
+	graphStatePausing
-	graphPaused
+	graphStatePaused
 )
 
+// Graph is the graph structure in this library.
 // The graph abstract data type (ADT) is defined as follows:
 // * the directed graph arrows point from left to right ( -> )
 // * the arrows point away from their dependencies (eg: arrows mean "before")
@@ -52,82 +54,93 @@ type Graph struct {
 	Adjacency map[*Vertex]map[*Vertex]*Edge // *Vertex -> *Vertex (edge)
 	state     graphState
 	mutex     sync.Mutex // used when modifying graph State variable
-	//Directed  bool
 }
 
+// Vertex is the primary vertex struct in this library.
 type Vertex struct {
-	graph *Graph            // store a pointer to the graph it's on
+	Res             // anonymous field
-	Type                    // anonymous field
+	timestamp int64 // last updated timestamp ?
-	data  map[string]string // XXX: currently unused i think, remove?
 }
 
+// Edge is the primary edge struct in this library.
 type Edge struct {
 	Name string
 }
 
+// NewGraph builds a new graph.
 func NewGraph(name string) *Graph {
 	return &Graph{
 		Name:      name,
 		Adjacency: make(map[*Vertex]map[*Vertex]*Edge),
-		state:     graphNil,
+		state:     graphStateNil,
 	}
 }
 
-func NewVertex(t Type) *Vertex {
+// NewVertex returns a new graph vertex struct with a contained resource.
+func NewVertex(r Res) *Vertex {
 	return &Vertex{
-		Type: t,
+		Res: r,
-		data: make(map[string]string),
 	}
 }
 
+// NewEdge returns a new graph edge struct.
 func NewEdge(name string) *Edge {
 	return &Edge{
 		Name: name,
 	}
 }
 
-// returns the name of the graph
+// Copy makes a copy of the graph struct
+func (g *Graph) Copy() *Graph {
+	newGraph := &Graph{
+		Name:      g.Name,
+		Adjacency: make(map[*Vertex]map[*Vertex]*Edge, len(g.Adjacency)),
+		state:     g.state,
+	}
+	for k, v := range g.Adjacency {
+		newGraph.Adjacency[k] = v // copy
+	}
+	return newGraph
+}
+
+// GetName returns the name of the graph.
 func (g *Graph) GetName() string {
 	return g.Name
 }
 
-// set name of the graph
+// SetName sets the name of the graph.
 func (g *Graph) SetName(name string) {
 	g.Name = name
 }
 
-func (g *Graph) GetState() graphState {
+// getState returns the state of the graph. This state is used for optimizing
+// certain algorithms by knowing what part of processing the graph is currently
+// undergoing.
+func (g *Graph) getState() graphState {
 	//g.mutex.Lock()
 	//defer g.mutex.Unlock()
 	return g.state
 }
 
-// set graph state and return previous state
+// setState sets the graph state and returns the previous state.
-func (g *Graph) SetState(state graphState) graphState {
+func (g *Graph) setState(state graphState) graphState {
 	g.mutex.Lock()
 	defer g.mutex.Unlock()
-	prev := g.GetState()
+	prev := g.getState()
 	g.state = state
 	return prev
 }
 
-// store a pointer in the type to it's parent vertex
+// AddVertex uses variadic input to add all listed vertices to the graph
-func (g *Graph) SetVertex() {
+func (g *Graph) AddVertex(xv ...*Vertex) {
-	for v := range g.GetVerticesChan() {
+	for _, v := range xv {
-		v.Type.SetVertex(v)
+		if _, exists := g.Adjacency[v]; !exists {
-	}
+			g.Adjacency[v] = make(map[*Vertex]*Edge)
-}
+		}
-
-// add a new vertex to the graph
-func (g *Graph) AddVertex(v *Vertex) {
-	if _, exists := g.Adjacency[v]; !exists {
-		g.Adjacency[v] = make(map[*Vertex]*Edge)
-
-		// store a pointer to the graph it's on for convenience and readability
-		v.graph = g
 	}
 }
 
+// DeleteVertex deletes a particular vertex from the graph.
 func (g *Graph) DeleteVertex(v *Vertex) {
 	delete(g.Adjacency, v)
 	for k := range g.Adjacency {
@@ -135,58 +148,40 @@ func (g *Graph) DeleteVertex(v *Vertex) {
 	}
 }
 
-// adds a directed edge to the graph from v1 to v2
+// AddEdge adds a directed edge to the graph from v1 to v2.
 func (g *Graph) AddEdge(v1, v2 *Vertex, e *Edge) {
 	// NOTE: this doesn't allow more than one edge between two vertexes...
-	// TODO: is this a problem?
+	g.AddVertex(v1, v2) // supports adding N vertices now
-	g.AddVertex(v1)
+	// TODO: check if an edge exists to avoid overwriting it!
-	g.AddVertex(v2)
+	// NOTE: VertexMerge() depends on overwriting it at the moment...
 	g.Adjacency[v1][v2] = e
 }
 
-// XXX: does it make sense to return a channel here?
+// GetVertexMatch searches for an equivalent resource in the graph and returns
-// GetVertex finds the vertex in the graph with a particular search name
+// the vertex it is found in, or nil if not found.
-func (g *Graph) GetVertex(name string) chan *Vertex {
+func (g *Graph) GetVertexMatch(obj Res) *Vertex {
-	ch := make(chan *Vertex, 1)
-	go func(name string) {
-		for k := range g.Adjacency {
-			if k.GetName() == name {
-				ch <- k
-				break
-			}
-		}
-		close(ch)
-	}(name)
-	return ch
-}
-
-func (g *Graph) GetVertexMatch(obj Type) *Vertex {
 	for k := range g.Adjacency {
-		if k.Compare(obj) { // XXX test
+		if k.Res.Compare(obj) {
 			return k
 		}
 	}
 	return nil
 }
 
+// HasVertex returns if the input vertex exists in the graph.
 func (g *Graph) HasVertex(v *Vertex) bool {
 	if _, exists := g.Adjacency[v]; exists {
 		return true
 	}
-	//for k := range g.Adjacency {
-	//	if k == v {
-	//		return true
-	//	}
-	//}
 	return false
 }
 
-// number of vertices in the graph
+// NumVertices returns the number of vertices in the graph.
 func (g *Graph) NumVertices() int {
 	return len(g.Adjacency)
 }
 
-// number of edges in the graph
+// NumEdges returns the number of edges in the graph.
 func (g *Graph) NumEdges() int {
 	count := 0
 	for k := range g.Adjacency {
@@ -195,7 +190,8 @@ func (g *Graph) NumEdges() int {
 	return count
 }
 
-// get an array (slice) of all vertices in the graph
+// GetVertices returns a randomly sorted slice of all vertices in the graph
+// The order is random, because the map implementation is intentionally so!
 func (g *Graph) GetVertices() []*Vertex {
 	var vertices []*Vertex
 	for k := range g.Adjacency {
@@ -204,7 +200,7 @@ func (g *Graph) GetVertices() []*Vertex {
 	return vertices
 }
 
-// returns a channel of all vertices in the graph
+// GetVerticesChan returns a channel of all vertices in the graph.
 func (g *Graph) GetVerticesChan() chan *Vertex {
 	ch := make(chan *Vertex)
 	go func(ch chan *Vertex) {
@@ -216,12 +212,35 @@ func (g *Graph) GetVerticesChan() chan *Vertex {
 	return ch
 }
 
-// make the graph pretty print
+// VertexSlice is a linear list of vertices. It can be sorted.
+type VertexSlice []*Vertex
+
+func (vs VertexSlice) Len() int           { return len(vs) }
+func (vs VertexSlice) Swap(i, j int)      { vs[i], vs[j] = vs[j], vs[i] }
+func (vs VertexSlice) Less(i, j int) bool { return vs[i].String() < vs[j].String() }
+
+// GetVerticesSorted returns a sorted slice of all vertices in the graph
+// The order is sorted by String() to avoid the non-determinism in the map type
+func (g *Graph) GetVerticesSorted() []*Vertex {
+	var vertices []*Vertex
+	for k := range g.Adjacency {
+		vertices = append(vertices, k)
+	}
+	sort.Sort(VertexSlice(vertices)) // add determinism
+	return vertices
+}
+
+// String makes the graph pretty print.
 func (g *Graph) String() string {
 	return fmt.Sprintf("Vertices(%d), Edges(%d)", g.NumVertices(), g.NumEdges())
 }
 
-// output the graph in graphviz format
+// String returns the canonical form for a vertex
+func (v *Vertex) String() string {
+	return fmt.Sprintf("%s[%s]", v.Res.Kind(), v.Res.GetName())
+}
+
+// Graphviz outputs the graph in graphviz format.
 // https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29
 func (g *Graph) Graphviz() (out string) {
 	//digraph g {
@@ -241,7 +260,7 @@ func (g *Graph) Graphviz() (out string) {
 	//out += "\tnode [shape=box];\n"
 	str := ""
 	for i := range g.Adjacency { // reverse paths
-		out += fmt.Sprintf("\t%v [label=\"%v[%v]\"];\n", i.GetName(), i.GetType(), i.GetName())
+		out += fmt.Sprintf("\t%v [label=\"%v[%v]\"];\n", i.GetName(), i.Kind(), i.GetName())
 		for j := range g.Adjacency[i] {
 			k := g.Adjacency[i][j]
 			// use str for clearer output ordering
@@ -253,7 +272,8 @@ func (g *Graph) Graphviz() (out string) {
 	return
 }
 
-// write out the graphviz data and run the correct graphviz filter command
+// ExecGraphviz writes out the graphviz data and runs the correct graphviz
+// filter command.
 func (g *Graph) ExecGraphviz(program, filename string) error {
 
 	switch program {
@@ -303,18 +323,8 @@ func (g *Graph) ExecGraphviz(program, filename string) error {
 	return nil
 }
 
-// google/golang hackers apparently do not think contains should be a built-in!
+// IncomingGraphEdges returns an array (slice) of all directed vertices to
-func Contains(s []*Vertex, element *Vertex) bool {
+// vertex v (??? -> v). OKTimestamp should probably use this.
-	for _, v := range s {
-		if element == v {
-			return true
-		}
-	}
-	return false
-}
-
-// return an array (slice) of all directed vertices to vertex v (??? -> v)
-// ostimestamp should use this
 func (g *Graph) IncomingGraphEdges(v *Vertex) []*Vertex {
 	// TODO: we might be able to implement this differently by reversing
 	// the Adjacency graph and then looping through it again...
@@ -329,8 +339,8 @@ func (g *Graph) IncomingGraphEdges(v *Vertex) []*Vertex {
 	return s
 }
 
-// return an array (slice) of all vertices that vertex v points to (v -> ???)
+// OutgoingGraphEdges returns an array (slice) of all vertices that vertex v
-// poke should use this
+// points to (v -> ???). Poke should probably use this.
 func (g *Graph) OutgoingGraphEdges(v *Vertex) []*Vertex {
 	var s []*Vertex
 	for k := range g.Adjacency[v] { // forward paths
@@ -339,7 +349,8 @@ func (g *Graph) OutgoingGraphEdges(v *Vertex) []*Vertex {
 	return s
 }
 
-// return an array (slice) of all vertices that connect to vertex v
+// GraphEdges returns an array (slice) of all vertices that connect to vertex v.
+// This is the union of IncomingGraphEdges and OutgoingGraphEdges.
 func (g *Graph) GraphEdges(v *Vertex) []*Vertex {
 	var s []*Vertex
 	s = append(s, g.IncomingGraphEdges(v)...)
@@ -347,6 +358,7 @@ func (g *Graph) GraphEdges(v *Vertex) []*Vertex {
 	return s
 }
 
+// DFS returns a depth first search for the graph, starting at the input vertex.
 func (g *Graph) DFS(start *Vertex) []*Vertex {
 	var d []*Vertex // discovered
 	var s []*Vertex // stack
@@ -356,10 +368,9 @@ func (g *Graph) DFS(start *Vertex) []*Vertex {
 	v := start
 	s = append(s, v)
 	for len(s) > 0 {
-
 		v, s = s[len(s)-1], s[:len(s)-1] // s.pop()
 
-		if !Contains(d, v) { // if not discovered
+		if !VertexContains(v, d) { // if not discovered
 			d = append(d, v) // label as discovered
 
 			for _, w := range g.GraphEdges(v) {
@@ -370,25 +381,22 @@ func (g *Graph) DFS(start *Vertex) []*Vertex {
 	return d
 }
 
-// build a new graph containing only vertices from the list...
+// FilterGraph builds a new graph containing only vertices from the list.
 func (g *Graph) FilterGraph(name string, vertices []*Vertex) *Graph {
 	newgraph := NewGraph(name)
-
 	for k1, x := range g.Adjacency {
 		for k2, e := range x {
-
 			//log.Printf("Filter: %v -> %v # %v", k1.Name, k2.Name, e.Name)
-			if Contains(vertices, k1) || Contains(vertices, k2) {
+			if VertexContains(k1, vertices) || VertexContains(k2, vertices) {
 				newgraph.AddEdge(k1, k2, e)
 			}
 		}
 	}
-
 	return newgraph
 }
 
-// return a channel containing the N disconnected graphs in our main graph
+// GetDisconnectedGraphs returns a channel containing the N disconnected graphs
-// we can then process each of these in parallel
+// in our main graph. We can then process each of these in parallel.
 func (g *Graph) GetDisconnectedGraphs() chan *Graph {
 	ch := make(chan *Graph)
 	go func() {
@@ -399,7 +407,7 @@ func (g *Graph) GetDisconnectedGraphs() chan *Graph {
 
 			// get an undiscovered vertex to start from
 			for _, s := range g.GetVertices() {
-				if !Contains(d, s) {
+				if !VertexContains(s, d) {
 					start = s
 				}
 			}
@@ -423,8 +431,7 @@ func (g *Graph) GetDisconnectedGraphs() chan *Graph {
 	return ch
 }
 
-// return the indegree for the graph, IOW the count of vertices that point to me
+// InDegree returns the count of vertices that point to me in one big lookup map.
-// NOTE: this returns the values for all vertices in one big lookup table
 func (g *Graph) InDegree() map[*Vertex]int {
 	result := make(map[*Vertex]int)
 	for k := range g.Adjacency {
@@ -439,25 +446,23 @@ func (g *Graph) InDegree() map[*Vertex]int {
 	return result
 }
 
-// return the outdegree for the graph, IOW the count of vertices that point away
+// OutDegree returns the count of vertices that point away in one big lookup map.
-// NOTE: this returns the values for all vertices in one big lookup table
 func (g *Graph) OutDegree() map[*Vertex]int {
 	result := make(map[*Vertex]int)
 
 	for k := range g.Adjacency {
 		result[k] = 0 // initialize
-		for _ = range g.Adjacency[k] {
+		for range g.Adjacency[k] {
 			result[k]++
 		}
 	}
 	return result
 }
 
-// returns a topological sort for the graph
+// TopologicalSort returns the sort of graph vertices in that order.
 // based on descriptions and code from wikipedia and rosetta code
 // TODO: add memoization, and cache invalidation to speed this up :)
 func (g *Graph) TopologicalSort() (result []*Vertex, ok bool) { // kahn's algorithm
-
 	var L []*Vertex                    // empty list that will contain the sorted elements
 	var S []*Vertex                    // set of all nodes with no incoming edges
 	remaining := make(map[*Vertex]int) // amount of edges remaining
@@ -503,60 +508,305 @@ func (g *Graph) TopologicalSort() (result []*Vertex, ok bool) { // kahn's algori
 	return L, true
 }
 
-func (v *Vertex) Value(key string) (string, bool) {
+// Reachability finds the shortest path in a DAG from a to b, and returns the
-	if value, exists := v.data[key]; exists {
+// slice of vertices that matched this particular path including both a and b.
-		return value, true
+// It returns nil if a or b is nil, and returns empty list if no path is found.
+// Since there could be more than one possible result for this operation, we
+// arbitrarily choose one of the shortest possible. As a result, this should
+// actually return a tree if we cared about correctness.
+// This operates by a recursive algorithm; a more efficient version is likely.
+// If you don't give this function a DAG, you might cause infinite recursion!
+func (g *Graph) Reachability(a, b *Vertex) []*Vertex {
+	if a == nil || b == nil {
+		return nil
 	}
-	return "", false
+	vertices := g.OutgoingGraphEdges(a) // what points away from a ?
+	if len(vertices) == 0 {
+		return []*Vertex{} // nope
+	}
+	if VertexContains(b, vertices) {
+		return []*Vertex{a, b} // found
+	}
+	// TODO: parallelize this with go routines?
+	var collected = make([][]*Vertex, len(vertices))
+	pick := -1
+	for i, v := range vertices {
+		collected[i] = g.Reachability(v, b) // find b by recursion
+		if l := len(collected[i]); l > 0 {
+			// pick shortest path
+			// TODO: technically i should return a tree
+			if pick < 0 || l < len(collected[pick]) {
+				pick = i
+			}
+		}
+	}
+	if pick < 0 {
+		return []*Vertex{} // nope
+	}
+	result := []*Vertex{a} // tack on a
+	result = append(result, collected[pick]...)
+	return result
 }
 
-func (v *Vertex) SetValue(key, value string) bool {
+// VertexMerge merges v2 into v1 by reattaching the edges where appropriate,
-	v.data[key] = value
+// and then by deleting v2 from the graph. Since more than one edge between two
+// vertices is not allowed, duplicate edges are merged as well. an edge merge
+// function can be provided if you'd like to control how you merge the edges!
+func (g *Graph) VertexMerge(v1, v2 *Vertex, vertexMergeFn func(*Vertex, *Vertex) (*Vertex, error), edgeMergeFn func(*Edge, *Edge) *Edge) error {
+	// methodology
+	// 1) edges between v1 and v2 are removed
+	//Loop:
+	for k1 := range g.Adjacency {
+		for k2 := range g.Adjacency[k1] {
+			// v1 -> v2 || v2 -> v1
+			if (k1 == v1 && k2 == v2) || (k1 == v2 && k2 == v1) {
+				delete(g.Adjacency[k1], k2) // delete map & edge
+				// NOTE: if we assume this is a DAG, then we can
+				// assume only v1 -> v2 OR v2 -> v1 exists, and
+				// we can break out of these loops immediately!
+				//break Loop
+				break
+			}
+		}
+	}
+
+	// 2) edges that point towards v2 from X now point to v1 from X (no dupes)
+	for _, x := range g.IncomingGraphEdges(v2) { // all to vertex v (??? -> v)
+		e := g.Adjacency[x][v2] // previous edge
+		r := g.Reachability(x, v1)
+		// merge e with ex := g.Adjacency[x][v1] if it exists!
+		if ex, exists := g.Adjacency[x][v1]; exists && edgeMergeFn != nil && len(r) == 0 {
+			e = edgeMergeFn(e, ex)
+		}
+		if len(r) == 0 { // if not reachable, add it
+			g.AddEdge(x, v1, e) // overwrite edge
+		} else if edgeMergeFn != nil { // reachable, merge e through...
+			prev := x // initial condition
+			for i, next := range r {
+				if i == 0 {
+					// next == prev, therefore skip
+					continue
+				}
+				// this edge is from: prev, to: next
+				ex, _ := g.Adjacency[prev][next] // get
+				ex = edgeMergeFn(ex, e)
+				g.Adjacency[prev][next] = ex // set
+				prev = next
+			}
+		}
+		delete(g.Adjacency[x], v2) // delete old edge
+	}
+
+	// 3) edges that point from v2 to X now point from v1 to X (no dupes)
+	for _, x := range g.OutgoingGraphEdges(v2) { // all from vertex v (v -> ???)
+		e := g.Adjacency[v2][x] // previous edge
+		r := g.Reachability(v1, x)
+		// merge e with ex := g.Adjacency[v1][x] if it exists!
+		if ex, exists := g.Adjacency[v1][x]; exists && edgeMergeFn != nil && len(r) == 0 {
+			e = edgeMergeFn(e, ex)
+		}
+		if len(r) == 0 {
+			g.AddEdge(v1, x, e) // overwrite edge
+		} else if edgeMergeFn != nil { // reachable, merge e through...
+			prev := v1 // initial condition
+			for i, next := range r {
+				if i == 0 {
+					// next == prev, therefore skip
+					continue
+				}
+				// this edge is from: prev, to: next
+				ex, _ := g.Adjacency[prev][next]
+				ex = edgeMergeFn(ex, e)
+				g.Adjacency[prev][next] = ex
+				prev = next
+			}
+		}
+		delete(g.Adjacency[v2], x)
+	}
+
+	// 4) merge and then remove the (now merged/grouped) vertex
+	if vertexMergeFn != nil { // run vertex merge function
+		if v, err := vertexMergeFn(v1, v2); err != nil {
+			return err
+		} else if v != nil { // replace v1 with the "merged" version...
+			v1 = v // XXX: will this replace v1 the way we want?
+		}
+	}
+	g.DeleteVertex(v2) // remove grouped vertex
+
+	// 5) creation of a cyclic graph should throw an error
+	if _, dag := g.TopologicalSort(); !dag { // am i a dag or not?
+		return fmt.Errorf("Graph is not a dag!")
+	}
+	return nil // success
+}
+
+// GetTimestamp returns the timestamp of a vertex
+func (v *Vertex) GetTimestamp() int64 {
+	return v.timestamp
+}
+
+// UpdateTimestamp updates the timestamp on a vertex and returns the new value
+func (v *Vertex) UpdateTimestamp() int64 {
+	v.timestamp = time.Now().UnixNano() // update
+	return v.timestamp
+}
+
+// OKTimestamp returns true if this element can run right now?
+func (g *Graph) OKTimestamp(v *Vertex) bool {
+	// these are all the vertices pointing TO v, eg: ??? -> v
+	for _, n := range g.IncomingGraphEdges(v) {
+		// if the vertex has a greater timestamp than any pre-req (n)
+		// then we can't run right now...
+		// if they're equal (eg: on init of 0) then we also can't run
+		// b/c we should let our pre-req's go first...
+		x, y := v.GetTimestamp(), n.GetTimestamp()
+		if DEBUG {
+			log.Printf("%v[%v]: OKTimestamp: (%v) >= %v[%v](%v): !%v", v.Kind(), v.GetName(), x, n.Kind(), n.GetName(), y, x >= y)
+		}
+		if x >= y {
+			return false
+		}
+	}
 	return true
 }
 
-func (g *Graph) GetVerticesKeyValue(key, value string) chan *Vertex {
+// Poke notifies nodes after me in the dependency graph that they need refreshing...
-	ch := make(chan *Vertex)
+// NOTE: this assumes that this can never fail or need to be rescheduled
-	go func() {
+func (g *Graph) Poke(v *Vertex, activity bool) {
-		for vertex := range g.GetVerticesChan() {
+	// these are all the vertices pointing AWAY FROM v, eg: v -> ???
-			if v, exists := vertex.Value(key); exists && v == value {
+	for _, n := range g.OutgoingGraphEdges(v) {
-				ch <- vertex
+		// XXX: if we're in state event and haven't been cancelled by
+		// apply, then we can cancel a poke to a child, right? XXX
+		// XXX: if n.Res.getState() != resStateEvent { // is this correct?
+		if true { // XXX
+			if DEBUG {
+				log.Printf("%v[%v]: Poke: %v[%v]", v.Kind(), v.GetName(), n.Kind(), n.GetName())
+			}
+			n.SendEvent(eventPoke, false, activity) // XXX: can this be switched to sync?
+		} else {
+			if DEBUG {
+				log.Printf("%v[%v]: Poke: %v[%v]: Skipped!", v.Kind(), v.GetName(), n.Kind(), n.GetName())
 			}
 		}
-		close(ch)
-	}()
-	return ch
-}
-
-// return a pointer to the graph a vertex is on
-func (v *Vertex) GetGraph() *Graph {
-	return v.graph
-}
-
-func HeisenbergCount(ch chan *Vertex) int {
-	c := 0
-	for x := range ch {
-		_ = x
-		c++
 	}
-	return c
 }
 
-// main kick to start the graph
+// BackPoke pokes the pre-requisites that are stale and need to run before I can run.
+func (g *Graph) BackPoke(v *Vertex) {
+	// these are all the vertices pointing TO v, eg: ??? -> v
+	for _, n := range g.IncomingGraphEdges(v) {
+		x, y, s := v.GetTimestamp(), n.GetTimestamp(), n.Res.GetState()
+		// if the parent timestamp needs poking AND it's not in state
+		// resStateEvent, then poke it. If the parent is in resStateEvent it
+		// means that an event is pending, so we'll be expecting a poke
+		// back soon, so we can safely discard the extra parent poke...
+		// TODO: implement a stateLT (less than) to tell if something
+		// happens earlier in the state cycle and that doesn't wrap nil
+		if x >= y && (s != resStateEvent && s != resStateCheckApply) {
+			if DEBUG {
+				log.Printf("%v[%v]: BackPoke: %v[%v]", v.Kind(), v.GetName(), n.Kind(), n.GetName())
+			}
+			n.SendEvent(eventBackPoke, false, false) // XXX: can this be switched to sync?
+		} else {
+			if DEBUG {
+				log.Printf("%v[%v]: BackPoke: %v[%v]: Skipped!", v.Kind(), v.GetName(), n.Kind(), n.GetName())
+			}
+		}
+	}
+}
+
+// Process is the primary function to execute for a particular vertex in the graph.
+// XXX: rename this function
+func (g *Graph) Process(v *Vertex) {
+	obj := v.Res
+	if DEBUG {
+		log.Printf("%v[%v]: Process()", obj.Kind(), obj.GetName())
+	}
+	obj.SetState(resStateEvent)
+	var ok = true
+	var apply = false // did we run an apply?
+	// is it okay to run dependency wise right now?
+	// if not, that's okay because when the dependency runs, it will poke
+	// us back and we will run if needed then!
+	if g.OKTimestamp(v) {
+		if DEBUG {
+			log.Printf("%v[%v]: OKTimestamp(%v)", obj.Kind(), obj.GetName(), v.GetTimestamp())
+		}
+
+		obj.SetState(resStateCheckApply)
+		// if this fails, don't UpdateTimestamp()
+		checkok, err := obj.CheckApply(!obj.Meta().Noop)
+		if checkok && err != nil { // should never return this way
+			log.Fatalf("%v[%v]: CheckApply(): %t, %+v", obj.Kind(), obj.GetName(), checkok, err)
+		}
+		if DEBUG {
+			log.Printf("%v[%v]: CheckApply(): %t, %v", obj.Kind(), obj.GetName(), checkok, err)
+		}
+
+		if !checkok { // if state *was* not ok, we had to have apply'ed
+			if err != nil { // error during check or apply
+				ok = false
+			} else {
+				apply = true
+			}
+		}
+
+		// when noop is true we always want to update timestamp
+		if obj.Meta().Noop && err == nil {
+			ok = true
+		}
+
+		if ok {
+			// update this timestamp *before* we poke or the poked
+			// nodes might fail due to having a too old timestamp!
+			v.UpdateTimestamp()          // this was touched...
+			obj.SetState(resStatePoking) // can't cancel parent poke
+			g.Poke(v, apply)
+		}
+		// poke at our pre-req's instead since they need to refresh/run...
+	} else {
+		// only poke at the pre-req's that need to run
+		go g.BackPoke(v)
+	}
+}
+
+// Start is a main kick to start the graph. It goes through in reverse topological
+// sort order so that events can't hit un-started vertices.
 func (g *Graph) Start(wg *sync.WaitGroup, first bool) { // start or continue
+	log.Printf("State: %v -> %v", g.setState(graphStateStarting), g.getState())
+	defer log.Printf("State: %v -> %v", g.setState(graphStateStarted), g.getState())
 	t, _ := g.TopologicalSort()
 	// TODO: only calculate indegree if `first` is true to save resources
 	indegree := g.InDegree() // compute all of the indegree's
 	for _, v := range Reverse(t) {
 
-		if !v.Type.IsWatching() { // if Watch() is not running...
+		if !v.Res.IsWatching() { // if Watch() is not running...
 			wg.Add(1)
 			// must pass in value to avoid races...
 			// see: https://ttboj.wordpress.com/2015/07/27/golang-parallelism-issues-causing-too-many-open-files-error/
 			go func(vv *Vertex) {
 				defer wg.Done()
-				vv.Type.Watch()
+				// listen for chan events from Watch() and run
-				log.Printf("%v[%v]: Exited", vv.GetType(), vv.GetName())
+				// the Process() function when they're received
+				// this avoids us having to pass the data into
+				// the Watch() function about which graph it is
+				// running on, which isolates things nicely...
+				chanProcess := make(chan Event)
+				go func() {
+					for event := range chanProcess {
+						// this has to be synchronous,
+						// because otherwise the Res
+						// event loop will keep running
+						// and change state, causing the
+						// converged timeout to fire!
+						g.Process(vv)
+						event.ACK() // sync
+					}
+				}()
+				vv.Res.Watch(chanProcess) // i block until i end
+				close(chanProcess)
+				log.Printf("%v[%v]: Exited", vv.Kind(), vv.GetName())
 			}(v)
 		}
 
@@ -574,10 +824,10 @@ func (g *Graph) Start(wg *sync.WaitGroup, first bool) { // start or continue
 		// and not just selectively the subset with no indegree.
 		if (!first) || indegree[v] == 0 {
 			// ensure state is started before continuing on to next vertex
-			for !v.Type.SendEvent(eventStart, true, false) {
+			for !v.SendEvent(eventStart, true, false) {
 				if DEBUG {
 					// if SendEvent fails, we aren't up yet
-					log.Printf("%v[%v]: Retrying SendEvent(Start)", v.GetType(), v.GetName())
+					log.Printf("%v[%v]: Retrying SendEvent(Start)", v.Kind(), v.GetName())
 					// sleep here briefly or otherwise cause
 					// a different goroutine to be scheduled
 					time.Sleep(1 * time.Millisecond)
@@ -587,14 +837,21 @@ func (g *Graph) Start(wg *sync.WaitGroup, first bool) { // start or continue
 	}
 }
 
+// Pause sends pause events to the graph in a topological sort order.
 func (g *Graph) Pause() {
+	log.Printf("State: %v -> %v", g.setState(graphStatePausing), g.getState())
+	defer log.Printf("State: %v -> %v", g.setState(graphStatePaused), g.getState())
 	t, _ := g.TopologicalSort()
 	for _, v := range t { // squeeze out the events...
-		v.Type.SendEvent(eventPause, true, false)
+		v.SendEvent(eventPause, true, false)
 	}
 }
 
+// Exit sends exit events to the graph in a topological sort order.
 func (g *Graph) Exit() {
+	if g == nil {
+		return
+	} // empty graph that wasn't populated yet
 	t, _ := g.TopologicalSort()
 	for _, v := range t { // squeeze out the events...
 		// turn off the taps...
@@ -602,27 +859,28 @@ func (g *Graph) Exit() {
 		// when we hit the 'default' in the select statement!
 		// XXX: we can do this to quiesce, but it's not necessary now
 
-		v.Type.SendEvent(eventExit, true, false)
+		v.SendEvent(eventExit, true, false)
 	}
 }
 
-func (g *Graph) SetConvergedCallback(ctimeout int, converged chan bool) {
+// AssociateData associates some data with the object in the graph in question
+func (g *Graph) AssociateData(converger Converger) {
 	for v := range g.GetVerticesChan() {
-		v.Type.SetConvegedCallback(ctimeout, converged)
+		v.Res.AssociateData(converger)
 	}
 }
 
-// in array function to test *vertices in a slice of *vertices
+// VertexContains is an "in array" function to test for a vertex in a slice of vertices.
-func HasVertex(v *Vertex, haystack []*Vertex) bool {
+func VertexContains(needle *Vertex, haystack []*Vertex) bool {
-	for _, r := range haystack {
+	for _, v := range haystack {
-		if v == r {
+		if needle == v {
 			return true
 		}
 	}
 	return false
 }
 
-// reverse a list of vertices
+// Reverse reverses a list of vertices.
 func Reverse(vs []*Vertex) []*Vertex {
 	//var out []*Vertex       // XXX: golint suggests, but it fails testing
 	out := make([]*Vertex, 0) // empty list

pkg.go

@@ -0,0 +1,565 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	//"packagekit" // TODO
+	"encoding/gob"
+	"errors"
+	"fmt"
+	"log"
+	"path"
+	"strings"
+)
+
+func init() {
+	gob.Register(&PkgRes{})
+}
+
+// PkgRes is a package resource for packagekit.
+type PkgRes struct {
+	BaseRes          `yaml:",inline"`
+	State            string `yaml:"state"`            // state: installed, uninstalled, newest, <version>
+	AllowUntrusted   bool   `yaml:"allowuntrusted"`   // allow untrusted packages to be installed?
+	AllowNonFree     bool   `yaml:"allownonfree"`     // allow nonfree packages to be found?
+	AllowUnsupported bool   `yaml:"allowunsupported"` // allow unsupported packages to be found?
+	//bus              *Conn    // pk bus connection
+	fileList []string // FIXME: update if pkg changes
+}
+
+// NewPkgRes is a constructor for this resource. It also calls Init() for you.
+func NewPkgRes(name, state string, allowuntrusted, allownonfree, allowunsupported bool) *PkgRes {
+	obj := &PkgRes{
+		BaseRes: BaseRes{
+			Name: name,
+		},
+		State:            state,
+		AllowUntrusted:   allowuntrusted,
+		AllowNonFree:     allownonfree,
+		AllowUnsupported: allowunsupported,
+	}
+	obj.Init()
+	return obj
+}
+
+// Init runs some startup code for this resource.
+func (obj *PkgRes) Init() {
+	obj.BaseRes.kind = "Pkg"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+
+	bus := NewBus()
+	if bus == nil {
+		log.Fatal("Can't connect to PackageKit bus.")
+	}
+	defer bus.Close()
+
+	result, err := obj.pkgMappingHelper(bus)
+	if err != nil {
+		// FIXME: return error?
+		log.Fatalf("The pkgMappingHelper failed with: %v.", err)
+		return
+	}
+
+	data, ok := result[obj.Name] // lookup single package (init does just one)
+	// package doesn't exist, this is an error!
+	if !ok || !data.Found {
+		// FIXME: return error?
+		log.Fatalf("Can't find package named '%s'.", obj.Name)
+		return
+	}
+
+	packageIDs := []string{data.PackageID} // just one for now
+	filesMap, err := bus.GetFilesByPackageID(packageIDs)
+	if err != nil {
+		// FIXME: return error?
+		log.Fatalf("Can't run GetFilesByPackageID: %v", err)
+		return
+	}
+	if files, ok := filesMap[data.PackageID]; ok {
+		obj.fileList = DirifyFileList(files, false)
+	}
+}
+
+// Validate checks if the resource data structure was populated correctly.
+func (obj *PkgRes) Validate() bool {
+	if obj.State == "" {
+		return false
+	}
+
+	return true
+}
+
+// Watch is the primary listener for this resource and it outputs events.
+// It uses the PackageKit UpdatesChanged signal to watch for changes.
+// TODO: https://github.com/hughsie/PackageKit/issues/109
+// TODO: https://github.com/hughsie/PackageKit/issues/110
+func (obj *PkgRes) Watch(processChan chan Event) {
+	if obj.IsWatching() {
+		return
+	}
+	obj.SetWatching(true)
+	defer obj.SetWatching(false)
+	cuuid := obj.converger.Register()
+	defer cuuid.Unregister()
+
+	bus := NewBus()
+	if bus == nil {
+		log.Fatal("Can't connect to PackageKit bus.")
+	}
+	defer bus.Close()
+
+	ch, err := bus.WatchChanges()
+	if err != nil {
+		log.Fatalf("Error adding signal match: %v", err)
+	}
+
+	var send = false // send event?
+	var exit = false
+	var dirty = false
+
+	for {
+		if DEBUG {
+			log.Printf("%v: Watching...", obj.fmtNames(obj.getNames()))
+		}
+
+		obj.SetState(resStateWatching) // reset
+		select {
+		case event := <-ch:
+			cuuid.SetConverged(false)
+
+			// FIXME: ask packagekit for info on what packages changed
+			if DEBUG {
+				log.Printf("%v: Event: %v", obj.fmtNames(obj.getNames()), event.Name)
+			}
+
+			// since the chan is buffered, remove any supplemental
+			// events since they would just be duplicates anyways!
+			for len(ch) > 0 { // we can detect pending count here!
+				<-ch // discard
+			}
+
+			send = true
+			dirty = true
+
+		case event := <-obj.events:
+			cuuid.SetConverged(false)
+			if exit, send = obj.ReadEvent(&event); exit {
+				return // exit
+			}
+			dirty = false // these events don't invalidate state
+
+		case <-cuuid.ConvergedTimer():
+			cuuid.SetConverged(true) // converged!
+			continue
+		}
+
+		// do all our event sending all together to avoid duplicate msgs
+		if send {
+			send = false
+			// only invalid state on certain types of events
+			if dirty {
+				dirty = false
+				obj.isStateOK = false // something made state dirty
+			}
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
+		}
+	}
+}
+
+// get list of names when grouped or not
+func (obj *PkgRes) getNames() []string {
+	if g := obj.GetGroup(); len(g) > 0 { // grouped elements
+		names := []string{obj.GetName()}
+		for _, x := range g {
+			pkg, ok := x.(*PkgRes) // convert from Res
+			if ok {
+				names = append(names, pkg.Name)
+			}
+		}
+		return names
+	}
+	return []string{obj.GetName()}
+}
+
+// pretty print for header values
+func (obj *PkgRes) fmtNames(names []string) string {
+	if len(obj.GetGroup()) > 0 { // grouped elements
+		return fmt.Sprintf("%v[autogroup:(%v)]", obj.Kind(), strings.Join(names, ","))
+	}
+	return fmt.Sprintf("%v[%v]", obj.Kind(), obj.GetName())
+}
+
+func (obj *PkgRes) groupMappingHelper() map[string]string {
+	var result = make(map[string]string)
+	if g := obj.GetGroup(); len(g) > 0 { // add any grouped elements
+		for _, x := range g {
+			pkg, ok := x.(*PkgRes) // convert from Res
+			if !ok {
+				log.Fatalf("Grouped member %v is not a %v", x, obj.Kind())
+			}
+			result[pkg.Name] = pkg.State
+		}
+	}
+	return result
+}
+
+func (obj *PkgRes) pkgMappingHelper(bus *Conn) (map[string]*PkPackageIDActionData, error) {
+	packageMap := obj.groupMappingHelper() // get the grouped values
+	packageMap[obj.Name] = obj.State       // key is pkg name, value is pkg state
+	var filter uint64                      // initializes at the "zero" value of 0
+	filter += PK_FILTER_ENUM_ARCH          // always search in our arch (optional!)
+	// we're requesting latest version, or to narrow down install choices!
+	if obj.State == "newest" || obj.State == "installed" {
+		// if we add this, we'll still see older packages if installed
+		// this is an optimization, and is *optional*, this logic is
+		// handled inside of PackagesToPackageIDs now automatically!
+		filter += PK_FILTER_ENUM_NEWEST // only search for newest packages
+	}
+	if !obj.AllowNonFree {
+		filter += PK_FILTER_ENUM_FREE
+	}
+	if !obj.AllowUnsupported {
+		filter += PK_FILTER_ENUM_SUPPORTED
+	}
+	result, e := bus.PackagesToPackageIDs(packageMap, filter)
+	if e != nil {
+		return nil, fmt.Errorf("Can't run PackagesToPackageIDs: %v", e)
+	}
+	return result, nil
+}
+
+// CheckApply checks the resource state and applies the resource if the bool
+// input is true. It returns error info and if the state check passed or not.
+func (obj *PkgRes) CheckApply(apply bool) (checkok bool, err error) {
+	log.Printf("%v: CheckApply(%t)", obj.fmtNames(obj.getNames()), apply)
+
+	if obj.State == "" { // TODO: Validate() should replace this check!
+		log.Fatalf("%v: Package state is undefined!", obj.fmtNames(obj.getNames()))
+	}
+
+	if obj.isStateOK { // cache the state
+		return true, nil
+	}
+
+	bus := NewBus()
+	if bus == nil {
+		return false, errors.New("Can't connect to PackageKit bus.")
+	}
+	defer bus.Close()
+
+	result, err := obj.pkgMappingHelper(bus)
+	if err != nil {
+		return false, fmt.Errorf("The pkgMappingHelper failed with: %v.", err)
+	}
+
+	packageMap := obj.groupMappingHelper() // map[string]string
+	packageList := []string{obj.Name}
+	packageList = append(packageList, StrMapKeys(packageMap)...)
+	//stateList := []string{obj.State}
+	//stateList = append(stateList, StrMapValues(packageMap)...)
+
+	// TODO: at the moment, all the states are the same, but
+	// eventually we might be able to drop this constraint!
+	states, err := FilterState(result, packageList, obj.State)
+	if err != nil {
+		return false, fmt.Errorf("The FilterState method failed with: %v.", err)
+	}
+	data, _ := result[obj.Name] // if above didn't error, we won't either!
+	validState := BoolMapTrue(BoolMapValues(states))
+
+	// obj.State == "installed" || "uninstalled" || "newest" || "4.2-1.fc23"
+	switch obj.State {
+	case "installed":
+		fallthrough
+	case "uninstalled":
+		fallthrough
+	case "newest":
+		if validState {
+			obj.isStateOK = true // reset
+			return true, nil     // state is correct, exit!
+		}
+	default: // version string
+		if obj.State == data.Version && data.Version != "" {
+			obj.isStateOK = true // reset
+			return true, nil
+		}
+	}
+
+	// state is not okay, no work done, exit, but without error
+	if !apply {
+		return false, nil
+	}
+
+	// apply portion
+	log.Printf("%v: Apply", obj.fmtNames(obj.getNames()))
+	readyPackages, err := FilterPackageState(result, packageList, obj.State)
+	if err != nil {
+		return false, err // fail
+	}
+	// these are the packages that actually need their states applied!
+	applyPackages := StrFilterElementsInList(readyPackages, packageList)
+	packageIDs, _ := FilterPackageIDs(result, applyPackages) // would be same err as above
+
+	var transactionFlags uint64 // initializes at the "zero" value of 0
+	if !obj.AllowUntrusted {    // allow
+		transactionFlags += PK_TRANSACTION_FLAG_ENUM_ONLY_TRUSTED
+	}
+	// apply correct state!
+	log.Printf("%v: Set: %v...", obj.fmtNames(StrListIntersection(applyPackages, obj.getNames())), obj.State)
+	switch obj.State {
+	case "uninstalled": // run remove
+		// NOTE: packageID is different than when installed, because now
+		// it has the "installed" flag added to the data portion if it!!
+		err = bus.RemovePackages(packageIDs, transactionFlags)
+
+	case "newest": // TODO: isn't this the same operation as install, below?
+		err = bus.UpdatePackages(packageIDs, transactionFlags)
+
+	case "installed":
+		fallthrough // same method as for "set specific version", below
+	default: // version string
+		err = bus.InstallPackages(packageIDs, transactionFlags)
+	}
+	if err != nil {
+		return false, err // fail
+	}
+	log.Printf("%v: Set: %v success!", obj.fmtNames(StrListIntersection(applyPackages, obj.getNames())), obj.State)
+	obj.isStateOK = true // reset
+	return false, nil    // success
+}
+
+// PkgUUID is the UUID struct for PkgRes.
+type PkgUUID struct {
+	BaseUUID
+	name  string // pkg name
+	state string // pkg state or "version"
+}
+
+// if and only if they are equivalent, return true
+// if they are not equivalent, return false
+func (obj *PkgUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*PkgUUID)
+	if !ok {
+		return false
+	}
+	// FIXME: match on obj.state vs. res.state ?
+	return obj.name == res.name
+}
+
+// PkgResAutoEdges holds the state of the auto edge generator.
+type PkgResAutoEdges struct {
+	fileList   []string
+	svcUUIDs   []ResUUID
+	testIsNext bool   // safety
+	name       string // saved data from PkgRes obj
+	kind       string
+}
+
+// Next returns the next automatic edge.
+func (obj *PkgResAutoEdges) Next() []ResUUID {
+	if obj.testIsNext {
+		log.Fatal("Expecting a call to Test()")
+	}
+	obj.testIsNext = true // set after all the errors paths are past
+
+	// first return any matching svcUUIDs
+	if x := obj.svcUUIDs; len(x) > 0 {
+		return x
+	}
+
+	var result []ResUUID
+	// return UUID's for whatever is in obj.fileList
+	for _, x := range obj.fileList {
+		var reversed = false // cheat by passing a pointer
+		result = append(result, &FileUUID{
+			BaseUUID: BaseUUID{
+				name:     obj.name,
+				kind:     obj.kind,
+				reversed: &reversed,
+			},
+			path: x, // what matters
+		}) // build list
+	}
+	return result
+}
+
+// Test gets results of the earlier Next() call, & returns if we should continue!
+func (obj *PkgResAutoEdges) Test(input []bool) bool {
+	if !obj.testIsNext {
+		log.Fatal("Expecting a call to Next()")
+	}
+
+	// ack the svcUUID's...
+	if x := obj.svcUUIDs; len(x) > 0 {
+		if y := len(x); y != len(input) {
+			log.Fatalf("Expecting %d value(s)!", y)
+		}
+		obj.svcUUIDs = []ResUUID{} // empty
+		obj.testIsNext = false
+		return true
+	}
+
+	count := len(obj.fileList)
+	if count != len(input) {
+		log.Fatalf("Expecting %d value(s)!", count)
+	}
+	obj.testIsNext = false // set after all the errors paths are past
+
+	// while i do believe this algorithm generates the *correct* result, i
+	// don't know if it does so in the optimal way. improvements welcome!
+	// the basic logic is:
+	// 0) Next() returns whatever is in fileList
+	// 1) Test() computes the dirname of each file, and removes duplicates
+	// and dirname's that have been in the path of an ack from input results
+	// 2) It then simplifies the list by removing the common path prefixes
+	// 3) Lastly, the remaining set of files (dirs) is used as new fileList
+	// 4) We then iterate in (0) until the fileList is empty!
+	var dirs = make([]string, count)
+	done := []string{}
+	for i := 0; i < count; i++ {
+		dir := Dirname(obj.fileList[i]) // dirname of /foo/ should be /
+		dirs[i] = dir
+		if input[i] {
+			done = append(done, dir)
+		}
+	}
+	nodupes := StrRemoveDuplicatesInList(dirs)                // remove duplicates
+	nodones := StrFilterElementsInList(done, nodupes)         // filter out done
+	noempty := StrFilterElementsInList([]string{""}, nodones) // remove the "" from /
+	obj.fileList = RemoveCommonFilePrefixes(noempty)          // magic
+
+	if len(obj.fileList) == 0 { // nothing more, don't continue
+		return false
+	}
+	return true // continue, there are more files!
+}
+
+// AutoEdges produces an object which generates a minimal pkg file optimization
+// sequence of edges.
+func (obj *PkgRes) AutoEdges() AutoEdge {
+	// in contrast with the FileRes AutoEdges() function which contains
+	// more of the mechanics, most of the AutoEdge mechanics for the PkgRes
+	// is contained in the Test() method! This design is completely okay!
+
+	// add matches for any svc resources found in pkg definition!
+	var svcUUIDs []ResUUID
+	for _, x := range ReturnSvcInFileList(obj.fileList) {
+		var reversed = false
+		svcUUIDs = append(svcUUIDs, &SvcUUID{
+			BaseUUID: BaseUUID{
+				name:     obj.GetName(),
+				kind:     obj.Kind(),
+				reversed: &reversed,
+			},
+			name: x, // the svc name itself in the SvcUUID object!
+		}) // build list
+	}
+
+	return &PkgResAutoEdges{
+		fileList:   RemoveCommonFilePrefixes(obj.fileList), // clean start!
+		svcUUIDs:   svcUUIDs,
+		testIsNext: false,         // start with Next() call
+		name:       obj.GetName(), // save data for PkgResAutoEdges obj
+		kind:       obj.Kind(),
+	}
+}
+
+// GetUUIDs includes all params to make a unique identification of this object.
+// Most resources only return one, although some resources can return multiple.
+func (obj *PkgRes) GetUUIDs() []ResUUID {
+	x := &PkgUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		name:     obj.Name,
+		state:    obj.State,
+	}
+	result := []ResUUID{x}
+	return result
+}
+
+// GroupCmp returns whether two resources can be grouped together or not.
+// can these two resources be merged ?
+// (aka does this resource support doing so?)
+// will resource allow itself to be grouped _into_ this obj?
+func (obj *PkgRes) GroupCmp(r Res) bool {
+	res, ok := r.(*PkgRes)
+	if !ok {
+		return false
+	}
+	objStateIsVersion := (obj.State != "installed" && obj.State != "uninstalled" && obj.State != "newest") // must be a ver. string
+	resStateIsVersion := (res.State != "installed" && res.State != "uninstalled" && res.State != "newest") // must be a ver. string
+	if objStateIsVersion || resStateIsVersion {
+		// can't merge specific version checks atm
+		return false
+	}
+	// FIXME: keep it simple for now, only merge same states
+	if obj.State != res.State {
+		return false
+	}
+	return true
+}
+
+// Compare two resources and return if they are equivalent.
+func (obj *PkgRes) Compare(res Res) bool {
+	switch res.(type) {
+	case *PkgRes:
+		res := res.(*PkgRes)
+		if !obj.BaseRes.Compare(res) { // call base Compare
+			return false
+		}
+
+		if obj.Name != res.Name {
+			return false
+		}
+		if obj.State != res.State {
+			return false
+		}
+		if obj.AllowUntrusted != res.AllowUntrusted {
+			return false
+		}
+		if obj.AllowNonFree != res.AllowNonFree {
+			return false
+		}
+		if obj.AllowUnsupported != res.AllowUnsupported {
+			return false
+		}
+	default:
+		return false
+	}
+	return true
+}
+
+// return a list of svc names for matches like /usr/lib/systemd/system/*.service
+func ReturnSvcInFileList(fileList []string) []string {
+	result := []string{}
+	for _, x := range fileList {
+		dirname, basename := path.Split(path.Clean(x))
+		// TODO: do we also want to look for /etc/systemd/system/ ?
+		if dirname != "/usr/lib/systemd/system/" {
+			continue
+		}
+		if !strings.HasSuffix(basename, ".service") {
+			continue
+		}
+		if s := strings.TrimSuffix(basename, ".service"); !StrInList(s, result) {
+			result = append(result, s)
+		}
+	}
+	return result
+}

puppet.go

@@ -0,0 +1,141 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"bufio"
+	"io"
+	"log"
+	"os/exec"
+	"strconv"
+	"strings"
+)
+
+const (
+	// PuppetYAMLBufferSize is the maximum buffer size for the yaml input data
+	PuppetYAMLBufferSize = 65535
+)
+
+func runPuppetCommand(cmd *exec.Cmd) ([]byte, error) {
+	if DEBUG {
+		log.Printf("Puppet: running command: %v", cmd)
+	}
+
+	stdout, err := cmd.StdoutPipe()
+	if err != nil {
+		log.Printf("Puppet: Error opening pipe to puppet command: %v", err)
+		return nil, err
+	}
+	stderr, err := cmd.StderrPipe()
+	if err != nil {
+		log.Printf("Puppet: Error opening error pipe to puppet command: %v", err)
+		return nil, err
+	}
+
+	if err := cmd.Start(); err != nil {
+		log.Printf("Puppet: Error starting puppet command: %v", err)
+		return nil, err
+	}
+
+	// XXX: the current implementation is likely prone to fail
+	// as soon as the YAML data overflows the buffer.
+	data := make([]byte, PuppetYAMLBufferSize)
+	var result []byte
+	for err == nil {
+		var count int
+		count, err = stdout.Read(data)
+		if err != nil && err != io.EOF {
+			log.Printf("Puppet: Error reading YAML data from puppet: %v", err)
+			return nil, err
+		}
+		// Slicing down to the number of actual bytes is important, the YAML parser
+		// will choke on an oversized slice. http://stackoverflow.com/a/33726617/3356612
+		result = append(result, data[0:count]...)
+	}
+	if DEBUG {
+		log.Printf("Puppet: read %v bytes of data from puppet", len(result))
+	}
+	for scanner := bufio.NewScanner(stderr); scanner.Scan(); {
+		log.Printf("Puppet: (output) %v", scanner.Text())
+	}
+	if err := cmd.Wait(); err != nil {
+		log.Printf("Puppet: Error: puppet command did not complete: %v", err)
+		return nil, err
+	}
+
+	return result, nil
+}
+
+// ParseConfigFromPuppet takes a special puppet param string and config and
+// returns the graph configuration structure.
+func ParseConfigFromPuppet(puppetParam, puppetConf string) *GraphConfig {
+	var puppetConfArg string
+	if puppetConf != "" {
+		puppetConfArg = "--config=" + puppetConf
+	}
+
+	var cmd *exec.Cmd
+	if puppetParam == "agent" {
+		cmd = exec.Command("puppet", "mgmtgraph", "print", puppetConfArg)
+	} else if strings.HasSuffix(puppetParam, ".pp") {
+		cmd = exec.Command("puppet", "mgmtgraph", "print", puppetConfArg, "--manifest", puppetParam)
+	} else {
+		cmd = exec.Command("puppet", "mgmtgraph", "print", puppetConfArg, "--code", puppetParam)
+	}
+
+	log.Println("Puppet: launching translator")
+
+	var config GraphConfig
+	if data, err := runPuppetCommand(cmd); err != nil {
+		return nil
+	} else if err := config.Parse(data); err != nil {
+		log.Printf("Puppet: Error: Could not parse YAML output with Parse: %v", err)
+		return nil
+	}
+
+	return &config
+}
+
+// PuppetInterval returns the graph refresh interval from the puppet configuration.
+func PuppetInterval(puppetConf string) int {
+	if DEBUG {
+		log.Printf("Puppet: determining graph refresh interval")
+	}
+	var cmd *exec.Cmd
+	if puppetConf != "" {
+		cmd = exec.Command("puppet", "config", "print", "runinterval", "--config", puppetConf)
+	} else {
+		cmd = exec.Command("puppet", "config", "print", "runinterval")
+	}
+
+	log.Println("Puppet: inspecting runinterval configuration")
+
+	interval := 1800
+	data, err := runPuppetCommand(cmd)
+	if err != nil {
+		log.Printf("Puppet: could not determine configured run interval (%v), using default of %v", err, interval)
+		return interval
+	}
+	result, err := strconv.ParseInt(strings.TrimSpace(string(data)), 10, 0)
+	if err != nil {
+		log.Printf("Puppet: error reading numeric runinterval value (%v), using default of %v", err, interval)
+		return interval
+	}
+
+	return int(result)
+}

resources.go

@@ -0,0 +1,366 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"bytes"
+	"encoding/base64"
+	"encoding/gob"
+	"fmt"
+	"log"
+)
+
+//go:generate stringer -type=resState -output=resstate_stringer.go
+type resState int
+
+const (
+	resStateNil resState = iota
+	resStateWatching
+	resStateEvent // an event has happened, but we haven't poked yet
+	resStateCheckApply
+	resStatePoking
+)
+
+// ResUUID is a unique identifier for a resource, namely it's name, and the kind ("type").
+type ResUUID interface {
+	GetName() string
+	Kind() string
+	IFF(ResUUID) bool
+
+	Reversed() bool // true means this resource happens before the generator
+}
+
+// The BaseUUID struct is used to provide a unique resource identifier.
+type BaseUUID struct {
+	name string // name and kind are the values of where this is coming from
+	kind string
+
+	reversed *bool // piggyback edge information here
+}
+
+// The AutoEdge interface is used to implement the autoedges feature.
+type AutoEdge interface {
+	Next() []ResUUID  // call to get list of edges to add
+	Test([]bool) bool // call until false
+}
+
+// MetaParams is a struct will all params that apply to every resource.
+type MetaParams struct {
+	AutoEdge  bool `yaml:"autoedge"`  // metaparam, should we generate auto edges? // XXX should default to true
+	AutoGroup bool `yaml:"autogroup"` // metaparam, should we auto group? // XXX should default to true
+	Noop      bool `yaml:"noop"`
+}
+
+// The Base interface is everything that is common to all resources.
+// Everything here only needs to be implemented once, in the BaseRes.
+type Base interface {
+	GetName() string // can't be named "Name()" because of struct field
+	SetName(string)
+	setKind(string)
+	Kind() string
+	Meta() *MetaParams
+	AssociateData(Converger)
+	IsWatching() bool
+	SetWatching(bool)
+	GetState() resState
+	SetState(resState)
+	SendEvent(eventName, bool, bool) bool
+	ReadEvent(*Event) (bool, bool) // TODO: optional here?
+	GroupCmp(Res) bool             // TODO: is there a better name for this?
+	GroupRes(Res) error            // group resource (arg) into self
+	IsGrouped() bool               // am I grouped?
+	SetGrouped(bool)               // set grouped bool
+	GetGroup() []Res               // return everyone grouped inside me
+	SetGroup([]Res)
+}
+
+// Res is the minimum interface you need to implement to define a new resource.
+type Res interface {
+	Base // include everything from the Base interface
+	Init()
+	//Validate() bool    // TODO: this might one day be added
+	GetUUIDs() []ResUUID // most resources only return one
+	Watch(chan Event)    // send on channel to signal process() events
+	CheckApply(bool) (bool, error)
+	AutoEdges() AutoEdge
+	Compare(Res) bool
+	CollectPattern(string) // XXX: temporary until Res collection is more advanced
+}
+
+// BaseRes is the base struct that gets used in every resource.
+type BaseRes struct {
+	Name       string     `yaml:"name"`
+	MetaParams MetaParams `yaml:"meta"` // struct of all the metaparams
+	kind       string
+	events     chan Event
+	converger  Converger // converged tracking
+	state      resState
+	watching   bool  // is Watch() loop running ?
+	isStateOK  bool  // whether the state is okay based on events or not
+	isGrouped  bool  // am i contained within a group?
+	grouped    []Res // list of any grouped resources
+}
+
+// UUIDExistsInUUIDs wraps the IFF method when used with a list of UUID's.
+func UUIDExistsInUUIDs(uuid ResUUID, uuids []ResUUID) bool {
+	for _, u := range uuids {
+		if uuid.IFF(u) {
+			return true
+		}
+	}
+	return false
+}
+
+// GetName returns the name of the resource.
+func (obj *BaseUUID) GetName() string {
+	return obj.name
+}
+
+// Kind returns the kind of resource.
+func (obj *BaseUUID) Kind() string {
+	return obj.kind
+}
+
+// IFF looks at two UUID's and if and only if they are equivalent, returns true.
+// If they are not equivalent, it returns false.
+// Most resources will want to override this method, since it does the important
+// work of actually discerning if two resources are identical in function.
+func (obj *BaseUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*BaseUUID)
+	if !ok {
+		return false
+	}
+	return obj.name == res.name
+}
+
+// Reversed is part of the ResUUID interface, and true means this resource
+// happens before the generator.
+func (obj *BaseUUID) Reversed() bool {
+	if obj.reversed == nil {
+		log.Fatal("Programming error!")
+	}
+	return *obj.reversed
+}
+
+// Init initializes structures like channels if created without New constructor.
+func (obj *BaseRes) Init() {
+	obj.events = make(chan Event) // unbuffered chan size to avoid stale events
+}
+
+// GetName is used by all the resources to Get the name.
+func (obj *BaseRes) GetName() string {
+	return obj.Name
+}
+
+// SetName is used to set the name of the resource.
+func (obj *BaseRes) SetName(name string) {
+	obj.Name = name
+}
+
+// setKind sets the kind. This is used internally for exported resources.
+func (obj *BaseRes) setKind(kind string) {
+	obj.kind = kind
+}
+
+// Kind returns the kind of resource this is.
+func (obj *BaseRes) Kind() string {
+	return obj.kind
+}
+
+// Meta returns the MetaParams as a reference, which we can then get/set on.
+func (obj *BaseRes) Meta() *MetaParams {
+	return &obj.MetaParams
+}
+
+// AssociateData associates some data with the object in question.
+func (obj *BaseRes) AssociateData(converger Converger) {
+	obj.converger = converger
+}
+
+// IsWatching tells us if the Watch() function is running.
+func (obj *BaseRes) IsWatching() bool {
+	return obj.watching
+}
+
+// SetWatching stores the status of if the Watch() function is running.
+func (obj *BaseRes) SetWatching(b bool) {
+	obj.watching = b
+}
+
+// GetState returns the state of the resource.
+func (obj *BaseRes) GetState() resState {
+	return obj.state
+}
+
+// SetState sets the state of the resource.
+func (obj *BaseRes) SetState(state resState) {
+	if DEBUG {
+		log.Printf("%v[%v]: State: %v -> %v", obj.Kind(), obj.GetName(), obj.GetState(), state)
+	}
+	obj.state = state
+}
+
+// SendEvent pushes an event into the message queue for a particular vertex
+func (obj *BaseRes) SendEvent(event eventName, sync bool, activity bool) bool {
+	// TODO: isn't this race-y ?
+	if !obj.IsWatching() { // element has already exited
+		return false // if we don't return, we'll block on the send
+	}
+	if !sync {
+		obj.events <- Event{event, nil, "", activity}
+		return true
+	}
+
+	resp := make(chan bool)
+	obj.events <- Event{event, resp, "", activity}
+	for {
+		value := <-resp
+		// wait until true value
+		if value {
+			return true
+		}
+	}
+}
+
+// ReadEvent processes events when a select gets one, and handles the pause
+// code too! The return values specify if we should exit and poke respectively.
+func (obj *BaseRes) ReadEvent(event *Event) (exit, poke bool) {
+	event.ACK()
+	switch event.Name {
+	case eventStart:
+		return false, true
+
+	case eventPoke:
+		return false, true
+
+	case eventBackPoke:
+		return false, true // forward poking in response to a back poke!
+
+	case eventExit:
+		return true, false
+
+	case eventPause:
+		// wait for next event to continue
+		select {
+		case e := <-obj.events:
+			e.ACK()
+			if e.Name == eventExit {
+				return true, false
+			} else if e.Name == eventStart { // eventContinue
+				return false, false // don't poke on unpause!
+			} else {
+				// if we get a poke event here, it's a bug!
+				log.Fatalf("%v[%v]: Unknown event: %v, while paused!", obj.Kind(), obj.GetName(), e)
+			}
+		}
+
+	default:
+		log.Fatal("Unknown event: ", event)
+	}
+	return true, false // required to keep the stupid go compiler happy
+}
+
+// GroupCmp compares two resources and decides if they're suitable for grouping
+// You'll probably want to override this method when implementing a resource...
+func (obj *BaseRes) GroupCmp(res Res) bool {
+	return false // base implementation assumes false, override me!
+}
+
+// GroupRes groups resource (arg) into self.
+func (obj *BaseRes) GroupRes(res Res) error {
+	if l := len(res.GetGroup()); l > 0 {
+		return fmt.Errorf("Res: %v already contains %d grouped resources!", res, l)
+	}
+	if res.IsGrouped() {
+		return fmt.Errorf("Res: %v is already grouped!", res)
+	}
+
+	obj.grouped = append(obj.grouped, res)
+	res.SetGrouped(true) // i am contained _in_ a group
+	return nil
+}
+
+// IsGrouped determines if we are grouped.
+func (obj *BaseRes) IsGrouped() bool { // am I grouped?
+	return obj.isGrouped
+}
+
+// SetGrouped sets a flag to tell if we are grouped.
+func (obj *BaseRes) SetGrouped(b bool) {
+	obj.isGrouped = b
+}
+
+// GetGroup returns everyone grouped inside me.
+func (obj *BaseRes) GetGroup() []Res { // return everyone grouped inside me
+	return obj.grouped
+}
+
+// SetGroup sets the grouped resources into me.
+func (obj *BaseRes) SetGroup(g []Res) {
+	obj.grouped = g
+}
+
+// Compare is the base compare method, which also handles the metaparams cmp
+func (obj *BaseRes) Compare(res Res) bool {
+	if obj.Meta().Noop != res.Meta().Noop {
+		// obj is the existing res, res is the *new* resource
+		// if we go from no-noop -> noop, we can re-use the obj
+		// if we go from noop -> no-noop, we need to regenerate
+		if obj.Meta().Noop { // asymmetrical
+			return false // going from noop to no-noop!
+		}
+	}
+	return true
+}
+
+// CollectPattern is used for resource collection.
+func (obj *BaseRes) CollectPattern(pattern string) {
+	// XXX: default method is empty
+}
+
+// ResToB64 encodes a resource to a base64 encoded string (after serialization)
+func ResToB64(res Res) (string, error) {
+	b := bytes.Buffer{}
+	e := gob.NewEncoder(&b)
+	err := e.Encode(&res) // pass with &
+	if err != nil {
+		return "", fmt.Errorf("Gob failed to encode: %v", err)
+	}
+	return base64.StdEncoding.EncodeToString(b.Bytes()), nil
+}
+
+// B64ToRes decodes a resource from a base64 encoded string (after deserialization)
+func B64ToRes(str string) (Res, error) {
+	var output interface{}
+	bb, err := base64.StdEncoding.DecodeString(str)
+	if err != nil {
+		return nil, fmt.Errorf("Base64 failed to decode: %v", err)
+	}
+	b := bytes.NewBuffer(bb)
+	d := gob.NewDecoder(b)
+	err = d.Decode(&output) // pass with &
+	if err != nil {
+		return nil, fmt.Errorf("Gob failed to decode: %v", err)
+	}
+	res, ok := output.(Res)
+	if !ok {
+		return nil, fmt.Errorf("Output %v is not a Res", res)
+
+	}
+	return res, nil
+}

resources_test.go

@@ -0,0 +1,173 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"bytes"
+	"encoding/base64"
+	"encoding/gob"
+	"testing"
+)
+
+func TestMiscEncodeDecode1(t *testing.T) {
+	var err error
+	//gob.Register( &NoopRes{} ) // happens in noop.go : init()
+	//gob.Register( &FileRes{} ) // happens in file.go : init()
+	// ...
+
+	// encode
+	var input interface{} = &FileRes{}
+	b1 := bytes.Buffer{}
+	e := gob.NewEncoder(&b1)
+	err = e.Encode(&input) // pass with &
+	if err != nil {
+		t.Errorf("Gob failed to Encode: %v", err)
+	}
+	str := base64.StdEncoding.EncodeToString(b1.Bytes())
+
+	// decode
+	var output interface{}
+	bb, err := base64.StdEncoding.DecodeString(str)
+	if err != nil {
+		t.Errorf("Base64 failed to Decode: %v", err)
+	}
+	b2 := bytes.NewBuffer(bb)
+	d := gob.NewDecoder(b2)
+	err = d.Decode(&output) // pass with &
+	if err != nil {
+		t.Errorf("Gob failed to Decode: %v", err)
+	}
+
+	res1, ok := input.(Res)
+	if !ok {
+		t.Errorf("Input %v is not a Res", res1)
+		return
+	}
+	res2, ok := output.(Res)
+	if !ok {
+		t.Errorf("Output %v is not a Res", res2)
+		return
+	}
+	if !res1.Compare(res2) {
+		t.Error("The input and output Res values do not match!")
+	}
+}
+
+func TestMiscEncodeDecode2(t *testing.T) {
+	var err error
+	//gob.Register( &NoopRes{} ) // happens in noop.go : init()
+	//gob.Register( &FileRes{} ) // happens in file.go : init()
+	// ...
+
+	// encode
+	var input Res = &FileRes{}
+
+	b64, err := ResToB64(input)
+	if err != nil {
+		t.Errorf("Can't encode: %v", err)
+		return
+	}
+
+	output, err := B64ToRes(b64)
+	if err != nil {
+		t.Errorf("Can't decode: %v", err)
+		return
+	}
+
+	res1, ok := input.(Res)
+	if !ok {
+		t.Errorf("Input %v is not a Res", res1)
+		return
+	}
+	res2, ok := output.(Res)
+	if !ok {
+		t.Errorf("Output %v is not a Res", res2)
+		return
+	}
+	if !res1.Compare(res2) {
+		t.Error("The input and output Res values do not match!")
+	}
+}
+
+func TestIFF(t *testing.T) {
+	uuid := &BaseUUID{name: "/tmp/unit-test"}
+	same := &BaseUUID{name: "/tmp/unit-test"}
+	diff := &BaseUUID{name: "/tmp/other-file"}
+
+	if !uuid.IFF(same) {
+		t.Error("basic resource UUIDs with the same name should satisfy each other's IFF condition.")
+	}
+
+	if uuid.IFF(diff) {
+		t.Error("basic resource UUIDs with different names should NOT satisfy each other's IFF condition.")
+	}
+}
+
+func TestReadEvent(t *testing.T) {
+	res := FileRes{}
+
+	shouldExit := map[eventName]bool{
+		eventStart:    false,
+		eventPoke:     false,
+		eventBackPoke: false,
+		eventExit:     true,
+	}
+	shouldPoke := map[eventName]bool{
+		eventStart:    true,
+		eventPoke:     true,
+		eventBackPoke: true,
+		eventExit:     false,
+	}
+
+	for event, _ := range shouldExit {
+		exit, poke := res.ReadEvent(&Event{Name: event})
+		if exit != shouldExit[event] {
+			t.Errorf("resource.ReadEvent returned wrong exit flag for a %v event (%v, should be %v)",
+				event, exit, shouldExit[event])
+		}
+		if poke != shouldPoke[event] {
+			t.Errorf("resource.ReadEvent returned wrong poke flag for a %v event (%v, should be %v)",
+				event, poke, shouldPoke[event])
+		}
+	}
+
+	res.Init()
+	res.SetWatching(true)
+
+	// test result when a pause event is followed by start
+	go res.SendEvent(eventStart, false, false)
+	exit, poke := res.ReadEvent(&Event{Name: eventPause})
+	if exit {
+		t.Error("resource.ReadEvent returned wrong exit flag for a pause+start event (true, should be false)")
+	}
+	if poke {
+		t.Error("resource.ReadEvent returned wrong poke flag for a pause+start event (true, should be false)")
+	}
+
+	// test result when a pause event is followed by exit
+	go res.SendEvent(eventExit, false, false)
+	exit, poke = res.ReadEvent(&Event{Name: eventPause})
+	if !exit {
+		t.Error("resource.ReadEvent returned wrong exit flag for a pause+start event (false, should be true)")
+	}
+	if poke {
+		t.Error("resource.ReadEvent returned wrong poke flag for a pause+start event (true, should be false)")
+	}
+
+	// TODO: create a wrapper API around log, so that Fatals can be mocked and tested
+}

service.go

@@ -1,339 +0,0 @@
-// Mgmt
-// Copyright (C) 2013-2016+ James Shubin and the project contributors
-// Written by James Shubin <james@shubin.ca> and the project contributors
-//
-// This program is free software: you can redistribute it and/or modify
-// it under the terms of the GNU Affero General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-//
-// This program is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU Affero General Public License for more details.
-//
-// You should have received a copy of the GNU Affero General Public License
-// along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-// NOTE: docs are found at: https://godoc.org/github.com/coreos/go-systemd/dbus
-
-package main
-
-import (
-	"fmt"
-	systemd "github.com/coreos/go-systemd/dbus" // change namespace
-	"github.com/coreos/go-systemd/util"
-	"github.com/godbus/dbus" // namespace collides with systemd wrapper
-	"log"
-)
-
-type ServiceType struct {
-	BaseType `yaml:",inline"`
-	State    string `yaml:"state"`   // state: running, stopped
-	Startup  string `yaml:"startup"` // enabled, disabled, undefined
-}
-
-func NewServiceType(name, state, startup string) *ServiceType {
-	return &ServiceType{
-		BaseType: BaseType{
-			Name:   name,
-			events: make(chan Event),
-			vertex: nil,
-		},
-		State:   state,
-		Startup: startup,
-	}
-}
-
-func (obj *ServiceType) GetType() string {
-	return "Service"
-}
-
-// Service watcher
-func (obj *ServiceType) Watch() {
-	if obj.IsWatching() {
-		return
-	}
-	obj.SetWatching(true)
-	defer obj.SetWatching(false)
-
-	// obj.Name: service name
-	//vertex := obj.GetVertex()         // stored with SetVertex
-	if !util.IsRunningSystemd() {
-		log.Fatal("Systemd is not running.")
-	}
-
-	conn, err := systemd.NewSystemdConnection() // needs root access
-	if err != nil {
-		log.Fatal("Failed to connect to systemd: ", err)
-	}
-	defer conn.Close()
-
-	bus, err := dbus.SystemBus()
-	if err != nil {
-		log.Fatal("Failed to connect to bus: ", err)
-	}
-
-	// XXX: will this detect new units?
-	bus.BusObject().Call("org.freedesktop.DBus.AddMatch", 0,
-		"type='signal',interface='org.freedesktop.systemd1.Manager',member='Reloading'")
-	buschan := make(chan *dbus.Signal, 10)
-	bus.Signal(buschan)
-
-	var service = fmt.Sprintf("%v.service", obj.Name) // systemd name
-	var send = false                                  // send event?
-	var exit = false
-	var dirty = false
-	var invalid = false              // does the service exist or not?
-	var previous bool                // previous invalid value
-	set := conn.NewSubscriptionSet() // no error should be returned
-	subChannel, subErrors := set.Subscribe()
-	var activeSet = false
-
-	for {
-		// XXX: watch for an event for new units...
-		// XXX: detect if startup enabled/disabled value changes...
-
-		previous = invalid
-		invalid = false
-
-		// firstly, does service even exist or not?
-		loadstate, err := conn.GetUnitProperty(service, "LoadState")
-		if err != nil {
-			log.Printf("Failed to get property: %v", err)
-			invalid = true
-		}
-
-		if !invalid {
-			var notFound = (loadstate.Value == dbus.MakeVariant("not-found"))
-			if notFound { // XXX: in the loop we'll handle changes better...
-				log.Printf("Failed to find service: %v", service)
-				invalid = true // XXX ?
-			}
-		}
-
-		if previous != invalid { // if invalid changed, send signal
-			send = true
-			dirty = true
-		}
-
-		if invalid {
-			log.Printf("Waiting for: %v", service) // waiting for service to appear...
-			if activeSet {
-				activeSet = false
-				set.Remove(service) // no return value should ever occur
-			}
-
-			obj.SetState(typeWatching) // reset
-			select {
-			case _ = <-buschan: // XXX wait for new units event to unstick
-				obj.SetConvergedState(typeConvergedNil)
-				// loop so that we can see the changed invalid signal
-				log.Printf("Service[%v]->DaemonReload()", service)
-
-			case event := <-obj.events:
-				obj.SetConvergedState(typeConvergedNil)
-				if exit, send = obj.ReadEvent(&event); exit {
-					return // exit
-				}
-				if event.GetActivity() {
-					dirty = true
-				}
-
-			case _ = <-TimeAfterOrBlock(obj.ctimeout):
-				obj.SetConvergedState(typeConvergedTimeout)
-				obj.converged <- true
-				continue
-			}
-		} else {
-			if !activeSet {
-				activeSet = true
-				set.Add(service) // no return value should ever occur
-			}
-
-			log.Printf("Watching: %v", service) // attempting to watch...
-			obj.SetState(typeWatching)          // reset
-			select {
-			case event := <-subChannel:
-
-				log.Printf("Service event: %+v", event)
-				// NOTE: the value returned is a map for some reason...
-				if event[service] != nil {
-					// event[service].ActiveState is not nil
-					if event[service].ActiveState == "active" {
-						log.Printf("Service[%v]->Started()", service)
-					} else if event[service].ActiveState == "inactive" {
-						log.Printf("Service[%v]->Stopped!()", service)
-					} else {
-						log.Fatal("Unknown service state: ", event[service].ActiveState)
-					}
-				} else {
-					// service stopped (and ActiveState is nil...)
-					log.Printf("Service[%v]->Stopped", service)
-				}
-				send = true
-				dirty = true
-
-			case err := <-subErrors:
-				obj.SetConvergedState(typeConvergedNil) // XXX ?
-				log.Println("error:", err)
-				log.Fatal(err)
-				//vertex.events <- fmt.Sprintf("service: %v", "error") // XXX: how should we handle errors?
-
-			case event := <-obj.events:
-				obj.SetConvergedState(typeConvergedNil)
-				if exit, send = obj.ReadEvent(&event); exit {
-					return // exit
-				}
-				if event.GetActivity() {
-					dirty = true
-				}
-			}
-		}
-
-		if send {
-			send = false
-			if dirty {
-				dirty = false
-				obj.isStateOK = false // something made state dirty
-			}
-			Process(obj) // XXX: rename this function
-		}
-
-	}
-}
-
-func (obj *ServiceType) StateOK() bool {
-	if obj.isStateOK { // cache the state
-		return true
-	}
-
-	if !util.IsRunningSystemd() {
-		log.Fatal("Systemd is not running.")
-	}
-
-	conn, err := systemd.NewSystemdConnection() // needs root access
-	if err != nil {
-		log.Fatal("Failed to connect to systemd: ", err)
-	}
-	defer conn.Close()
-
-	var service = fmt.Sprintf("%v.service", obj.Name) // systemd name
-
-	loadstate, err := conn.GetUnitProperty(service, "LoadState")
-	if err != nil {
-		log.Printf("Failed to get load state: %v", err)
-		return false
-	}
-
-	// NOTE: we have to compare variants with other variants, they are really strings...
-	var notFound = (loadstate.Value == dbus.MakeVariant("not-found"))
-	if notFound {
-		log.Printf("Failed to find service: %v", service)
-		return false
-	}
-
-	// XXX: check service "enabled at boot" or not status...
-
-	//conn.GetUnitProperties(service)
-	activestate, err := conn.GetUnitProperty(service, "ActiveState")
-	if err != nil {
-		log.Fatal("Failed to get active state: ", err)
-	}
-
-	var running = (activestate.Value == dbus.MakeVariant("active"))
-
-	if obj.State == "running" {
-		if !running {
-			return false // we are in the wrong state
-		}
-	} else if obj.State == "stopped" {
-		if running {
-			return false
-		}
-	} else {
-		log.Fatal("Unknown state: ", obj.State)
-	}
-
-	return true // all is good, no state change needed
-}
-
-func (obj *ServiceType) Apply() bool {
-	log.Printf("%v[%v]: Apply", obj.GetType(), obj.GetName())
-
-	if !util.IsRunningSystemd() {
-		log.Fatal("Systemd is not running.")
-	}
-
-	conn, err := systemd.NewSystemdConnection() // needs root access
-	if err != nil {
-		log.Fatal("Failed to connect to systemd: ", err)
-	}
-	defer conn.Close()
-
-	var service = fmt.Sprintf("%v.service", obj.Name) // systemd name
-	var files = []string{service}                     // the service represented in a list
-	if obj.Startup == "enabled" {
-		_, _, err = conn.EnableUnitFiles(files, false, true)
-
-	} else if obj.Startup == "disabled" {
-		_, err = conn.DisableUnitFiles(files, false)
-	} else {
-		err = nil
-	}
-	if err != nil {
-		log.Printf("Unable to change startup status: %v", err)
-		return false
-	}
-
-	result := make(chan string, 1) // catch result information
-
-	if obj.State == "running" {
-		_, err := conn.StartUnit(service, "fail", result)
-		if err != nil {
-			log.Fatal("Failed to start unit: ", err)
-			return false
-		}
-	} else if obj.State == "stopped" {
-		_, err = conn.StopUnit(service, "fail", result)
-		if err != nil {
-			log.Fatal("Failed to stop unit: ", err)
-			return false
-		}
-	} else {
-		log.Fatal("Unknown state: ", obj.State)
-	}
-
-	status := <-result
-	if &status == nil {
-		log.Fatal("Result is nil")
-		return false
-	}
-	if status != "done" {
-		log.Fatal("Unknown return string: ", status)
-		return false
-	}
-
-	// XXX: also set enabled on boot
-
-	return true
-}
-
-func (obj *ServiceType) Compare(typ Type) bool {
-	switch typ.(type) {
-	case *ServiceType:
-		typ := typ.(*ServiceType)
-		if obj.Name != typ.Name {
-			return false
-		}
-		if obj.State != typ.State {
-			return false
-		}
-		if obj.Startup != typ.Startup {
-			return false
-		}
-	default:
-		return false
-	}
-	return true
-}

spec.in

@@ -1,13 +1,13 @@
 %global project_version __VERSION__
 %define debug_package %{nil}
 
-Name:		mgmt
+Name:		__PROGRAM__
 Version:	__VERSION__
 Release:	__RELEASE__
 Summary:	A next generation config management prototype!
 License:	AGPLv3+
 URL:		https://github.com/purpleidea/mgmt
-Source0:	https://dl.fedoraproject.org/pub/alt/purpleidea/mgmt/SOURCES/mgmt-%{project_version}.tar.bz2
+Source0:	https://dl.fedoraproject.org/pub/alt/purpleidea/__PROGRAM__/SOURCES/__PROGRAM__-%{project_version}.tar.bz2
 
 # graphviz should really be a "suggests", since technically it's optional
 Requires: graphviz
@@ -37,27 +37,36 @@ make build
 
 %install
 rm -rf %{buildroot}
-# _datadir is typically /usr/share/
+mkdir -p %{buildroot}/%{_unitdir}/
-install -d -m 0755 %{buildroot}/%{_datadir}/mgmt/
+install -pm 0644 misc/__PROGRAM__.service %{buildroot}/%{_unitdir}/
-cp -a AUTHORS COPYING COPYRIGHT DOCUMENTATION.md README.md THANKS examples/ %{buildroot}/%{_datadir}/mgmt/
 
 # install the binary
 mkdir -p %{buildroot}/%{_bindir}
-install -m 0755 mgmt %{buildroot}/%{_bindir}/mgmt
+install -m 0755 __PROGRAM__ %{buildroot}/%{_bindir}/__PROGRAM__
 
 # profile.d bash completion
 mkdir -p %{buildroot}%{_sysconfdir}/profile.d
-install misc/mgmt.bashrc -m 0755 %{buildroot}%{_sysconfdir}/profile.d/mgmt.sh
+install misc/bashrc.sh -m 0755 %{buildroot}%{_sysconfdir}/profile.d/__PROGRAM__.sh
 
 # etc dir
-mkdir -p %{buildroot}%{_sysconfdir}/mgmt/
+mkdir -p %{buildroot}%{_sysconfdir}/__PROGRAM__/
-install -m 0644 misc/mgmt.conf.example %{buildroot}%{_sysconfdir}/mgmt/mgmt.conf
+install -m 0644 misc/example.conf %{buildroot}%{_sysconfdir}/__PROGRAM__/__PROGRAM__.conf
 
 %files
-%attr(0755, root, root) %{_sysconfdir}/profile.d/mgmt.sh
+%attr(0755, root, root) %{_sysconfdir}/profile.d/__PROGRAM__.sh
-%{_datadir}/mgmt/*
+%{_bindir}/__PROGRAM__
-%{_bindir}/mgmt
+%{_sysconfdir}/__PROGRAM__/*
-%{_sysconfdir}/mgmt/*
+%{_unitdir}/__PROGRAM__.service
+
+# https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#Documentation
+# Please add docs one per line in alpha order to avoid diff churn.
+%doc AUTHORS
+%doc COPYING
+%doc COPYRIGHT
+%doc DOCUMENTATION.md
+%doc README.md
+%doc THANKS
+%doc examples/*
 
 # this changelog is auto-generated by git log
 %changelog

svc.go

@@ -0,0 +1,457 @@
+// Mgmt
+// Copyright (C) 2013-2016+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+// DOCS: https://godoc.org/github.com/coreos/go-systemd/dbus
+
+package main
+
+import (
+	"encoding/gob"
+	"errors"
+	"fmt"
+	systemd "github.com/coreos/go-systemd/dbus" // change namespace
+	systemdUtil "github.com/coreos/go-systemd/util"
+	"github.com/godbus/dbus" // namespace collides with systemd wrapper
+	"log"
+)
+
+func init() {
+	gob.Register(&SvcRes{})
+}
+
+// SvcRes is a service resource for systemd units.
+type SvcRes struct {
+	BaseRes `yaml:",inline"`
+	State   string `yaml:"state"`   // state: running, stopped, undefined
+	Startup string `yaml:"startup"` // enabled, disabled, undefined
+}
+
+// NewSvcRes is a constructor for this resource. It also calls Init() for you.
+func NewSvcRes(name, state, startup string) *SvcRes {
+	obj := &SvcRes{
+		BaseRes: BaseRes{
+			Name: name,
+		},
+		State:   state,
+		Startup: startup,
+	}
+	obj.Init()
+	return obj
+}
+
+// Init runs some startup code for this resource.
+func (obj *SvcRes) Init() {
+	obj.BaseRes.kind = "Svc"
+	obj.BaseRes.Init() // call base init, b/c we're overriding
+}
+
+// Validate checks if the resource data structure was populated correctly.
+func (obj *SvcRes) Validate() bool {
+	if obj.State != "running" && obj.State != "stopped" && obj.State != "" {
+		return false
+	}
+	if obj.Startup != "enabled" && obj.Startup != "disabled" && obj.Startup != "" {
+		return false
+	}
+	return true
+}
+
+// Watch is the primary listener for this resource and it outputs events.
+func (obj *SvcRes) Watch(processChan chan Event) {
+	if obj.IsWatching() {
+		return
+	}
+	obj.SetWatching(true)
+	defer obj.SetWatching(false)
+	cuuid := obj.converger.Register()
+	defer cuuid.Unregister()
+
+	// obj.Name: svc name
+	if !systemdUtil.IsRunningSystemd() {
+		log.Fatal("Systemd is not running.")
+	}
+
+	conn, err := systemd.NewSystemdConnection() // needs root access
+	if err != nil {
+		log.Fatal("Failed to connect to systemd: ", err)
+	}
+	defer conn.Close()
+
+	// if we share the bus with others, we will get each others messages!!
+	bus, err := SystemBusPrivateUsable() // don't share the bus connection!
+	if err != nil {
+		log.Fatal("Failed to connect to bus: ", err)
+	}
+
+	// XXX: will this detect new units?
+	bus.BusObject().Call("org.freedesktop.DBus.AddMatch", 0,
+		"type='signal',interface='org.freedesktop.systemd1.Manager',member='Reloading'")
+	buschan := make(chan *dbus.Signal, 10)
+	bus.Signal(buschan)
+
+	var svc = fmt.Sprintf("%v.service", obj.Name) // systemd name
+	var send = false                              // send event?
+	var exit = false
+	var dirty = false
+	var invalid = false              // does the svc exist or not?
+	var previous bool                // previous invalid value
+	set := conn.NewSubscriptionSet() // no error should be returned
+	subChannel, subErrors := set.Subscribe()
+	var activeSet = false
+
+	for {
+		// XXX: watch for an event for new units...
+		// XXX: detect if startup enabled/disabled value changes...
+
+		previous = invalid
+		invalid = false
+
+		// firstly, does svc even exist or not?
+		loadstate, err := conn.GetUnitProperty(svc, "LoadState")
+		if err != nil {
+			log.Printf("Failed to get property: %v", err)
+			invalid = true
+		}
+
+		if !invalid {
+			var notFound = (loadstate.Value == dbus.MakeVariant("not-found"))
+			if notFound { // XXX: in the loop we'll handle changes better...
+				log.Printf("Failed to find svc: %v", svc)
+				invalid = true // XXX ?
+			}
+		}
+
+		if previous != invalid { // if invalid changed, send signal
+			send = true
+			dirty = true
+		}
+
+		if invalid {
+			log.Printf("Waiting for: %v", svc) // waiting for svc to appear...
+			if activeSet {
+				activeSet = false
+				set.Remove(svc) // no return value should ever occur
+			}
+
+			obj.SetState(resStateWatching) // reset
+			select {
+			case <-buschan: // XXX wait for new units event to unstick
+				cuuid.SetConverged(false)
+				// loop so that we can see the changed invalid signal
+				log.Printf("Svc[%v]->DaemonReload()", svc)
+
+			case event := <-obj.events:
+				cuuid.SetConverged(false)
+				if exit, send = obj.ReadEvent(&event); exit {
+					return // exit
+				}
+				if event.GetActivity() {
+					dirty = true
+				}
+
+			case <-cuuid.ConvergedTimer():
+				cuuid.SetConverged(true) // converged!
+				continue
+			}
+		} else {
+			if !activeSet {
+				activeSet = true
+				set.Add(svc) // no return value should ever occur
+			}
+
+			log.Printf("Watching: %v", svc) // attempting to watch...
+			obj.SetState(resStateWatching)  // reset
+			select {
+			case event := <-subChannel:
+
+				log.Printf("Svc event: %+v", event)
+				// NOTE: the value returned is a map for some reason...
+				if event[svc] != nil {
+					// event[svc].ActiveState is not nil
+					if event[svc].ActiveState == "active" {
+						log.Printf("Svc[%v]->Started()", svc)
+					} else if event[svc].ActiveState == "inactive" {
+						log.Printf("Svc[%v]->Stopped!()", svc)
+					} else {
+						log.Fatal("Unknown svc state: ", event[svc].ActiveState)
+					}
+				} else {
+					// svc stopped (and ActiveState is nil...)
+					log.Printf("Svc[%v]->Stopped", svc)
+				}
+				send = true
+				dirty = true
+
+			case err := <-subErrors:
+				cuuid.SetConverged(false) // XXX ?
+				log.Printf("error: %v", err)
+				log.Fatal(err)
+				//vertex.events <- fmt.Sprintf("svc: %v", "error") // XXX: how should we handle errors?
+
+			case event := <-obj.events:
+				cuuid.SetConverged(false)
+				if exit, send = obj.ReadEvent(&event); exit {
+					return // exit
+				}
+				if event.GetActivity() {
+					dirty = true
+				}
+
+			case <-cuuid.ConvergedTimer():
+				cuuid.SetConverged(true) // converged!
+				continue
+			}
+		}
+
+		if send {
+			send = false
+			if dirty {
+				dirty = false
+				obj.isStateOK = false // something made state dirty
+			}
+			resp := NewResp()
+			processChan <- Event{eventNil, resp, "", true} // trigger process
+			resp.ACKWait()                                 // wait for the ACK()
+		}
+
+	}
+}
+
+// CheckApply checks the resource state and applies the resource if the bool
+// input is true. It returns error info and if the state check passed or not.
+func (obj *SvcRes) CheckApply(apply bool) (checkok bool, err error) {
+	log.Printf("%v[%v]: CheckApply(%t)", obj.Kind(), obj.GetName(), apply)
+
+	if obj.isStateOK { // cache the state
+		return true, nil
+	}
+
+	if !systemdUtil.IsRunningSystemd() {
+		return false, errors.New("Systemd is not running.")
+	}
+
+	conn, err := systemd.NewSystemdConnection() // needs root access
+	if err != nil {
+		return false, fmt.Errorf("Failed to connect to systemd: %v", err)
+	}
+	defer conn.Close()
+
+	var svc = fmt.Sprintf("%v.service", obj.Name) // systemd name
+
+	loadstate, err := conn.GetUnitProperty(svc, "LoadState")
+	if err != nil {
+		return false, fmt.Errorf("Failed to get load state: %v", err)
+	}
+
+	// NOTE: we have to compare variants with other variants, they are really strings...
+	var notFound = (loadstate.Value == dbus.MakeVariant("not-found"))
+	if notFound {
+		return false, fmt.Errorf("Failed to find svc: %v", svc)
+	}
+
+	// XXX: check svc "enabled at boot" or not status...
+
+	//conn.GetUnitProperties(svc)
+	activestate, err := conn.GetUnitProperty(svc, "ActiveState")
+	if err != nil {
+		return false, fmt.Errorf("Failed to get active state: %v", err)
+	}
+
+	var running = (activestate.Value == dbus.MakeVariant("active"))
+	var stateOK = ((obj.State == "") || (obj.State == "running" && running) || (obj.State == "stopped" && !running))
+	var startupOK = true // XXX DETECT AND SET
+
+	if stateOK && startupOK {
+		return true, nil // we are in the correct state
+	}
+
+	// state is not okay, no work done, exit, but without error
+	if !apply {
+		return false, nil
+	}
+
+	// apply portion
+	log.Printf("%v[%v]: Apply", obj.Kind(), obj.GetName())
+	var files = []string{svc} // the svc represented in a list
+	if obj.Startup == "enabled" {
+		_, _, err = conn.EnableUnitFiles(files, false, true)
+
+	} else if obj.Startup == "disabled" {
+		_, err = conn.DisableUnitFiles(files, false)
+	}
+
+	if err != nil {
+		return false, fmt.Errorf("Unable to change startup status: %v", err)
+	}
+
+	// XXX: do we need to use a buffered channel here?
+	result := make(chan string, 1) // catch result information
+
+	if obj.State == "running" {
+		_, err = conn.StartUnit(svc, "fail", result)
+		if err != nil {
+			return false, fmt.Errorf("Failed to start unit: %v", err)
+		}
+	} else if obj.State == "stopped" {
+		_, err = conn.StopUnit(svc, "fail", result)
+		if err != nil {
+			return false, fmt.Errorf("Failed to stop unit: %v", err)
+		}
+	}
+
+	status := <-result
+	if &status == nil {
+		return false, errors.New("Systemd service action result is nil")
+	}
+	if status != "done" {
+		return false, fmt.Errorf("Unknown systemd return string: %v", status)
+	}
+
+	// XXX: also set enabled on boot
+
+	return false, nil // success
+}
+
+// SvcUUID is the UUID struct for SvcRes.
+type SvcUUID struct {
+	// NOTE: there is also a name variable in the BaseUUID struct, this is
+	// information about where this UUID came from, and is unrelated to the
+	// information about the resource we're matching. That data which is
+	// used in the IFF function, is what you see in the struct fields here.
+	BaseUUID
+	name string // the svc name
+}
+
+// if and only if they are equivalent, return true
+// if they are not equivalent, return false
+func (obj *SvcUUID) IFF(uuid ResUUID) bool {
+	res, ok := uuid.(*SvcUUID)
+	if !ok {
+		return false
+	}
+	return obj.name == res.name
+}
+
+// SvcResAutoEdges holds the state of the auto edge generator.
+type SvcResAutoEdges struct {
+	data    []ResUUID
+	pointer int
+	found   bool
+}
+
+// Next returns the next automatic edge.
+func (obj *SvcResAutoEdges) Next() []ResUUID {
+	if obj.found {
+		log.Fatal("Shouldn't be called anymore!")
+	}
+	if len(obj.data) == 0 { // check length for rare scenarios
+		return nil
+	}
+	value := obj.data[obj.pointer]
+	obj.pointer++
+	return []ResUUID{value} // we return one, even though api supports N
+}
+
+// Test gets results of the earlier Next() call, & returns if we should continue!
+func (obj *SvcResAutoEdges) Test(input []bool) bool {
+	// if there aren't any more remaining
+	if len(obj.data) <= obj.pointer {
+		return false
+	}
+	if obj.found { // already found, done!
+		return false
+	}
+	if len(input) != 1 { // in case we get given bad data
+		log.Fatal("Expecting a single value!")
+	}
+	if input[0] { // if a match is found, we're done!
+		obj.found = true // no more to find!
+		return false
+	}
+	return true // keep going
+}
+
+// The AutoEdges method returns the AutoEdges. In this case the systemd units.
+func (obj *SvcRes) AutoEdges() AutoEdge {
+	var data []ResUUID
+	svcFiles := []string{
+		fmt.Sprintf("/etc/systemd/system/%s.service", obj.Name),     // takes precedence
+		fmt.Sprintf("/usr/lib/systemd/system/%s.service", obj.Name), // pkg default
+	}
+	for _, x := range svcFiles {
+		var reversed = true
+		data = append(data, &FileUUID{
+			BaseUUID: BaseUUID{
+				name:     obj.GetName(),
+				kind:     obj.Kind(),
+				reversed: &reversed,
+			},
+			path: x, // what matters
+		})
+	}
+	return &FileResAutoEdges{
+		data:    data,
+		pointer: 0,
+		found:   false,
+	}
+}
+
+// GetUUIDs includes all params to make a unique identification of this object.
+// Most resources only return one, although some resources can return multiple.
+func (obj *SvcRes) GetUUIDs() []ResUUID {
+	x := &SvcUUID{
+		BaseUUID: BaseUUID{name: obj.GetName(), kind: obj.Kind()},
+		name:     obj.Name, // svc name
+	}
+	return []ResUUID{x}
+}
+
+// GroupCmp returns whether two resources can be grouped together or not.
+func (obj *SvcRes) GroupCmp(r Res) bool {
+	_, ok := r.(*SvcRes)
+	if !ok {
+		return false
+	}
+	// TODO: depending on if the systemd service api allows batching, we
+	// might be able to build this, although not sure how useful it is...
+	// it might just eliminate parallelism be bunching up the graph
+	return false // not possible atm
+}
+
+// Compare two resources and return if they are equivalent.
+func (obj *SvcRes) Compare(res Res) bool {
+	switch res.(type) {
+	case *SvcRes:
+		res := res.(*SvcRes)
+		if !obj.BaseRes.Compare(res) { // call base Compare
+			return false
+		}
+
+		if obj.Name != res.Name {
+			return false
+		}
+		if obj.State != res.State {
+			return false
+		}
+		if obj.Startup != res.Startup {
+			return false
+		}
+	default:
+		return false
+	}
+	return true
+}

test.sh

@@ -1,34 +1,48 @@
 #!/bin/bash -e
-
 # test suite...
 echo running test.sh
 echo "ENV:"
 env
 
+failures=''
+function run-test()
+{
+	$@ || failures=$( [ -n "$failures" ] && echo "$failures\\n$@" || echo "$@" )
+}
+
 # ensure there is no trailing whitespace or other whitespace errors
-git diff-tree --check $(git hash-object -t tree /dev/null) HEAD
+run-test git diff-tree --check $(git hash-object -t tree /dev/null) HEAD
 
 # ensure entries to authors file are sorted
 start=$(($(grep -n '^[[:space:]]*$' AUTHORS | awk -F ':' '{print $1}' | head -1) + 1))
-diff <(tail -n +$start AUTHORS | sort) <(tail -n +$start AUTHORS)
+run-test diff <(tail -n +$start AUTHORS | sort) <(tail -n +$start AUTHORS)
 
-./test/test-gofmt.sh
+run-test ./test/test-gofmt.sh
-./test/test-yamlfmt.sh
+run-test ./test/test-yamlfmt.sh
-./test/test-bashfmt.sh
+run-test ./test/test-bashfmt.sh
-go test
+run-test ./test/test-headerfmt.sh
-echo running go vet	# since it doesn't output an ok message on pass
+run-test go test
-go vet && echo PASS
+run-test ./test/test-govet.sh
 
 # do these longer tests only when running on ci
 if env | grep -q -e '^TRAVIS=true$' -e '^JENKINS_URL=' -e '^BUILD_TAG=jenkins'; then
-	go test -race
+	run-test go test -race
-	./test/test-shell.sh
+	run-test ./test/test-shell.sh
-else
-	# FIXME: this fails on travis for some reason
-	./test/test-reproducible.sh
 fi
 
+# FIXME: this now fails everywhere :(
+#run-test ./test/test-reproducible.sh
+
 # run omv tests on jenkins physical hosts only
 if env | grep -q -e '^JENKINS_URL=' -e '^BUILD_TAG=jenkins'; then
-	./test/test-omv.sh
+	run-test ./test/test-omv.sh
 fi
+run-test ./test/test-golint.sh	# test last, because this test is somewhat arbitrary
+
+if [[ -n "$failures" ]]; then
+	echo 'FAIL'
+	echo 'The following tests have failed:'
+	echo -e "$failures"
+	exit 1
+fi
+echo 'ALL PASSED'

test/omv/helloworld.yaml

@@ -1,7 +1,7 @@
 ---
 :domain: example.com
 :network: 192.168.123.0/24
-:image: fedora-23
+:image: centos-7.1
 :cpus: ''
 :memory: ''
 :disks: 0
@@ -27,12 +27,11 @@
 :ansible_extras: {}
 :cachier: false
 :vms:
-- :name: etcd
+- :name: mgmt0
   :shell:
   - iptables -F
   - cd /vagrant/mgmt/ && make path
   - cd /vagrant/mgmt/ && make deps && make build && cp mgmt ~/bin/
-  - etcd -bind-addr "`hostname --ip-address`:2379" &
   - cd && mgmt --help
 :namespace: omv
 :count: 0
@@ -45,8 +44,7 @@
 :unsafe: false
 :nested: false
 :tests:
-- omv up etcd
+- omv up mgmt0
-- vssh root@etcd -c pidof etcd
 - omv destroy
 :comment: simple hello world test case for mgmt
 :reallyrm: false

test/omv/pkg1a.yaml

@@ -0,0 +1,51 @@
+---
+:domain: example.com
+:network: 192.168.123.0/24
+:image: centos-7.1
+:cpus: ''
+:memory: ''
+:disks: 0
+:disksize: 40G
+:boxurlprefix: ''
+:sync: rsync
+:syncdir: ''
+:syncsrc: ''
+:folder: ".omv"
+:extern:
+- type: git
+  repository: https://github.com/purpleidea/mgmt
+  directory: mgmt
+:cd: ''
+:puppet: false
+:classes: []
+:shell:
+- mkdir /tmp/mgmt/
+:docker: false
+:kubernetes: false
+:ansible: []
+:playbook: []
+:ansible_extras: {}
+:cachier: false
+:vms:
+- :name: mgmt1
+  :shell:
+  - iptables -F
+  - cd /vagrant/mgmt/ && make path
+  - cd /vagrant/mgmt/ && make deps && make build && cp mgmt ~/bin/
+  - cd && mgmt run --file /vagrant/mgmt/examples/pkg1.yaml --converged-timeout=5
+:namespace: omv
+:count: 0
+:username: ''
+:password: ''
+:poolid: true
+:repos: []
+:update: false
+:reboot: false
+:unsafe: false
+:nested: false
+:tests:
+- omv up
+- vssh root@mgmt1 -c which powertop
+- omv destroy
+:comment: simple package install test case
+:reallyrm: false

test/omv/pkg1b.yaml

@@ -0,0 +1,52 @@
+---
+:domain: example.com
+:network: 192.168.123.0/24
+:image: debian-8
+:cpus: ''
+:memory: ''
+:disks: 0
+:disksize: 40G
+:boxurlprefix: ''
+:sync: rsync
+:syncdir: ''
+:syncsrc: ''
+:folder: ".omv"
+:extern:
+- type: git
+  repository: https://github.com/purpleidea/mgmt
+  directory: mgmt
+:cd: ''
+:puppet: false
+:classes: []
+:shell:
+- mkdir /tmp/mgmt/
+:docker: false
+:kubernetes: false
+:ansible: []
+:playbook: []
+:ansible_extras: {}
+:cachier: false
+:vms:
+- :name: mgmt1
+  :shell:
+  - apt-get install -y make
+  - iptables -F
+  - cd /vagrant/mgmt/ && make path
+  - cd /vagrant/mgmt/ && make deps && make build && cp mgmt ~/bin/
+  - cd && mgmt run --file /vagrant/mgmt/examples/pkg1.yaml --converged-timeout=5
+:namespace: omv
+:count: 0
+:username: ''
+:password: ''
+:poolid: true
+:repos: []
+:update: false
+:reboot: false
+:unsafe: false
+:nested: false
+:tests:
+- omv up
+- vssh root@mgmt1 -c which powertop
+- omv destroy
+:comment: simple package install test case
+:reallyrm: false

test/shell/etcd.sh

@@ -1,12 +0,0 @@
-# NOTE: boiler plate to run etcd; source with: . etcd.sh; should NOT be +x
-cleanup ()
-{
-	killall etcd || killall -9 etcd || true	# kill etcd
-	rm -rf /tmp/etcd/
-}
-trap cleanup INT QUIT TERM EXIT ERR
-mkdir -p /tmp/etcd/
-cd /tmp/etcd/ >/dev/null	# shush the cd operation
-etcd &	# start etcd as job # 1
-sleep 1s	# let etcd startup
-cd - >/dev/null

test/shell/t1.sh

@@ -1,14 +1,15 @@
-#!/bin/bash
+#!/bin/bash -e
 # NOTES:
 #	* this is a simple shell based `mgmt` test case
 #	* it is recommended that you run mgmt wrapped in the timeout command
 #	* it is recommended that you run mgmt with --no-watch
 #	* it is recommended that you run mgmt --converged-timeout=<seconds>
 #	* you can run mgmt with --max-runtime=<seconds> in special scenarios
-#	* you can get a separate etcd going by sourcing etcd.sh: . etcd.sh
 
 set -o errexit
-set -o nounset
 set -o pipefail
 
 timeout --kill-after=3s 1s ./mgmt --help # hello world!
+pid=$!
+wait $pid	# get exit status
+exit $?

test/shell/t2.sh

@@ -1,14 +1,20 @@
-#!/bin/bash
+#!/bin/bash -e
 
-. etcd.sh	# start etcd as job # 1
+if env | grep -q -e '^TRAVIS=true$'; then
+	# inotify doesn't seem to work properly on travis
+	echo "Travis and Jenkins give wonky results here, skipping test!"
+	exit
+fi
 
 # run till completion
-timeout --kill-after=15s 10s ./mgmt run --file t2.yaml --converged-timeout=5 --no-watch &
+timeout --kill-after=15s 10s ./mgmt run --file t2.yaml --converged-timeout=5 --no-watch --tmp-prefix &
-
+pid=$!
-#jobs	# etcd is 1
+wait $pid	# get exit status
-wait -n 2	# wait for mgmt to exit
+e=$?
 
 test -e /tmp/mgmt/f1
 test -e /tmp/mgmt/f2
 test -e /tmp/mgmt/f3
 test ! -e /tmp/mgmt/f4
+
+exit $e

test/shell/t2.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   noop:
   - name: noop1
   file:
@@ -27,15 +27,15 @@ types:
 edges:
 - name: e1
   from:
-    type: file
+    kind: file
     name: file1
   to:
-    type: file
+    kind: file
     name: file2
 - name: e2
   from:
-    type: file
+    kind: file
     name: file2
   to:
-    type: file
+    kind: file
     name: file3

test/shell/t3-a.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   file:
   - name: file1a
     path: "/tmp/mgmt/mgmtA/f1a"
@@ -23,6 +23,6 @@ types:
       i am f4, exported from host A
     state: exists
 collect:
-- type: file
+- kind: file
   pattern: "/tmp/mgmt/mgmtA/"
 edges: []

test/shell/t3-b.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   file:
   - name: file1b
     path: "/tmp/mgmt/mgmtB/f1b"
@@ -23,6 +23,6 @@ types:
       i am f4, exported from host B
     state: exists
 collect:
-- type: file
+- kind: file
   pattern: "/tmp/mgmt/mgmtB/"
 edges: []

test/shell/t3-c.yaml

@@ -1,6 +1,6 @@
 ---
 graph: mygraph
-types:
+resources:
   file:
   - name: file1c
     path: "/tmp/mgmt/mgmtC/f1c"
@@ -23,6 +23,6 @@ types:
       i am f4, exported from host C
     state: exists
 collect:
-- type: file
+- kind: file
   pattern: "/tmp/mgmt/mgmtC/"
 edges: []

test/shell/t3.sh

@@ -1,16 +1,28 @@
-#!/bin/bash
+#!/bin/bash -e
 
-. etcd.sh	# start etcd as job # 1
+if env | grep -q -e '^TRAVIS=true$'; then
+	# inotify doesn't seem to work properly on travis
+	echo "Travis and Jenkins give wonky results here, skipping test!"
+	exit
+fi
 
 # setup
 mkdir -p "${MGMT_TMPDIR}"mgmt{A..C}
 
 # run till completion
-timeout --kill-after=15s 10s ./mgmt run --file t3-a.yaml --converged-timeout=5 --no-watch &
+timeout --kill-after=15s 10s ./mgmt run --file t3-a.yaml --converged-timeout=5 --no-watch --tmp-prefix &
-timeout --kill-after=15s 10s ./mgmt run --file t3-b.yaml --converged-timeout=5 --no-watch &
+pid1=$!
-timeout --kill-after=15s 10s ./mgmt run --file t3-c.yaml --converged-timeout=5 --no-watch &
+timeout --kill-after=15s 10s ./mgmt run --file t3-b.yaml --converged-timeout=5 --no-watch --tmp-prefix &
+pid2=$!
+timeout --kill-after=15s 10s ./mgmt run --file t3-c.yaml --converged-timeout=5 --no-watch --tmp-prefix &
+pid3=$!
 
-. wait.sh	# wait for everything except etcd
+wait $pid1	# get exit status
+e1=$?
+wait $pid2	# get exit status
+e2=$?
+wait $pid3	# get exit status
+e3=$?
 
 # A: collected
 test -e "${MGMT_TMPDIR}"mgmtA/f3b
@@ -65,3 +77,5 @@ test ! -e "${MGMT_TMPDIR}"mgmtC/f1a
 test ! -e "${MGMT_TMPDIR}"mgmtC/f2a
 test ! -e "${MGMT_TMPDIR}"mgmtC/f1b
 test ! -e "${MGMT_TMPDIR}"mgmtC/f2b
+
+exit $(($e1+$e2+$e3))

test/shell/t4.sh

@@ -1,10 +1,7 @@
-#!/bin/bash
+#!/bin/bash -e
-
-. etcd.sh	# start etcd as job # 1
 
 # should take slightly more than 25s, but fail if we take 35s)
-timeout --kill-after=35s 30s ./mgmt run --file t4.yaml --converged-timeout=5 --no-watch &
+timeout --kill-after=35s 30s ./mgmt run --file t4.yaml --converged-timeout=5 --no-watch --tmp-prefix &
-
+pid=$!
-#jobs	# etcd is 1
+wait $pid	# get exit status
-#wait -n 2	# wait for mgmt to exit
+exit $?
-. wait.sh	# wait for everything except etcd

test/shell/t4.yaml

@@ -1,7 +1,7 @@
 ---
 graph: mygraph
 comment: simple exec fan in example to demonstrate optimization)
-types:
+resources:
   exec:
   - name: exec1
     cmd: sleep 10s
@@ -56,22 +56,22 @@ types:
 edges:
 - name: e1
   from:
-    type: exec
+    kind: exec
     name: exec1
   to:
-    type: exec
+    kind: exec
     name: exec5
 - name: e2
   from:
-    type: exec
+    kind: exec
     name: exec2
   to:
-    type: exec
+    kind: exec
     name: exec5
 - name: e3
   from:
-    type: exec
+    kind: exec
     name: exec3
   to:
-    type: exec
+    kind: exec
     name: exec5

test/shell/t5.sh

@@ -1,10 +1,7 @@
-#!/bin/bash
+#!/bin/bash -e
-
-. etcd.sh	# start etcd as job # 1
 
 # should take slightly more than 35s, but fail if we take 45s)
-timeout --kill-after=45s 40s ./mgmt run --file t5.yaml --converged-timeout=5 --no-watch &
+timeout --kill-after=45s 40s ./mgmt run --file t5.yaml --converged-timeout=5 --no-watch --tmp-prefix &
-
+pid=$!
-#jobs	# etcd is 1
+wait $pid	# get exit status
-#wait -n 2	# wait for mgmt to exit
+exit $?
-. wait.sh	# wait for everything except etcd

test/shell/t5.yaml

@@ -1,7 +1,7 @@
 ---
 graph: mygraph
 comment: simple exec fan in to fan out example to demonstrate optimization
-types:
+resources:
   exec:
   - name: exec1
     cmd: sleep 10s
@@ -86,43 +86,43 @@ types:
 edges:
 - name: e1
   from:
-    type: exec
+    kind: exec
     name: exec1
   to:
-    type: exec
+    kind: exec
     name: exec4
 - name: e2
   from:
-    type: exec
+    kind: exec
     name: exec2
   to:
-    type: exec
+    kind: exec
     name: exec4
 - name: e3
   from:
-    type: exec
+    kind: exec
     name: exec3
   to:
-    type: exec
+    kind: exec
     name: exec4
 - name: e4
   from:
-    type: exec
+    kind: exec
     name: exec4
   to:
-    type: exec
+    kind: exec
     name: exec5
 - name: e5
   from:
-    type: exec
+    kind: exec
     name: exec4
   to:
-    type: exec
+    kind: exec
     name: exec6
 - name: e6
   from:
-    type: exec
+    kind: exec
     name: exec4
   to:
-    type: exec
+    kind: exec
     name: exec7

test/shell/t6.sh

@@ -0,0 +1,33 @@
+#!/bin/bash -e
+
+if env | grep -q -e '^TRAVIS=true$'; then
+	# inotify doesn't seem to work properly on travis
+	echo "Travis and Jenkins give wonky results here, skipping test!"
+	exit
+fi
+
+# run till completion
+timeout --kill-after=20s 15s ./mgmt run --file t6.yaml --no-watch --tmp-prefix &
+pid=$!
+sleep 1s	# let it converge
+test -e /tmp/mgmt/f1
+test -e /tmp/mgmt/f2
+test -e /tmp/mgmt/f3
+test ! -e /tmp/mgmt/f4
+rm -f /tmp/mgmt/f2
+sleep 0.1s	# let it converge or tests will fail
+test -e /tmp/mgmt/f2
+rm -f /tmp/mgmt/f2
+sleep 0.1s
+test -e /tmp/mgmt/f2
+echo foo > /tmp/mgmt/f2
+sleep 0.1s
+test "`cat /tmp/mgmt/f2`" = "i am f2"
+rm -f /tmp/mgmt/f2
+sleep 0.1s
+test -e /tmp/mgmt/f2
+
+killall -SIGINT mgmt	# send ^C to exit mgmt
+
+wait $pid	# get exit status
+exit $?

test/shell/t6.yaml

@@ -0,0 +1,41 @@
+---
+graph: mygraph
+resources:
+  noop:
+  - name: noop1
+  file:
+  - name: file1
+    path: "/tmp/mgmt/f1"
+    content: |
+      i am f1
+    state: exists
+  - name: file2
+    path: "/tmp/mgmt/f2"
+    content: |
+      i am f2
+    state: exists
+  - name: file3
+    path: "/tmp/mgmt/f3"
+    content: |
+      i am f3
+    state: exists
+  - name: file4
+    path: "/tmp/mgmt/f4"
+    content: |
+      i am f4 and i should not be here
+    state: absent
+edges:
+- name: e1
+  from:
+    kind: file
+    name: file1
+  to:
+    kind: file
+    name: file2
+- name: e2
+  from:
+    kind: file
+    name: file2
+  to:
+    kind: file
+    name: file3