Commit Graph

748 Commits

Author SHA1 Message Date
Toshaan Bharvani
2fdf8d5dc3 lang: Interface sorting order
as golang does not loop over the same map/list always the same
we use a helper list to sort it

Signed-off-by: Toshaan Bharvani <toshaan@vantosh.com>
2018-02-18 18:32:15 -05:00
James Shubin
28ec7a1e54 etcd: scheduler: Remove etcd 3.2 specific hacks
Now that we're using etcd 3.3, we can simplify our code now that our
patches are in a release.
2018-02-18 18:28:45 -05:00
James Shubin
24cb2e6450 etcd: Increase the default max txn op count
The default of 128 is fairly low for large code bases. Please let us
know if you hit the new limit of 512.
2018-02-18 18:28:00 -05:00
James Shubin
915b022901 test: Show test output as it happens 2018-02-18 17:31:45 -05:00
James Shubin
4a623c1891 docs: Add an entry to the faq about converged timeouts 2018-02-18 16:28:50 -05:00
Johan Bloemberg
7508161c39 test: Exclude generated files from golint 2018-02-18 15:07:54 -05:00
Johan Bloemberg
d33861ccb4 test: Fix augeas test for macOS, improve test debuggability
- resolve a discrepancy in augeas behaviour on macOS
- on macOS `sed` requires an argument for `-i`.
- made the test fail as early as it can
- provide information about why the test is failing
2018-02-18 14:36:59 -05:00
Johan Bloemberg
572b2575c5 test: Export the mgmt command to be used during test 2018-02-18 14:19:05 -05:00
James Shubin
d99190b166 travis: Add golang 1.10.x to builds 2018-02-18 13:58:11 -05:00
James Shubin
bc91b03276 docs: Add faq entry about production readiness 2018-02-18 13:34:21 -05:00
James Shubin
9ba893c06c etcd: Bump to etcd v3.3 and golang 1.9
This moves us to etcd v3.3 (a new major release) which has some useful
features but that requires version 1.9 of golang.
2018-02-15 18:47:55 -05:00
James Shubin
27e51f1bcb authors: Clarify wording in AUTHORS file 2018-02-15 18:45:32 -05:00
James Shubin
e3a26483e8 test: Improve gometalinter test so that it skips generated files
This should improve things significantly, and avoid the failures now
that we're testing after the files have already been built.
2018-02-15 16:26:06 -05:00
James Shubin
33a4fd6fbe build: Add -i flag to go build
It got accidentally dropped, but is crucial for happiness.
See: https://purpleidea.com/blog/2017/02/26/faster-golang-builds/
2018-02-15 12:24:56 -05:00
Johan Bloemberg
b34b359860 test: Streamline test suite a little
This change aims to streamline the integrationtest suite and reduce friction when running (parts of) test suites.

Changes:
- add `test-testname` to makefile to easily run one suite
- made skipping tests first class citizen in test.sh (all available testsuites and the reasons they are skipped are now better exposed and discovered)
- suppress some output of gotest unless there is an error
- no longer build binary for examples and gotest suites
- removed .SILENT from makefile as it being applied to only some targets makes it feel weird (I just learned about this option btw, feel free to comment on this change)
- move individual tests out of `test.sh` and into `test-misc.sh`
- introduced the concept of testsuites to `test.sh`
2018-02-15 17:21:49 +01:00
James Shubin
8b9491823d etcd: Fix golint issue in test
Found with new gometalinter version.
2018-02-14 18:53:10 -05:00
James Shubin
b8b6e5266f build: Improve speed of make
Generating a huge amount of unnecessary targets caused "noop" make runs
to take seven seconds on my machine. This limits the list of these
drastically and now "noop" make's are now < 1s on my machine.

Issue discussed in:
https://github.com/purpleidea/mgmt/issues/331
2018-02-14 18:42:12 -05:00
James Shubin
5f80c1ac2a resources: nspawn: Don't panic if one svc is nil
Not sure why one of them was nil, but this prevents the panic.
2018-02-14 16:44:21 -05:00
James Shubin
3af7e815d0 docs: Add newly recorded talks and blog posts about mgmt 2018-02-14 15:45:40 -05:00
James Shubin
714afe35a1 test: Fix broken gometalinter test
The test for gometalinter got silently broken in an earlier commit.
Look for the missing space that was added back in this commit to see
why! In any case, this now fixes some of the things that weren't
previously caught by this change.

If anyone knows how to run these sorts of tests properly so that entire
packages are tested and so that we can enable additional tests, please
let me know!

It's also unclear why goreportcard catches a few additional problems
which aren't found by running this ourselves.

See:
https://goreportcard.com/report/github.com/purpleidea/mgmt
for more information.
2018-02-14 14:34:36 -05:00
James Shubin
b0a8f585c3 readme: Fix broken link 2018-02-14 14:03:07 -05:00
James Shubin
1a2918082d docs: Add FAQ entry about vendoring dependencies 2018-02-14 14:01:10 -05:00
Johan Bloemberg
22e4dfa534 build: Unify build/crossbuild
Changes:

- allows explicit crossbuild targets (eg: `make mgmt-darwin-amd64`)
- adds darwin/amd64 to default crossbuild targets
- gitignore only build artifacts (eg: not all files starting with `mgmt-`)
- `build` and `crossbuild` target now utilize the same build function (`build` still generates only a `mgmt` binary for the current os/arch)
- test crossbuilding
- allow specifying custom GOOSARCHES envvar to override defaults
- crossbuild artifacts go into `build/` now
- add `build-debug` which includes symbol tables and debug info
- the build function now has `-s -w` linker arguments which discards some debug info afaict, to build a debug release use `make build-debug`

On my mac crossbuilding won't work unless I disable augeas and libvirt:

```
~/.g/s/g/p/mgmt (build|●1✚8…3) $ make build
Generating: bindata...
Generating: lang...
/Applications/Xcode.app/Contents/Developer/usr/bin/make --quiet -C lang
Building: mgmt, os/arch: darwin-amd64, version: 0.0.14-12-g94c8bc1-dirty...
env GOOS=darwin GOARCH=amd64 time go build -ldflags "-X main.program=mgmt -X main.version=0.0.14-12-g94c8bc1-dirty -s -w" -o mgmt-darwin-amd64 ;
        7.14 real        10.36 user         1.73 sys
mv mgmt-darwin-amd64 mgmt
```

```
~/.g/s/g/p/mgmt (build|●1✚8…3) $ time env GOTAGS='noaugeas novirt' make crossbuild
Generating: bindata...
Generating: lang...
/Applications/Xcode.app/Contents/Developer/usr/bin/make --quiet -C lang
Building: mgmt, os/arch: linux-amd64, version: 0.0.14-12-g94c8bc1-dirty...
env GOOS=linux GOARCH=amd64 time go build -ldflags "-X main.program=mgmt -X main.version=0.0.14-12-g94c8bc1-dirty -s -w" -o mgmt-linux-amd64 -tags 'noaugeas novirt';
       18.48 real        50.02 user         5.83 sys
Building: mgmt, os/arch: linux-ppc64, version: 0.0.14-12-g94c8bc1-dirty...
env GOOS=linux GOARCH=ppc64 time go build -ldflags "-X main.program=mgmt -X main.version=0.0.14-12-g94c8bc1-dirty -s -w" -o mgmt-linux-ppc64 -tags 'noaugeas novirt';
       29.83 real        85.09 user        11.54 sys
Building: mgmt, os/arch: linux-ppc64le, version: 0.0.14-12-g94c8bc1-dirty...
env GOOS=linux GOARCH=ppc64le time go build -ldflags "-X main.program=mgmt -X main.version=0.0.14-12-g94c8bc1-dirty -s -w" -o mgmt-linux-ppc64le -tags 'noaugeas novirt';
       29.74 real        85.84 user        11.76 sys
Building: mgmt, os/arch: linux-arm64, version: 0.0.14-12-g94c8bc1-dirty...
env GOOS=linux GOARCH=arm64 time go build -ldflags "-X main.program=mgmt -X main.version=0.0.14-12-g94c8bc1-dirty -s -w" -o mgmt-linux-arm64 -tags 'noaugeas novirt';
       28.33 real        83.24 user        11.40 sys
Building: mgmt, os/arch: darwin-amd64, version: 0.0.14-12-g94c8bc1-dirty...
env GOOS=darwin GOARCH=amd64 time go build -ldflags "-X main.program=mgmt -X main.version=0.0.14-12-g94c8bc1-dirty -s -w" -o mgmt-darwin-amd64 -tags 'noaugeas novirt';
        7.16 real        10.15 user         1.74 sys
      114.71 real       315.26 user        42.44 sys
```
2018-02-14 12:37:49 -05:00
Johan Bloemberg
41eb850b3d debian: Add graphviz and packagekit runtime dependencies 2018-02-12 15:11:13 -05:00
Wim
3a50171d19 build: Add gcc,pkg-config deps 2018-02-12 15:10:19 -05:00
Wim
6c9e0ff974 docs: Add GOPATH/bin to PATH 2018-02-12 15:09:32 -05:00
James Shubin
644e5164b1 test: Increase timeouts for when travis is slow
Hopefully this cuts down on spurious failures.
2018-02-12 15:08:47 -05:00
James Shubin
4fefa9f2f0 travis: Disable fast finish for now
This causes a notification for each entry in the matrix which is now too
many emails. When travis adds and option to send just one notification,
but to still allow you to fast finish, then please lmk :)
2018-02-10 18:49:12 -05:00
Johan Bloemberg
4c793e0ee6 misc: Fix graphviz output for hostnames with dot in them 2018-02-11 00:02:52 +01:00
James Shubin
68a7de41ae etcd: Update broken link 2018-02-10 10:38:54 -05:00
dsx
94c8bc1de9 debian: Add packaging 2018-02-10 05:12:31 -05:00
Johan Bloemberg
8fb0373f82 resources: Do not return GID for UID lookup
On linux it is convention for users to have a group with the same GID as the users UID. On macOS this is not the case. This broke the test which lead to discovering this bug.
2018-02-10 05:01:12 -05:00
Johan Bloemberg
d567dc3769 lang: Use universal way to retrieve load
Sysinfo is not supported on macOS and results in a build error.
2018-02-10 05:01:12 -05:00
Johan Bloemberg
ba21554c5f build, docs: Improve macOS building
- New docker command for quickly running tasks in a Linux environment.
- Updated docs with macOS specific details.
- Fixed some test issues.
- Add (fallible) macOS test target for Travis.
2018-02-10 05:01:12 -05:00
James Shubin
e37bb3ac8a test: Add new test for language prefix 2018-02-07 21:52:06 -05:00
Carsten Thiel
79845f0dfd test: Refactor unification_test to subtests
Testsuite for unification now uses subtests feature.
2018-02-07 16:13:18 +01:00
Carsten Thiel
eb33a5a5df docs: Improve file resource documentation
Info on how to create a directory.
Explain more parameter options.
2018-02-07 14:19:14 +01:00
jonathangold
adbe9c7be1 misc: Replace missing go-bindata dependency 2018-02-07 06:15:03 -05:00
James Shubin
b19583e7d3 lang: Initial implementation of the mgmt language
This is an initial implementation of the mgmt language. It is a
declarative (immutable) functional, reactive, domain specific
programming language. It is intended to be a language that is:

* safe
* powerful
* easy to reason about

With these properties, we hope this language, and the mgmt engine will
allow you to model the real-time systems that you'd like to automate.

This also includes a number of other associated changes. Sorry for the
large size of this patch.
2018-01-20 08:09:29 -05:00
Joe Julian
1c8c0b2915 misc: Don't install packages that are already installed
pacman needs `--needed` to prevent reinstalling packages that are
already installed. Additionally I added `--asdeps` to allow later
cleanup of unneeded dependencies.
2018-02-03 17:17:23 -05:00
Joe Julian
bee1aa00f1 misc: Use bash's command instead of which
Bash has a built-in command, `command`, that will search the path and
return the full path to a command if it exists (or an exit code of 1 if
it does not), preventing the requirement of the `which` package.
2018-02-03 11:23:52 -05:00
Toshaan Bharvani
077b6e540a build: Add cross building option
Added a cross build option using a buildrelease function

Signed-off-by: Toshaan Bharvani <toshaan@vantosh.com>

build: Add gitignore entry for mgmt-* binaries

Signed-off-by: Toshaan Bharvani <toshaan@vantosh.com>

build: Update makefile based upon feed back

* rename cross to crossbuild
* added crossbuild to PHONY

Signed-off-by: Toshaan Bharvani <toshaan@vantosh.com>

build: Change the order of .PHONY as per the rest of the file

Signed-off-by: Toshaan Bharvani <toshaan@vantosh.com>
2018-02-03 16:43:40 +01:00
James Shubin
e8b03545bb test: Don't fail on tag builds
This seems to be causing our failures with:

$ git fetch --unshallow
fatal: Couldn't find remote ref refs/heads/0.0.x

where x is some tag.

Hopefully this doesn't break the other use case we added this patch for!
0.0.14
2018-01-11 18:03:56 -05:00
James Shubin
70c59eab4a misc: Don't display script name in output 2018-01-11 18:03:04 -05:00
jonathangold
3c677543e0 resources: aws: ec2: Fix closed channel handling
If awschan closes, longpollWatch and snsWatch return nil
instead of an error. This will prevent the engine from
shutting down in case we choose to close the channel
early or from other struct methods.
2018-01-06 15:15:30 -05:00
jonathangold
c455ef2c62 resources: aws: ec2: Send IP addresses and InstanceID 2018-01-03 21:34:28 -05:00
Jonathan Gold
032d0992d6 resources: aws: ec2: Refactor CheckApply
CheckApply was rewritten, using the new describe methods to improve
readability and maintainability.
2018-01-03 21:34:28 -05:00
jonathangold
67837a47ac resources: aws: ec2: Refactor longpollWatch
Complete rewrite of longpollWatch() for correctness and maintanability.
2018-01-03 21:34:28 -05:00
Jonathan Gold
32e3c4e029 resources: aws: ec2: Refactor longpollWatch
This patch simplifies longpollwatch by getting rid of some unnecessary
api calls and breaking the waiters out into their own functions.
2018-01-03 21:34:28 -05:00
Jonathan Gold
76fcb7a06e resources: aws: ec2: Wait for stop and terminate concurrently
In longpollWatch it was no longer sufficient to use only
WaitUntilInstanceStopped as it would block if the instance was
terminated. This patch launches two goroutines in its place, one
waits until the instance stops and the other waits until it
terminates. When either one returns, it cancels their context,
and execution continues.
2018-01-03 21:34:28 -05:00