Commit Graph

104 Commits

Author SHA1 Message Date
Jean-Philippe Evrard
01c2131436 misc: Make-deps should assume go vet present
While the code comment says to check if go vet command is present,
it actually tests if go vet command returns properly.

This is a problem if go vet is _not_ returning 0 due to a
failure while go vet is present: it will try to install the
legacy go vet.

This fixes it by removing this block of code completely,
as we assume a golang version which contains it anyway.
2020-12-09 19:25:19 -05:00
James Shubin
76ede10e0a misc: Update path to go-fuzz
They moved it :/
2020-09-23 13:42:51 -04:00
Ahmed Al-Hulaibi
274e01bb75 misc, docs: Update minimum required golang version to 1.13 2020-09-23 12:34:54 -04:00
David Randall
10ce7178c0 misc: Incorrect dependency path for dvyukov/go-fuzz
Fix the dependency path for dvyukov/go-fuzz to github.com from golang.org.

Fixes #601
2020-06-07 15:05:46 -04:00
James Shubin
380d03257f misc, lang: Small fixups
Change some minor style issues.
2020-02-27 17:34:03 -05:00
Patrick Meyer
006de6da14 lang: Add fuzz target to lang Makefile 2020-02-26 12:07:00 +01:00
James Shubin
525e2bafee misc: Let sigtee work on older golang versions 2020-01-25 04:43:33 -05:00
James Shubin
b65a9abf8e misc: Add two scripts to help debug things
This adds two new helper scripts that are good for debugging mgmt. Just
build and add `sigtee` to your ~/bin/ along with filter-golang-stack.py
and mgmt_debug.sh and use the later to call mgmt as you would normally.

For example, I might do:

$ mgmt_debug.sh ./mgmt run --tmp-prefix lang examples/lang/hello0.mcl

And if I kill it with ^\ then I'll get a filtered trace at the end in my
$PAGER (which is assumed to be `less`) and this should make my life
easier.

As a cool bonus, this means we use bash, python, and golang all
together!
2020-01-13 01:07:00 -05:00
James Shubin
2648fb1bb1 legal: Happy 2020 everyone...
Done with:

ack '2019+' -l | xargs sed -i -e 's/2019+/2020+/g'

Checked manually with:

git add -p

Hello to future James from 2021, and Happy Hacking!
2020-01-03 20:08:37 -05:00
James Shubin
7a587ee8d1 misc: mkosi: Switch to copy-git-more
I added a new feature in mkosi which got merged in:

31801e89e77188e697ed937ca6b8668fde4c4a4d

This allows us to pull in all of the git repository so that we can see
the version number that comes from git.
2019-11-02 07:34:04 -04:00
James Shubin
d57f7aa03f misc: Specific mkosi fixes for centos-7
Seems we need golang from epel, to mask out the old git version, and to
workaround mkosi bugs.
2019-10-05 01:49:42 -04:00
Jimmy Tang
d64f9f5401 misc: Add CentOS 7 rpm build 2019-10-04 23:47:12 -04:00
James Shubin
6a7d904fae misc: Improve tagging script
This way we can push the tag *after* all the builds succeed. If
something goes wrong, we can always delete our local tag and try again.
2019-10-04 06:49:04 -04:00
James Shubin
d4043d3f86 misc, make: Add full file path into fpm script
This is needed for our fancier, unique file names.
2019-10-04 06:43:17 -04:00
James Shubin
ffe402f201 misc: Add fedora-30 mkosi+fpm build environment
Good example of how to add a new distro or version.
2019-10-04 06:02:08 -04:00
James Shubin
09cc7da282 misc: Add proper archlinux prefix in build script 2019-10-04 06:01:23 -04:00
James Shubin
b2a495f593 misc: Add mkosi target for ubuntu bionic
The name of these is pretty weird.
2019-09-23 06:50:00 -04:00
James Shubin
65ee904377 misc: Work around old golang in ubuntu
Hopefully this helps.
2019-09-23 06:48:55 -04:00
James Shubin
36d2a0de1e misc: Make mkosi building suitable for different distro versions
We'd like to be able to build both Fedora N and N-1 at the same time if
possible. This makes it more generally applicable for this scenario, as
well as for other distros.
2019-09-23 06:48:55 -04:00
James Shubin
a4db9fc8e5 misc: Add mkosi based package building with fpm
Building distro packages is great, however if they aren't built in the
correct environment with the associated dependencies, then they won't
work properly on those distros.

This patch adds an `mkosi` based image building environment that builds
the packages in their respective distros, and then copies them out into
our releases directory.

You'll now want to `make tag && make mkosi && make release` to get a new
release out. We use a small hack to trick the `make release` portion to
not re-build the distro packages if they're already present in the
releases/ directory for that version.

This commit depends on a very recent version of mkosi (it was tested
with git master) and also depends on two currently unmerged patches:
https://github.com/systemd/mkosi/pull/363 and
https://github.com/systemd/mkosi/pull/365
2019-09-20 12:32:41 -04:00
James Shubin
18872194af misc: Warn users with weird computers
A user seemed to experience a weird golang issue when they had deps from
both package managers installed. I won't block or fail their install,
but we can print a warning message so that someone sees it in their
logs.
2019-08-23 22:10:34 -04:00
James Shubin
bafd7ba282 misc: Use apt install of apt-get where possible
The future is now!
2019-08-23 22:10:12 -04:00
James Shubin
4f1c463bdd misc: Add graphviz deps for travis
Some tests run graphviz.
2019-07-17 00:27:09 -04:00
James Shubin
e9257580cd misc: Update to golang 1.11.x
Bump to the newer version.
2019-05-05 09:32:04 -04:00
James Shubin
a5842a41b2 etcd: Rewrite embed etcd implementation
This is a giant cleanup of the etcd code. The earlier version was
written when I was less experienced with golang.

This is still not perfect, and does contain some races, but at least
it's a decent base to start from. The automatic elastic clustering
should be considered an experimental feature. If you need a more
battle-tested cluster, then you should manage etcd manually and point
mgmt at your existing cluster.
2019-04-11 21:43:48 -04:00
James Shubin
07f542b4d7 legal: Happy 2019 everyone...
Done with:

ack '2018+' -l | xargs sed -i -e 's/2018+/2019+/g'

Checked manually with:

git add -p

Hello to future James from 2020, and Happy Hacking!
2019-03-24 15:08:50 -04:00
James Shubin
6b803656b2 engine: resources: Improve exec resource
The exec resource was an early addition to the project, and it was due
for some fixes and integration into our automated tests. This patch
fixes a number of issues, and makes it ready for more general use.
2019-03-04 10:16:21 -05:00
James Shubin
62a6e0da1d misc: Add two test helpers
Hopefully these make testing and debugging easier!
2019-02-24 12:28:59 -05:00
Jeff Waugh
16d7c6a933 build: Fix macOS build
Add pkg-config to fix builds with augeas and libvirt on macOS.
2019-02-14 23:06:18 +11:00
Johan Bloemberg
0a052494c4 misc: Add goimports dep 2019-02-08 00:14:17 +01:00
Lander Van den Bulcke
10193a2796 make: Use gem --no-document instead of deprecated flags
Signed-off-by: Lander Van den Bulcke <landervdb@inuits.eu>
2019-02-06 12:02:10 +01:00
James Shubin
9966ca2e85 examples: Improve dynamic cpus virt example 2019-02-01 07:58:38 -05:00
Jonathan Gold
74484bcbdf make: deps: Only install gometalinter on CI/CD servers 2019-01-15 20:23:24 -05:00
James Shubin
96dccca475 lang: Add module imports and more
This enables imports in mcl code, and is one of last remaining blockers
to using mgmt. Now we can start writing standalone modules, and adding
standard library functions as needed. There's still lots to do, but this
was a big missing piece. It was much harder to get right than I had
expected, but I think it's solid!

This unfortunately large commit is the result of some wild hacking I've
been doing for the past little while. It's the result of a rebase that
broke many "wip" commits that tracked my private progress, into
something that's not gratuitously messy for our git logs. Since this was
a learning and discovery process for me, I've "erased" the confusing git
history that wouldn't have helped. I'm happy to discuss the dead-ends,
and a small portion of that code was even left in for possible future
use.

This patch includes:

* A change to the cli interface:
You now specify the front-end explicitly, instead of leaving it up to
the front-end to decide when to "activate". For example, instead of:

mgmt run --lang code.mcl

we now do:

mgmt run lang --lang code.mcl

We might rename the --lang flag in the future to avoid the awkward word
repetition. Suggestions welcome, but I'm considering "input". One
side-effect of this change, is that flags which are "engine" specific
now must be specified with "run" before the front-end name. Eg:

mgmt run --tmp-prefix lang --lang code.mcl

instead of putting --tmp-prefix at the end. We also changed the GAPI
slightly, but I've patched all code that used it. This also makes things
consistent with the "deploy" command.

* The deploys are more robust and let you deploy after a run
This has been vastly improved and let's mgmt really run as a smart
engine that can handle different workloads. If you don't want to deploy
when you've started with `run` or if one comes in, you can use the
--no-watch-deploy option to block new deploys.

* The import statement exists and works!
We now have a working `import` statement. Read the docs, and try it out.
I think it's quite elegant how it fits in with `SetScope`. Have a look.
As a result, we now have some built-in functions available in modules.
This also adds the metadata.yaml entry-point for all modules. Have a
look at the examples or the tests. The bulk of the patch is to support
this.

* Improved lang input parsing code:
I re-wrote the parsing that determined what ran when we passed different
things to --lang. Deciding between running an mcl file or raw code is
now handled in a more intelligent, and re-usable way. See the inputs.go
file if you want to have a look. One casualty is that you can't stream
code from stdin *directly* to the front-end, it's encapsulated into a
deploy first. You can still use stdin though! I doubt anyone will notice
this change.

* The scope was extended to include functions and classes:
Go forth and import lovely code. All these exist in scopes now, and can
be re-used!

* Function calls actually use the scope now. Glad I got this sorted out.

* There is import cycle detection for modules!
Yes, this is another dag. I think that's #4. I guess they're useful.

* A ton of tests and new test infra was added!
This should make it much easier to add new tests that run mcl code. Have
a look at TestAstFunc1 to see how to add more of these.

As usual, I'll try to keep these commits smaller in the future!
2018-12-21 06:22:12 -05:00
Felix Frank
9db0fc4ee4 make: Speed up the build by skipping gem docs
Per default, the Ruby gems renerate documentation in two distinct formats
during installation. By passing --no-ri and --no-rdoc, gem is instructed
to skip this step for both formats.

If the user needs documentation for any of the gems after all, they can
manually generate the docs themselves.
2018-12-04 16:56:23 -05:00
Felix Frank
9ed830bb81 make: Remove spurious dependency package 'rubygems' for Debian-like systems
On Ubuntu, the apt-get install call to ruby, ruby-devel, and rubygems will
fail because there is no "rubygems" package in Ubuntu.

In Debian, this package is virtual only. In both cases, the ruby package
is sufficient. (See also https://packages.debian.org/jessie/rubygems)
2018-12-04 16:55:46 -05:00
James Shubin
0ec00fe57f make: Improve release pipeline
Hopefully this makes releases a little better for users.
In particular, this avoids listing old build artifacts in the SHA256SUMS
files when we make new releases, and users can now download them
directly.

Now to make a release you run: `make tag && make release`.
After the first make session ends, you'll have a new tag released
publicly, and then during the second make session, the release target
will notice this new tag, build some assets, and upload them!
2018-11-30 19:08:53 -05:00
Jonathan Gold
80931e1cb4 make: Release pipeline
This commit adds new make targets for rpm, deb, and pacman packages.
It also adds a phony target that uploads tarballs of the packages,
along with their signed (and unsigned) checksums to the github release
page. Once the current commit is tagged as a release, run `make release`
to build the packages and upload them to github.
2018-11-30 04:53:51 -05:00
Jonathan Gold
43b5b4f5a4 build: Add rubygems to make deps target
cffdb06 adds a linter for markdown which requires rubygems.
This commit adds the dependency to the make target.
2018-10-30 17:16:50 -04:00
Johan Bloemberg
9544ab2e02 recwatch: Fix watching newly created files on macOS
Fixes: https://github.com/purpleidea/mgmt/issues/33
2018-02-22 16:52:26 -05:00
Oliver Frommel
318fe4a5dc misc: Small fixes for makedeps script
- install Go distribution package only if no go binary found
2018-02-22 16:50:13 -05:00
James Shubin
05c60d9a59 test, docs: Restrict long lines in markdown linter
It's getting out of hand...
2018-02-22 16:19:23 -05:00
Peter Oliver
f01eea33e9 emacs: Bundle an Emacs major mode, mgmtconfig-mode
This provides syntax highlighting, commenting, and rudimentary indentation of the mgmt language.
2018-02-22 15:57:05 -05:00
James Shubin
9992c367bf misc: Update golint to new location
Somehow this got changed...
2018-02-22 02:01:14 -05:00
James Shubin
d275a23a81 misc: Add dependency on time package
Some environments apparently don't have this installed. We have it in
certain places where we like to time things.
2018-02-21 22:52:41 -05:00
James Shubin
cffdb06181 test, docs: Add a linter for testing markdown, and fix up our docs
While writing docs, I couldn't remember what the correct style was
supposed to be, and I remember someone complaining about this
previously, so I decided to add a linter! I excluded a bunch of annoying
style rules, but if we find more we can add those to the list too.

Hopefully this gives us a more consistent feel throughout.
2018-02-21 22:52:41 -05:00
James Shubin
4898297cce travis: Avoid notification noise from forks
Encrypt name of IRC channel to workaround forks spamming us with their
testing messages.

Docs: https://docs.travis-ci.com/user/environment-variables/#Defining-encrypted-variables-in-.travis.yml
2018-02-20 14:12:09 -05:00
Johan Bloemberg
572b2575c5 test: Export the mgmt command to be used during test 2018-02-18 14:19:05 -05:00
Johan Bloemberg
b34b359860 test: Streamline test suite a little
This change aims to streamline the integrationtest suite and reduce friction when running (parts of) test suites.

Changes:
- add `test-testname` to makefile to easily run one suite
- made skipping tests first class citizen in test.sh (all available testsuites and the reasons they are skipped are now better exposed and discovered)
- suppress some output of gotest unless there is an error
- no longer build binary for examples and gotest suites
- removed .SILENT from makefile as it being applied to only some targets makes it feel weird (I just learned about this option btw, feel free to comment on this change)
- move individual tests out of `test.sh` and into `test-misc.sh`
- introduced the concept of testsuites to `test.sh`
2018-02-15 17:21:49 +01:00
Wim
3a50171d19 build: Add gcc,pkg-config deps 2018-02-12 15:10:19 -05:00