lang: types, funcs: Add simple function API

This patch adds a simple function API for writing simple, pure
functions. This should reduce the amount of boilerplate required for
most functions, and make growing a stdlib significantly easier. If you
need to build more complex, event-generating functions, or statically
polymorphic functions, then you'll still need to use the normal API for
now.

This also makes all of these pure functions available automatically
within templates. It might make sense to group these functions into
packages to make their logical organization easier, but this is a good
enough start for now.

Lastly, this added some missing pieces to our types library. You can now
use `ValueOf` to convert from a `reflect.Value` to the corresponding
`Value` in our type system, if an equivalent exists.

Unfortunately, we're severely lacking in tests for these new types
library additions, but look forward to growing some in the future!

docs/function-guide.md

@@ -0,0 +1,318 @@
+# Function guide
+
+## Overview
+
+The `mgmt` tool has built-in functions which add useful, reactive functionality
+to the language. This guide describes the different function API's that are
+available. It is meant to instruct developers on how to write new functions.
+Since `mgmt` and the core functions are written in golang, some prior golang
+knowledge is assumed.
+
+## Theory
+
+Functions in `mgmt` are similar to functions in other languages, however they
+also have a [reactive](https://en.wikipedia.org/wiki/Functional_reactive_programming)
+component. Our functions can produce events over time, and there are different
+ways to write functions. For some background on this design, please read the
+[original article](https://purpleidea.com/blog/2018/02/05/mgmt-configuration-language/)
+on the subject.
+
+## Native Functions
+
+Native functions are functions which are implemented in the mgmt language
+itself. These are currently not available yet, but are coming soon. Stay tuned!
+
+## Simple Function API
+
+Most functions should be implemented using the simple function API. This API
+allows you to implement simple, static, [pure](https://en.wikipedia.org/wiki/Pure_function)
+functions that don't require you to write much boilerplate code. They will be
+automatically re-evaluated as needed when their input values change. These will
+all be automatically made available as helper functions within mgmt templates,
+and are also available for use anywhere inside mgmt programs.
+
+You'll need some basic knowledge of using the [`types`](https://github.com/purpleidea/mgmt/tree/master/lang/types)
+library which is included with mgmt. This library lets you interact with the
+available types and values in the mgmt language. It is very easy to use, and
+should be fairly intuitive. Most of what you'll need to know can be inferred
+from looking at example code.
+
+To implement a function, you'll need to create a file in
+[`lang/funcs/simple/`](https://github.com/purpleidea/mgmt/tree/master/lang/funcs/simple/).
+The function should be implemented as a `FuncValue` in our type system. It is
+then registered with the engine during `init()`. An example explains it best:
+
+### Example
+
+```golang
+package simple
+
+import (
+	"fmt"
+
+	"github.com/purpleidea/mgmt/lang/types"
+)
+
+// you must register your functions in init when the program starts up
+func init() {
+	// Example function that squares an int and prints out answer as an str.
+	Register("talkingsquare", &types.FuncValue{
+		T: types.NewType("func(a int) str"), // declare the signature
+		V: func(input []types.Value) (types.Value, error) {
+			i := input[0].Int() // get first arg as an int64
+			// must return the above specified value
+			return &types.StrValue{
+				V: fmt.Sprintf("%d^2 is %d", i, i * i),
+			}, nil // no serious errors occurred
+		},
+	})
+}
+```
+
+This simple function accepts one `int` as input, and returns one `str`.
+Functions can have zero or more inputs, and must have exactly one output. You
+must be sure to use the `types` library correctly, since if you try and access
+an input which should not exist (eg: `input[2]`, when there are only two
+that are expected), then you will cause a panic. If you have declared that a
+particular argument is an `int` but you try to read it with `.Bool()` you will
+also cause a panic. Lastly, make sure that you return a value in the correct
+type or you will also cause a panic!
+
+If anything goes wrong, you can return an error, however this will cause the
+mgmt engine to shutdown. It should be seen as the equivalent to calling a
+`panic()`, however it is safer because it brings the engine down cleanly.
+Ideally, your functions should never need to error. You should never cause a
+real `panic()`, since this could have negative consequences to the system.
+
+## Function API
+
+To implement a reactive function in `mgmt` it must satisfy the
+[`Func`](https://github.com/purpleidea/mgmt/blob/master/lang/interfaces/func.go)
+interface. Using the [Simple Function API](#simple-function-api) is preferable
+if it meets your needs. Most functions will be able to use that API. If you
+really need something more powerful, then you can use the regular function API.
+What follows are each of the method signatures and a description of each.
+
+### Default
+
+```golang
+Info() *interfaces.Info
+```
+
+This returns some information about the function. It is necessary so that the
+compiler can type check the code correctly, and know what optimizations can be
+performed. This is usually the first method which is called by the engine.
+
+#### Example
+
+```golang
+func (obj *FooFunc) Info() *interfaces.Info {
+	return &interfaces.Info{
+		Pure: true,
+		Sig:  types.NewType("func(a int) str"),
+	}
+}
+```
+
+### Init
+
+```golang
+Init(init *interfaces.Init) error
+```
+
+This is called to initialize the function. If something goes wrong, it should
+return an error. It is passed a struct that contains all the important
+information and poiinters that it might need to work with throughout its
+lifetime. As a result, it will need to save a copy to that pointer for future
+use in the other methods.
+
+#### Example
+
+```golang
+// Init runs some startup code for this function.
+func (obj *FooFunc) Init(init *interfaces.Init) error {
+	obj.init = init
+	obj.closeChan = make(chan struct{}) // shutdown signal
+	return nil
+}
+```
+
+### Close
+
+```golang
+Close() error
+```
+
+This is called to cleanup the function. It usually causes the stream to
+shutdown. Even if `Stream()` decided to shutdown early, it might still get
+called. It is usually called by the engine to tell the function to shutdown.
+
+#### Example
+
+```golang
+// Close runs some shutdown code for this function and turns off the stream.
+func (obj *FooFunc) Close() error {
+	close(obj.closeChan) // send a signal to tell the stream to close
+	return nil
+}
+```
+
+### Stream
+
+```golang
+Stream() error
+```
+
+`Stream` is where the real _work_ is done. This method is started by the
+language function engine. It will run this function while simultaneously sending
+it values on the `input` channel. It will only send a complete set of input
+values. You should send a value to the output channel when you have decided that
+one should be produced. Make sure to only use input values of the expected type
+as declared in the `Info` struct, and send values of the similarly declared
+appropriate return type. Failure to do so will may result in a panic and
+sadness.
+
+#### Example
+
+```golang
+// Stream returns the single value that was generated and then closes.
+func (obj *FooFunc) Stream() error {
+	defer close(obj.init.Output) // the sender closes
+	var result string
+	for {
+		select {
+		case input, ok := <-obj.init.Input:
+			if !ok {
+				return nil // can't output any more
+			}
+
+			ix := input.Struct()["a"].Int()
+			if ix < 0 {
+				return fmt.Errorf("we can't deal with negatives")
+			}
+
+			result = fmt.Sprintf("the input is: %d", ix)
+
+		case <-obj.closeChan:
+			return nil
+		}
+
+		select {
+		case obj.init.Output <- &types.StrValue{
+			V: result,
+		}:
+
+		case <-obj.closeChan:
+			return nil
+		}
+	}
+}
+```
+
+As you can see, we read our inputs from the `input` channel, and write to the
+`output` channel. Our code is careful to never block or deadlock, and can always
+exit if a close signal is requested. It also cleans up after itself by closing
+the `output` channel when it is done using it. This is done easily with `defer`.
+If it notices that the `input` channel closes, then it knows that no more input
+values are coming and it can consider shutting down early.
+
+## Further considerations
+
+There is some additional information that any function author will need to know.
+Each issue is listed separately below!
+
+### Function struct
+
+Each function will implement methods as pointer receivers on a function struct.
+The naming convention for resources is that they end with a `Func` suffix.
+
+#### Example
+
+```golang
+type FooFunc struct {
+	init *interfaces.Init
+
+	// this space can be used if needed
+
+	closeChan chan struct{} // shutdown signal
+}
+```
+
+### Function registration
+
+All functions must be registered with the engine so that they can be found. This
+also ensures they can be encoded and decoded. Make sure to include the following
+code snippet for this to work.
+
+```golang
+func init() { // special golang method that runs once
+	funcs.Register("foo", func() interfaces.Func { return &FooFunc{} })
+}
+```
+
+### Composite functions
+
+Composite functions are functions which import one or more existing functions.
+This is useful to prevent code duplication in higher level function scenarios.
+Unfortunately no further documentation about this subject has been written. To
+expand this section, please send a patch! Please contact us if you'd like to
+work on a function that uses this feature, or to add it to an existing one!
+We don't expect this functionality to be particularly useful or common, as it's
+probably easier and preferable to simply import common golang library code into
+multiple different functions instead.
+
+## Polymorphic Function API
+
+The polymorphic function API is an API that lets you implement functions which
+do not necessarily have a single static function signature. After compile time,
+all functions must have a static function signature. We also know that there
+might be different ways you would want to call `printf`, such as:
+`printf("the %s is %d", "answer", 42)` or `printf("3 * 2 = %d", 3 * 2)`. Since
+you couldn't implement the infinite number of possible signatures, this API lets
+you write code which can be coerced into different forms. This makes
+implementing what would appear to be generic or polymorphic, instead something
+that is actually static and that still has the static type safety properties
+that were guaranteed by the mgmt language.
+
+Since this is an advanced topic, it is not described in full at this time. For
+more information please have a look at the source code comments, some of the
+existing implementations, and ask around in the community.
+
+## Frequently asked questions
+
+(Send your questions as a patch to this FAQ! I'll review it, merge it, and
+respond by commit with the answer.)
+
+### Can I use global variables?
+
+Probably not. You must assume that multiple copies of your function may be used
+at the same time. If they require a global variable, it's likely this won't
+work. Instead it's probably better to use a struct local variable if you need to
+store some state.
+
+There might be some rare instances where a global would be acceptable, but if
+you need one of these, you're probably already an internals expert. If you think
+they need to lock or synchronize so as to not overwhelm an external resource,
+then you have to be especially careful not to cause deadlocking the mgmt engine.
+
+### Can I write functions in a different language?
+
+Currently `golang` is the only supported language for built-in functions. We
+might consider allowing external functions to be imported in the future. This
+will likely require a language that can expose a C-like API, such as `python` or
+`ruby`. Custom `golang` functions are already possible when using mgmt as a lib.
+
+### What new functions need writing?
+
+There are still many ideas for new functions that haven't been written yet. If
+you'd like to contribute one, please contact us and tell us about your idea!
+
+### Where can I find more information about mgmt?
+
+Additional blog posts, videos and other material [is available!](https://github.com/purpleidea/mgmt/blob/master/docs/on-the-web.md).
+
+## Suggestions
+
+If you have any ideas for API changes or other improvements to function writing,
+please let us know! We're still pre 1.0 and pre 0.1 and happy to break API in
+order to get it right!

docs/language-guide.md

@@ -212,6 +212,8 @@ Your function must have a specific type. For example, a simple math function
 might have a signature of `func(x int, x int) int`. As you can see, all the
 types are known _before_ compile time.
 
+A separate discussion on this matter can be found in the [function guide](function-guide.md).
+
 What follows are each of the method signatures and a description of each.
 Failure to implement the API correctly can cause the function graph engine to
 block, or the program to panic.
@@ -279,7 +281,7 @@ one value must be produced.
 #### Example
 ```golang
 Please see the example functions in
-[lang/funcs/public/](https://github.com/purpleidea/mgmt/tree/master/lang/funcs/public/).
+[lang/funcs/core/](https://github.com/purpleidea/mgmt/tree/master/lang/funcs/core/).
 ```
 
 ### Stream
@@ -298,7 +300,7 @@ whether or not you close the `Output` channel.
 #### Example
 ```golang
 Please see the example functions in
-[lang/funcs/public/](https://github.com/purpleidea/mgmt/tree/master/lang/funcs/public/).
+[lang/funcs/core/](https://github.com/purpleidea/mgmt/tree/master/lang/funcs/core/).
 ```
 
 ### Close
@@ -312,7 +314,7 @@ return.
 #### Example
 ```golang
 Please see the example functions in
-[lang/funcs/public/](https://github.com/purpleidea/mgmt/tree/master/lang/funcs/public/).
+[lang/funcs/core/](https://github.com/purpleidea/mgmt/tree/master/lang/funcs/core/).
 ```
 
 ### Polymorphic Function API

examples/lang/datetime1.mcl

@@ -1,4 +1,4 @@
 $d = datetime()
 file "/tmp/mgmt/datetime" {
-	content => template("Hello! It is now: {{ datetimeprint . }}\n", $d),
+	content => template("Hello! It is now: {{ datetime_print . }}\n", $d),
 }

examples/lang/datetime2.mcl

@@ -9,6 +9,6 @@ $theload = structlookup(load(), "x1")
 
 if 5 > 3 {
 	file "/tmp/mgmt/datetime" {
-		content => template("Now + 1 year is: {{ .year }} seconds, aka: {{ datetimeprint .year }}\n\nload average: {{ .load }}\n", $tmplvalues),
+		content => template("Now + 1 year is: {{ .year }} seconds, aka: {{ datetime_print .year }}\n\nload average: {{ .load }}\n", $tmplvalues),
 	}
 }

examples/lang/datetime3.mcl

@@ -10,5 +10,5 @@ $theload = structlookup(load(), "x1")
 $vumeter = vumeter("====", 10, 0.9)
 
 file "/tmp/mgmt/datetime" {
-	content => template("Now + 1 year is: {{ .year }} seconds, aka: {{ datetimeprint .year }}\n\nload average: {{ .load }}\n\nvu: {{ .vumeter }}\n", $tmplvalues),
+	content => template("Now + 1 year is: {{ .year }} seconds, aka: {{ datetime_print .year }}\n\nload average: {{ .load }}\n\nvu: {{ .vumeter }}\n", $tmplvalues),
 }

examples/lang/history1.mcl

@@ -3,5 +3,5 @@ $dt = datetime()
 $hystvalues = {"ix0" => $dt, "ix1" => $dt{1}, "ix2" => $dt{2}, "ix3" => $dt{3},}
 
 file "/tmp/mgmt/history" {
-	content => template("Index(0) {{.ix0}}: {{ datetimeprint .ix0 }}\nIndex(1) {{.ix1}}: {{ datetimeprint .ix1 }}\nIndex(2) {{.ix2}}: {{ datetimeprint .ix2 }}\nIndex(3) {{.ix3}}: {{ datetimeprint .ix3 }}\n", $hystvalues),
+	content => template("Index(0) {{.ix0}}: {{ datetime_print .ix0 }}\nIndex(1) {{.ix1}}: {{ datetime_print .ix1 }}\nIndex(2) {{.ix2}}: {{ datetime_print .ix2 }}\nIndex(3) {{.ix3}}: {{ datetime_print .ix3 }}\n", $hystvalues),
 }

examples/lang/template0.mcl

@@ -0,0 +1,10 @@
+$answer = 42
+$s = int2str($answer)
+
+print "print1" {
+	msg => printf("an str is: %s", $s),
+}
+
+print "print2" {
+	msg => template("an str is: {{ int2str . }}", $answer),
+}

lang/funcs/core/template_polyfunc.go

@@ -20,16 +20,23 @@ package core // TODO: should this be in its own individual package?
 import (
 	"bytes"
 	"fmt"
+	"reflect"
 	"text/template"
-	"time"
 
 	"github.com/purpleidea/mgmt/lang/funcs"
+	"github.com/purpleidea/mgmt/lang/funcs/simple"
 	"github.com/purpleidea/mgmt/lang/interfaces"
 	"github.com/purpleidea/mgmt/lang/types"
 
 	errwrap "github.com/pkg/errors"
 )
 
+var (
+	// errorType represents a reflection type of error as seen in:
+	// https://github.com/golang/go/blob/ec62ee7f6d3839fe69aeae538dadc1c9dc3bf020/src/text/template/exec.go#L612
+	errorType = reflect.TypeOf((*error)(nil)).Elem()
+)
+
 func init() {
 	funcs.Register("template", func() interfaces.Func { return &TemplateFunc{} })
 }
@@ -42,7 +49,8 @@ const TemplateName = "template"
 // to it. It examines the type of the second argument (the input data vars) at
 // compile time and then determines the static functions signature by including
 // that in the overall signature.
-// XXX: do we need to add events if any of the internal functions change over time?
+// TODO: We *might* need to add events for internal function changes over time,
+// but only if they are not pure. We currently only use simple, pure functions.
 type TemplateFunc struct {
 	Type *types.Type // type of vars
 
@@ -159,13 +167,40 @@ func (obj *TemplateFunc) Init(init *interfaces.Init) error {
 
 // run runs a template and returns the result.
 func (obj *TemplateFunc) run(templateText string, vars types.Value) (string, error) {
+	// see: https://golang.org/pkg/text/template/#FuncMap for more info
+	// note: we can override any other functions by adding them here...
 	funcMap := map[string]interface{}{
-		// XXX: can these functions come from normal funcValue things
+	//"test1": func(in interface{}) (interface{}, error) { // ok
-		// that we build for the interfaces.Func part?
+	//	return fmt.Sprintf("got(%T): %+v", in, in), nil
-		// TODO: add a bunch of stdlib-like stuff here...
+	//},
-		"datetimeprint": func(epochDelta int64) string { // TODO: rename
+	//"test2": func(in interface{}) interface{} { // NOT ok
-			return time.Unix(epochDelta, 0).String()
+	//	panic("panic") // a panic here brings down everything!
-		},
+	//},
+	//"test3": func(foo int64) (string, error) { // ok, but errors
+	//	return "", fmt.Errorf("i am an error")
+	//},
+	//"test4": func(in1, in2 reflect.Value) (reflect.Value, error) { // ok
+	//	s := fmt.Sprintf("got: %+v and: %+v", in1, in2)
+	//	return reflect.ValueOf(s), nil
+	//},
+	}
+
+	// FIXME: should we do this once in init() instead, or in the Register
+	// function in the simple package?
+	// TODO: loop through this map in a sorted, deterministic order
+	for name, fn := range simple.RegisteredFuncs {
+		if _, exists := funcMap[name]; exists {
+			obj.init.Logf("warning, existing function named: `%s` exists", name)
+			continue
+		}
+
+		// When template execution invokes a function with an argument
+		// list, that list must be assignable to the function's
+		// parameter types. Functions meant to apply to arguments of
+		// arbitrary type can use parameters of type interface{} or of
+		// type reflect.Value.
+		f := wrap(name, fn) // wrap it so that it meets API expectations
+		funcMap[name] = f   // add it
 	}
 
 	var err error
@@ -288,3 +323,70 @@ func (obj *TemplateFunc) Close() error {
 	close(obj.closeChan)
 	return nil
 }
+
+// wrap builds a function in the format expected by the template engine, and
+// returns it as an interface{}. It does so by wrapping our type system and
+// function API with what is expected from the reflection API. It returns a
+// version that includes the optional second error return value so that our
+// functions can return errors without causing a panic.
+func wrap(name string, fn *types.FuncValue) interface{} {
+	if fn.T.Map == nil {
+		panic("malformed func type")
+	}
+	if len(fn.T.Map) != len(fn.T.Ord) {
+		panic("malformed func length")
+	}
+	in := []reflect.Type{}
+	for _, k := range fn.T.Ord {
+		t, ok := fn.T.Map[k]
+		if !ok {
+			panic("malformed func order")
+		}
+		if t == nil {
+			panic("malformed func arg")
+		}
+
+		in = append(in, t.Reflect())
+	}
+	out := []reflect.Type{fn.T.Out.Reflect(), errorType}
+	var variadic = false // currently not supported in our function value
+	typ := reflect.FuncOf(in, out, variadic)
+
+	// wrap our function with the translation that is necessary
+	f := func(args []reflect.Value) (results []reflect.Value) { // build
+		innerArgs := []types.Value{}
+		zeroValue := reflect.Zero(fn.T.Out.Reflect()) // zero value of return type
+		for _, x := range args {
+			v, err := types.ValueOf(x) // reflect.Value -> Value
+			if err != nil {
+				r := reflect.ValueOf(errwrap.Wrapf(err, "function `%s` errored", name))
+				if !r.Type().ConvertibleTo(errorType) { // for fun!
+					r = reflect.ValueOf(fmt.Errorf("function `%s` errored: %+v", name, err))
+				}
+				e := r.Convert(errorType) // must be seen as an `error`
+				return []reflect.Value{zeroValue, e}
+			}
+			innerArgs = append(innerArgs, v)
+		}
+
+		result, err := fn.Call(innerArgs) // call it
+		if err != nil {                   // function errored :(
+			// errwrap is a better way to report errors, if allowed!
+			r := reflect.ValueOf(errwrap.Wrapf(err, "function `%s` errored", name))
+			if !r.Type().ConvertibleTo(errorType) { // for fun!
+				r = reflect.ValueOf(fmt.Errorf("function `%s` errored: %+v", name, err))
+			}
+			e := r.Convert(errorType) // must be seen as an `error`
+			return []reflect.Value{zeroValue, e}
+		} else if result == nil { // someone wrote a bad function
+			r := reflect.ValueOf(fmt.Errorf("function `%s` returned nil", name))
+			e := r.Convert(errorType) // must be seen as an `error`
+			return []reflect.Value{zeroValue, e}
+		}
+
+		nilError := reflect.Zero(errorType)
+		return []reflect.Value{reflect.ValueOf(result.Value()), nilError}
+	}
+	val := reflect.MakeFunc(typ, f)
+	return val.Interface()
+}

lang/funcs/simple/datetime_print_func.go

@@ -0,0 +1,42 @@
+// Mgmt
+// Copyright (C) 2013-2018+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package simple // TODO: should this be in its own individual package?
+
+import (
+	"fmt"
+	"time"
+
+	"github.com/purpleidea/mgmt/lang/types"
+)
+
+func init() {
+	// TODO: should we support namespacing these, eg: datetime.print ?
+	// FIXME: consider renaming this to printf, and add in a format string?
+	Register("datetime_print", &types.FuncValue{
+		T: types.NewType("func(a int) str"),
+		V: func(input []types.Value) (types.Value, error) {
+			epochDelta := input[0].Int()
+			if epochDelta < 0 {
+				return nil, fmt.Errorf("epoch delta must be positive")
+			}
+			return &types.StrValue{
+				V: time.Unix(epochDelta, 0).String(),
+			}, nil
+		},
+	})
+}

lang/funcs/simple/example_errorbool_func.go

@@ -0,0 +1,39 @@
+// Mgmt
+// Copyright (C) 2013-2018+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package simple // TODO: should this be in its own individual package?
+
+import (
+	"fmt"
+
+	"github.com/purpleidea/mgmt/lang/types"
+)
+
+func init() {
+	// TODO: should we support namespacing these, eg: example.errorbool ?
+	Register("example_errorbool", &types.FuncValue{
+		T: types.NewType("func(a bool) str"),
+		V: func(input []types.Value) (types.Value, error) {
+			if input[0].Bool() {
+				return nil, fmt.Errorf("we errored on request")
+			}
+			return &types.StrValue{
+				V: "set input to true to generate an error",
+			}, nil
+		},
+	})
+}

lang/funcs/simple/int2str_func.go

@@ -0,0 +1,35 @@
+// Mgmt
+// Copyright (C) 2013-2018+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package simple // TODO: should this be in its own individual package?
+
+import (
+	"fmt"
+
+	"github.com/purpleidea/mgmt/lang/types"
+)
+
+func init() {
+	Register("int2str", &types.FuncValue{
+		T: types.NewType("func(a int) str"),
+		V: func(input []types.Value) (types.Value, error) {
+			return &types.StrValue{
+				V: fmt.Sprintf("%d", input[0].Int()),
+			}, nil
+		},
+	})
+}

lang/funcs/simple/simple.go

@@ -0,0 +1,135 @@
+// Mgmt
+// Copyright (C) 2013-2018+ James Shubin and the project contributors
+// Written by James Shubin <james@shubin.ca> and the project contributors
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package simple // TODO: should this be in its own individual package?
+
+import (
+	"fmt"
+
+	"github.com/purpleidea/mgmt/lang/funcs"
+	"github.com/purpleidea/mgmt/lang/interfaces"
+	"github.com/purpleidea/mgmt/lang/types"
+
+	errwrap "github.com/pkg/errors"
+)
+
+// RegisteredFuncs maps a function name to the corresponding static, pure func.
+var RegisteredFuncs = make(map[string]*types.FuncValue) // must initialize
+
+// Register registers a simple, static, pure function. It is easier to use that
+// the raw function API, but also limits you to simple, static, pure functions.
+func Register(name string, fn *types.FuncValue) {
+	if _, exists := RegisteredFuncs[name]; exists {
+		panic(fmt.Sprintf("a simple func named %s is already registered", name))
+	}
+	RegisteredFuncs[name] = fn // store a copy for ourselves
+
+	// register a copy in the main function database
+	funcs.Register(name, func() interfaces.Func { return &simpleFunc{Fn: fn} })
+}
+
+// simpleFunc is a scaffolding function struct which fulfills the boiler-plate
+// for the function API, but that can run a very simple, static, pure function.
+type simpleFunc struct {
+	Fn *types.FuncValue
+
+	init *interfaces.Init
+	last types.Value // last value received to use for diff
+
+	result types.Value // last calculated output
+
+	closeChan chan struct{}
+}
+
+// Validate makes sure we've built our struct properly. It is usually unused for
+// normal functions that users can use directly.
+func (obj *simpleFunc) Validate() error {
+	if obj.Fn == nil { // build must be run first
+		return fmt.Errorf("type is still unspecified")
+	}
+	return nil
+}
+
+// Info returns some static info about itself.
+func (obj *simpleFunc) Info() *interfaces.Info {
+	return &interfaces.Info{
+		Pure: true,
+		Memo: false, // TODO: should this be something we specify here?
+		Sig:  obj.Fn.Type(),
+		Err:  obj.Validate(),
+	}
+}
+
+// Init runs some startup code for this function.
+func (obj *simpleFunc) Init(init *interfaces.Init) error {
+	obj.init = init
+	obj.closeChan = make(chan struct{})
+	return nil
+}
+
+// Stream returns the changing values that this func has over time.
+func (obj *simpleFunc) Stream() error {
+	defer close(obj.init.Output) // the sender closes
+	for {
+		select {
+		case input, ok := <-obj.init.Input:
+			if !ok {
+				return nil // can't output any more
+			}
+			//if err := input.Type().Cmp(obj.Info().Sig.Input); err != nil {
+			//	return errwrap.Wrapf(err, "wrong function input")
+			//}
+
+			if obj.last != nil && input.Cmp(obj.last) == nil {
+				continue // value didn't change, skip it
+			}
+			obj.last = input // store for next
+
+			values := []types.Value{}
+			for _, name := range obj.Fn.Type().Ord {
+				x := input.Struct()[name]
+				values = append(values, x)
+			}
+
+			result, err := obj.Fn.Call(values) // (Value, error)
+			if err != nil {
+				return errwrap.Wrapf(err, "simple function errored")
+			}
+
+			if obj.result == result {
+				continue // result didn't change
+			}
+			obj.result = result // store new result
+
+		case <-obj.closeChan:
+			return nil
+		}
+
+		select {
+		case obj.init.Output <- obj.result: // send
+			// pass
+		case <-obj.closeChan:
+			return nil
+		}
+	}
+}
+
+// Close runs some shutdown code for this function and turns off the stream.
+func (obj *simpleFunc) Close() error {
+	close(obj.closeChan)
+	return nil
+}

lang/lang.go

@@ -26,6 +26,7 @@ import (
 	"github.com/purpleidea/mgmt/lang/funcs"
 	_ "github.com/purpleidea/mgmt/lang/funcs/core"       // import so the funcs register
 	_ "github.com/purpleidea/mgmt/lang/funcs/facts/core" // import so the facts register
+	_ "github.com/purpleidea/mgmt/lang/funcs/simple"     // import so the funcs register
 	"github.com/purpleidea/mgmt/lang/interfaces"
 	"github.com/purpleidea/mgmt/lang/unification"
 	"github.com/purpleidea/mgmt/pgraph"

lang/types/value.go

@@ -47,8 +47,154 @@ type Value interface {
 }
 
 // ValueOf takes a reflect.Value and returns an equivalent Value.
-func ValueOf(value reflect.Value) Value {
+func ValueOf(v reflect.Value) (Value, error) {
-	panic("not implemented") // XXX: not implemented
+	value := v
+	typ := value.Type()
+	kind := typ.Kind()
+	for kind == reflect.Ptr {
+		typ = typ.Elem() // un-nest one pointer
+		kind = typ.Kind()
+
+		// un-nest value from pointer
+		value = value.Elem() // XXX: is this correct?
+	}
+
+	switch kind { // match on destination field kind
+	case reflect.Bool:
+		return &BoolValue{V: value.Bool()}, nil
+
+	case reflect.String:
+		return &StrValue{V: value.String()}, nil
+
+	case reflect.Int, reflect.Int64, reflect.Int32, reflect.Int16, reflect.Int8:
+		return &IntValue{V: value.Int()}, nil
+
+	case reflect.Uint, reflect.Uint64, reflect.Uint32, reflect.Uint16, reflect.Uint8:
+		return &IntValue{V: int64(value.Uint())}, nil
+
+	case reflect.Float64, reflect.Float32:
+		return &FloatValue{V: value.Float()}, nil
+
+	case reflect.Array, reflect.Slice:
+		values := []Value{}
+		for i := 0; i < value.Len(); i++ {
+			x := value.Index(i)
+			v, err := ValueOf(x) // recurse
+			if err != nil {
+				return nil, err
+			}
+			values = append(values, v)
+		}
+
+		t, err := TypeOf(value.Type().Elem()) // type of contents
+		if err != nil {
+			return nil, errwrap.Wrapf(err, "can't determine type of %+v", value)
+		}
+
+		return &ListValue{
+			T: NewType(fmt.Sprintf("[]%s", t.String())),
+			V: values,
+		}, nil
+
+	case reflect.Map:
+		m := make(map[Value]Value)
+
+		// loop through the list of map keys in undefined order
+		for _, mk := range value.MapKeys() {
+			mv := value.MapIndex(mk)
+
+			k, err := ValueOf(mk) // recurse
+			if err != nil {
+				return nil, err
+			}
+			v, err := ValueOf(mv) // recurse
+			if err != nil {
+				return nil, err
+			}
+
+			m[k] = v
+		}
+
+		kt, err := TypeOf(value.Type().Key()) // type of key
+		if err != nil {
+			return nil, errwrap.Wrapf(err, "can't determine key type of %+v", value)
+		}
+		vt, err := TypeOf(value.Type().Elem()) // type of value
+		if err != nil {
+			return nil, errwrap.Wrapf(err, "can't determine value type of %+v", value)
+		}
+
+		return &MapValue{
+			T: NewType(fmt.Sprintf("{%s: %s}", kt.String(), vt.String())),
+			V: m,
+		}, nil
+
+	case reflect.Struct:
+		// TODO: we could take this simpler "get the full type" approach
+		// for all the values, but I think that building them up when
+		// possible for the other cases is a more robust approach!
+		t, err := TypeOf(value.Type())
+		if err != nil {
+			return nil, errwrap.Wrapf(err, "can't determine type of %+v", value)
+		}
+		l := value.Len() // number of struct fields according to value
+
+		if l != len(t.Ord) {
+			// programming error?
+			return nil, fmt.Errorf("incompatible number of fields")
+		}
+
+		values := make(map[string]Value)
+		for i := 0; i < l; i++ {
+			x := value.Field(i)
+			v, err := ValueOf(x) // recurse
+			if err != nil {
+				return nil, err
+			}
+			name := t.Ord[i] // how else can we get the field name?
+			values[name] = v
+		}
+
+		return &StructValue{
+			T: t,
+			V: values,
+		}, nil
+
+	case reflect.Func:
+		t, err := TypeOf(value.Type())
+		if err != nil {
+			return nil, errwrap.Wrapf(err, "can't determine type of %+v", value)
+		}
+		if t.Out == nil {
+			return nil, fmt.Errorf("cannot only represent functions with one output value")
+		}
+
+		f := func(args []Value) (Value, error) {
+			in := []reflect.Value{}
+			for _, x := range args {
+				// TODO: should we build this method instead?
+				//v := x.Reflect() // types.Value -> reflect.Value
+				v := reflect.ValueOf(x.Value())
+				in = append(in, v)
+			}
+
+			// FIXME: can we trap panic's ?
+			out := value.Call(in) // []reflect.Value
+			if len(out) != 1 {    // TODO: panic, b/c already checked in TypeOf?
+				return nil, fmt.Errorf("cannot only represent functions with one output value")
+			}
+
+			return ValueOf(out[0]) // recurse
+		}
+
+		return &FuncValue{
+			T: t,
+			V: f,
+		}, nil
+
+	default:
+		return nil, fmt.Errorf("unable to represent value of %+v", v)
+	}
 }
 
 // ValueSlice is a linear list of values. It is used for sorting purposes.
@@ -800,7 +946,10 @@ func (obj *FuncValue) Value() interface{} {
 	fn := func(args []reflect.Value) (results []reflect.Value) { // build
 		innerArgs := []Value{}
 		for _, x := range args {
-			v := ValueOf(x) // reflect.Value -> Value
+			v, err := ValueOf(x) // reflect.Value -> Value
+			if err != nil {
+				panic(fmt.Sprintf("can't determine value of %+v", x))
+			}
 			innerArgs = append(innerArgs, v)
 		}
 		result, err := obj.V(innerArgs) // call it
@@ -836,18 +985,23 @@ func (obj *FuncValue) Call(args []Value) (Value, error) {
 	// cmp input args type to obj.T
 	length := len(obj.T.Ord)
 	if length != len(args) {
-		panic(fmt.Sprintf("arg length of %d does not match expected of %d", len(args), length))
+		return nil, fmt.Errorf("arg length of %d does not match expected of %d", len(args), length)
 	}
 	for i := 0; i < length; i++ {
 		if err := args[i].Type().Cmp(obj.T.Map[obj.T.Ord[i]]); err != nil {
-			panic(fmt.Sprintf("cannot cmp input types: %+v", err))
+			return nil, errwrap.Wrapf(err, "cannot cmp input types")
 		}
 	}
 
 	result, err := obj.V(args) // call it
-
+	if result == nil {
+		if err == nil {
+			return nil, fmt.Errorf("function returned nil result")
+		}
+		return nil, errwrap.Wrapf(err, "function returned nil result during error")
+	}
 	if err := result.Type().Cmp(obj.T.Out); err != nil {
-		panic(fmt.Sprintf("cannot cmp return types: %+v", err))
+		return nil, errwrap.Wrapf(err, "cannot cmp return types")
 	}
 
 	return result, err

lang/types/value_test.go

@@ -586,3 +586,7 @@ func TestStruct2(t *testing.T) {
 		//t.Errorf("val2: %s", val2)
 	}
 }
+
+func TestValueOf0(t *testing.T) {
+	// TODO: implement testing of the ValueOf function
+}