demo.org

#+title: cogs ⚙ demo
#+subtitle: Demonstration of Configuration Object Generation System
#+setupfile: setup.org
#+options: broken-links:mark

#+name: grep
#+begin_src shell :var re="" :var file="/dev/null"  :var a=0 :var lang="shell" :exports none :results output code
grep -m1 -A$a "$re" "$file"
#+end_src

#+name: runcmd
#+begin_src shell :var cmd="echo hello world" :exports none :results output code :wrap "example"
$cmd
#+end_src



* Introduction
  :PROPERTIES:
  :CUSTOM_ID: intro
  :END:

The [[file:demo/][demo/]] directory of the [[https://github.com/brettviren/cogs][cogs]] repository holds a demonstration of
one way to make use of ~cogs~.  It includes:

- A mocked up framework, application and components.
- Schema to generate configuration classes.
- Example ~cogs~ configuration stream file.
- Simple tooling to rerun code generation.
- Integration into ~cogs~ build system.

This document describes how to build and run the demo.  It then gives
a tour of the mocked framework to understand one possible way to allow
~cogs~ to be used.  Details on the code generation steps come next and
it ends with a section that uses the demo's schema to illustrate how
one may develop schema for applications.

* Build
  :PROPERTIES:
  :CUSTOM_ID: build
  :END:

This section describes issues about building the demo code. 

** Prerequisites
   :PROPERTIES:
   :CUSTOM_ID: prereq
   :END:

In addition to what ~cogs~ library requires, the demo requires:

- ~moo~ Python program from [[https://github.com/brettviren/moo][moo]] (only to re-codegen)

** Compile
   :PROPERTIES:
   :CUSTOM_ID: compile
   :END:

With prerequisites satisfied, the demo builds with the ~cogs~ library.  For example:

#+begin_example
waf configure --prefix=$(pwd)/install \
    --with-nljs=$HOME/opt/nljs \
    --with-ers=$HOME/opt/ers 
waf install
#+end_example

You should be rewarded with:

#+begin_src shell :exports both :results output code :wrap "example"
./install/bin/cogs-demo || /bin/true
#+end_src

#+RESULTS:
#+begin_example
2020-Jun-25 12:17:22,617 INFO [main(...) at unknown/demo/cogs-demo.cpp:12] usage: ./install/bin/cogs-demo <uri>
#+end_example

** Regenerate
   :PROPERTIES:
   :CUSTOM_ID: regen
   :END:


The demo relies on generated code which is committed to the repository
to reduce the build-time dependency of ~cogs~.  If the additional
prerequisites are satisfied, it may be regenerated:

#+begin_src shell :exports both :results output code :wrap example
./demo/generate.sh
#+end_src

#+RESULTS:
#+begin_example
#+end_example

Below we will look more at what this script does.

* Run
  :PROPERTIES:
  :CUSTOM_ID: run
  :END:


The demo provides a ready made ~cogs~ configuration stream file:

#+begin_src shell :exports both :results output code :wrap example
./install/bin/cogs-demo file://demo/demo-config.json | sed -e 's/^[^]]*\]//'
#+end_src

#+RESULTS:
#+begin_example
#+end_example





The ~sed~ is simply to remove ERS output augmentation more appropriate
to log files.  The output shows the configuration driving the
construction of a "node" and a "component" (the "source") followed by
their configuration.  When the node is configured it makes a (dummy)
"port" and hands that C++ object to the source.  The next section on
the demo framework describes these terms.  They are not inherent to
~cogs~ itself, just this demo but they represent typical code patterns.

* Framework
  :PROPERTIES:
  :CUSTOM_ID: framework
  :END:

It is possible to use ~cogs~ in a variety of patterns.  This demo
illustrates one particular pattern such as may be used in an
"application framework".  This pattern might be named something like
"factory configuration".  It provides for a highly flexible,
configuration-driven method for "aggregating" an application instance
from a set of factory-instantiated components.  

** Demo configuration stream
   :PROPERTIES:
   :CUSTOM_ID: config-stream
   :END:


The construction of the demo application and its configuration is
driven by a ~cogs~ configuration stream.  The stream is composed of a
sequence of pairs of configuration objects.  The first object in each
pair co responds to a fixed type of ~demo::ConfigurableBase~.  It
provides information required to locate a component instance.  The
second object in a pair corresponds to the configuration of that
component instance.

The stream is illustrated as:

|---------------------------------------|
| component 1: ~democfg::ConfigHeader~    |
|---------------------------------------|
| component 1: corresponding cfg object |
|---------------------------------------|
| ...                                   |
|---------------------------------------|
| component N: ~democfg::ConfigHeader~    |
|---------------------------------------|
| component N: corresponding cfg object |
|---------------------------------------|

** Dispatching configuration
   :PROPERTIES:
   :CUSTOM_ID: dispatch-config
   :END:

The ~ConfigHeader~ provides two attributes:

- implementation identifier :: this is some name associated with a
  construction method for an implementation of ~ConfigurableBase~.  This
  identifier is some simple name, likely derived from the component's
  C++ class name.

- instance identifier :: multiple instances of one component may be
  constructed and this identifier keeps then distinct.

The main application walks the ~cogs::Stream~ using the ~ConfigHeader~ to
retrieve an instance from the demo factory.  It then reads the next
object from the ~cogs::Stream~ and passes it to the component's
~configure()~ method.  When the stream is exhausted the demo app simply
exits.  A real app would of course go on to some other phase of
execution.

** Non-trivial application patterns
   :PROPERTIES:
   :CUSTOM_ID: app-pattern
   :END:

The demo adds some non-trivial complexity by considering two types of
configurable objects:

- node :: an object which has a collection of /ports/ such may be
  associated with sockets.  The demo keeps ports as dummies but they
  represent some shared resource that is non-trivial to construct.

- components :: an object which is configurable and may also want to
  use /ports/.  There is only a single component in the demo called a
  "source".  It represents some arbitrary "code execution unit" aka
  "user module".

The *node* is really just another component but it is called out special
here as it uses the factory to locate instances of other components,
as directed by its configuration and in order to deliver fully formed
"ports".  A component must inherit from
~demo::PortuserBase~ and be listed in the node's configuration in order
to receive its ports.

This pattern is a mock of a real implementation found in [[https://github.com/brettviren/zio][ZIO]] which
uses a [[https://brettviren.github.io/zio/node.html][zio::Node]] to create and link [[https://brettviren.github.io/zio/port.html][zio::Port]] instances either
directly or automatically with the help of a [[https://brettviren.github.io/zio/peer.html][zio::Peer]] performing
distributed network discovery.


* Codegen
  :PROPERTIES:
  :CUSTOM_ID: codegen
  :END:

This section provides a tour of code generation part of the demo.  The
tour focuses on the short [[file:demo/generate.sh][generate.sh]] script.  This script runs its
commands from the [[file:demo/][demo/]] directory and that should be taken into
consideration when reading excerpts of the script which are shown
below.

** Generating C++ 
   :PROPERTIES:
   :CUSTOM_ID: gen-struct
   :END:

User code should not be burdened with validating and interpreting a
configuration byte stream or even interpreting dynamic C++ object like
~nlohman::json~.  Instead, with ~cogs~ the user code receives a fully
typed C++ ~struct~, thus guaranteeing at least valid object structure.

The code for a ~struct~ and its serialization methods is generated from
an application schema realized with the Avro domain schema and a few
extra bits of information.  This information for the demo's "node",
"comp" and "head" application schema is brought together in the short
file [[file:demo/demo-codegen.jsonnet][demo-codegen.jsonnet]] included here:

#+include: demo/demo-codegen.jsonnet src jsonnet

To produce the set of six header files (one for structs and one for
their serialization for each application schema) one runs:

#+call: grep("Codegen", "demo/generate.sh", 1) :wrap "src shell"

#+RESULTS:
#+begin_src shell
echo "Codegen for structs and serialization"
moo render-many demo-codegen.jsonnet
#+end_src

** Configuration stream
   :PROPERTIES:
   :CUSTOM_ID: gen-cfg
   :END:

We finally generate an example ~cogs~ configuration stream in the form
of a JSON file holding an array.  This file is created from Jsonnet by
~moo~:

#+call: grep("Compiling config", "demo/generate.sh", 1) :wrap "src shell"

#+RESULTS:
#+begin_src shell
echo "Compiling configuration to cogs stream file"
moo -D model compile demo-config.jsonnet > demo-config.json
#+end_src

The [[file:demo/demo-config.json][demo-config.json]] file is what was used above to run the demo.  It
is not long and so is included here:

#+include: demo/demo-config.json src json

You can see the paired objects, each preceded by what will be come a
~demo::ConfigHeader~ followed a an object of a specific type
corresponding to the component named in the preceding header.

Note, the choice of ordering is intentional.  It leads to the
construction and configuration of the ~demoSource~ prior to the use of
this component inside the node.  That use calls back to the component
in order to pass in the requested "port" objects.

* Schema
  :PROPERTIES:
  :CUSTOM_ID: schema
  :END:

This section describes how to develop schema.  It first describes the
layer of "application schema" and "abstract base schema".  It then
illustrates the elements of the the latter and walks through an
example of the former.

** Layers
   :PROPERTIES:
   :CUSTOM_ID: schema-layers
   :END:

The demo assumes two layers or schema.  The lowest is called an
"abstract base schema".  Strictly speaking it is a specification of a
set of function names and their arguments.  The demo then provides a
number of implementations of this base schema.  A implementation of a
base schema function then returns a corresponding data structure that
adheres to the schema vocabulary of a particular domain.  

For example, one base schema implementation provides structures
suitable for directly producing Avro schema JSON.  Another provides
structures which adhere to JSON Schema vocabulary.  Another example
given above is one that produces structure that may be applied to a
~message.proto.j2~ template to produce Protobuf ~.proto~ file that can
then be compiled into C++ classes via ~protoc~.

Using these primitive base functions, an application developer writes
the next layer of functions which emit schema that describes the
specific data types required by the developers components.  

The next section describes the functions provided by a base schema
followed by a tour of the application level schema for the
configuration used by the "node" component in the ~cogs~ demo.

** Base schema
   :PROPERTIES:
   :CUSTOM_ID: base-schema
   :END:

The base schema in its abstract form is a set of Jsonnet function
prototypes which are summarized here.  An implementation of an
abstract function is expected to return a description of the type
named by the function in some *domain vocabulary*.  For example the demo
provides one base implemented for the [[file:demo/avro-schema.jsonnet][Avro schema]] domain and one for
that of [[file:demo/json-schema.jsonnet][JSON Schema]].

Domains will differ in what they can meaningfully accept.  This means
that some domains may ignore some arguments to their functions.
Furthermore, some arguments are optional which are indicated by
setting default value to Jsonnet ~null~.  A domain may either provide a
default inside the function body or the argument shall be ignored (no
~null~ values should "leak out" from the functions).

The abstract base schema functions are:

- ~boolean()~ :: a Boolean type

- ~number(dtype, extra={})~ :: a numeric type.  The ~dtype~ argument should
  provide specific type information using Numpy codes (eg ~i4~ for C++
  ~int~, ~u2~ for C++ ~uint16_t~).  The ~extra~ may specify JSON Schema
  constraints.

- ~bytes(encoding=null, media_type=null)~ :: a sequence of byte values

- ~string(patern=null, format=null)~ :: a string type, ~pattern~ and
  ~format~ are JSON Schema arguments specifying a regular expression or
  a named format that a valid string must match.

- ~field(name, type, default=null, doc=null)~ :: an named and typed
  element in the context of a ~record~.  If the type is not scalar (eg,
  is a *record*) then ~type~ should be given as the name of the type.  The
  ~default~ may provide a default *value* of this field.  The ~doc~ provides
  a brief English description of the meaning of the field.

- ~record(name, fields=[], doc=null)~ :: a type which aggregates fields.
  This corresponds to a JSON object or a C++ ~struct~ or ~class~, etc.
  The ~fields~ array is a sequence of objects returned from the ~field()~
  function (from the same domain).

- ~sequence(type)~ :: an ordered sequence holding elements of type ~type~.  

- ~enum(name, symbols, default=null, doc=null)~ :: an enumerated type.
  The ~symbols~ is an array of string literals naming the enumerated
  values.  The ~default~ may specify an enumerated value to be used if
  otherwise not specified.

** Node schema
   :PROPERTIES:
   :CUSTOM_ID: node-schema
   :END:

The concept of a "node" in this demo has been descried above.  Here we
examine the [[file:demo/node-schema.jsonnet][node-schema.jsonnet]] file as an example of an
application-level schema.  

First we look at the high-level structure of the file:

#+begin_src jsonnet
function(schema) {
    // defines types
    types: [ typeA, typeB, ...]
}
#+end_src

This Jsonnet compiles down to a single function object which takes the
argument ~schema~ which provides a set of base schema functions such as
described in the previous sections.  The primary result of this
function is to return a Jsonnet object (~{...}~) which contains an
attribute ~types~ holding an array of objects describing types
constructed through calls to functions provided by ~schema~.

Looking at the first few lines:

#+call: grep("local re", "demo/node-schema.jsonnet", 2) :wrap "src jsonnet"

#+RESULTS:
#+begin_src jsonnet
local re = import "re.jsonnet";
function(schema) {
    local ident = schema.string("Ident", pattern=re.ident_only),
#+end_src

Here, the Jsonnet file [[file:demo/re.jsonnet][re.jsonnet]] is imported and is provided by ~moo~.
It contains a set of regular expressions that are to be used to
constrain the validity of strings in the schema.  For example it
begins with:

#+begin_src jsonnet
{
    // Basic identifier (restrict to legal C variable nam)
    ident: '[a-zA-Z][a-zA-Z0-9_]*',
    ident_only: '^' + self.ident + '$',
#+end_src

Thus a string with ~pattern~ set to ~ident~ may be validated to hold only
a limited alphanumeric content.  

Back to ~node-shcema.jsonnet~, we ~ident~ defined as a string with a
pattern ~re.ident_only~ as a Jsonnet ~local~.  This means the variable is
temporary and known only in the scope of the object.  This lets it be
referred to simply by the name ~ident~ later.

Next we find an example of an ~enum~ and a ~record~ which describe a "link":

#+call: grep("local address", "demo/node-schema.jsonnet", 9) :wrap "src jsonnet"

#+RESULTS:
#+begin_src jsonnet
    local address = schema.string("Address", pattern=re.uri),

    local ltype = schema.enum("LinkType", ["bind","connect"], default="bind",
                         doc="How a port links to an address"),
    local link = schema.record("Link", fields= [
        schema.field("linktype", ltype,
                doc="The socket may bind or connect the link"),
        schema.field("address", address, 
                doc="The address to link to")
    ], doc="Describes how a single link is to be made"),
#+end_src

A "link" is intended to generalize the concept of relationship of a
socket and an address.  The ~LinkType~ represents one of the two allowed
link mechanism (a ~bind()~ or a ~connect()~).  Note how the Jsonnet
representation of this type is used to further define the ~Link~ along
with the representation of a string type ~Address~.  In a real system
like ZIO, an address is in the form of a URL like ~tcp://127.0.0.1:5678~
for direct ZeroMQ addressing or ~zyre://nodename/portname~ for automated
network peer discovery.

Next we come to a "port":

#+call: grep("local port", "demo/node-schema.jsonnet", 5) :wrap "src jsonnet"

#+RESULTS:
#+begin_src jsonnet
    local port = schema.record("Port", fields=[
        schema.field("ident", ident,
                doc="Identify the port uniquely in th enode"),
        schema.field("links", linklist, 
                doc="Describe how this port should link to addresses"),
    ], doc="A port configuration object",),
#+end_src

A port is another ~record~ with an identifier (name) of a type ~ident~
which we defined above.  That is, a string which may be validated
against a regular expression.  The second field is ~links~ which is a
~sequence~.  In ZIO a port corresponds to a ZeroMQ socket which may have
a multitude of both ~bind()~ and ~connect()~ links.

Next we define the part of the node configuration which describes what
a node needs to know in order to interact with a component:

#+call: grep("local comp", "demo/node-schema.jsonnet", 9) :wrap "src jsonnet"

#+RESULTS:
#+begin_src jsonnet
    local comp = schema.record("Comp", fields=[
        schema.field("ident", ident, 
                doc="Identify copmponent instance uniquely in the node"),
        schema.field("type_name", ident, 
                doc="Identify the component implementation"),
        schema.field("portlist", portlist,
                doc="Identity of ports required by component"),
        schema.field("config", extra_config,
                doc="Per instance configuration string used by node")
    ], doc="An object used by the node to partly configure a component"),
#+end_src

The first two fields are identifiers used to look up the component
using the factory (ie, matching what is also provided to ~main()~ in the
header object).  The ~portlist~ is a sequence of identifiers which must
mach those used in defining a ~Port~ above.  This required consistency
can be enforced by Jsonnet when generating actual configuration
objects as described in the next section.  And, finally, an arbitrary
extra string is provided which the demo does not actually use for
anything.  It may be used by the node to interpret some special action
on the component (eg, "ignore" or something).

Penultimately, we get to the top level of the "node" schema:

#+call: grep("local node", "demo/node-schema.jsonnet", 7) :wrap "src jsonnet"

#+RESULTS:
#+begin_src jsonnet
    local node = schema.record("Node", fields=[
        schema.field("ident", ident,
                doc="Idenfity the node instance"),
        schema.field("portdefs", portdefs,
                doc="Define ports on the node to be used by components"),
        schema.field("compdefs", compdefs,
                doc="Define components the node should instantiate and configure"),
    ], doc="A node configures ports and components"),
#+end_src

This defines a ~record~ type ~Node~ with fields meant to hold the port and
component definitions.

And, finally, the "return" value collects all the types:

#+call: grep("types:", "demo/node-schema.jsonnet", 2) :wrap "src jsonnet"

#+RESULTS:
#+begin_src jsonnet
    types: [ ident, address, ltype, link, linklist,
             port, portlist, portdefs,
             extra_config, comp, compdefs, node ],
#+end_src

** Configuration objects

The demo generates a configuration stream (JSON array) with
[[file:demo/demo-config.jsonnet][demo-config.jsonnet]].  It defines two top level attributes: ~model~ which
provides the configuration sequence and ~schema~ which is a sequence of
objects in JSON Schema vocabulary.

The configuration stream can first be validated with ~moo~:

#+begin_example
moo -S schema -D model \
   validate --sequence \
   -s demo/demo-config.jsonnet \
      demo/demo-config.jsonnet
#+end_example

This tells ~moo~ to assume both the named model and schema are actually
each a sequence of model or schema, respectively and then to test each
pair one by one.  Success is marked by a ~null~ for each.  Failure will
be greeted with information indicating where the model is invalid.

When valid the model may be compiled into a form that can be consumed
as a ~cogs~ stream:

#+begin_example
moo -D model  compile demo/demo-config.jsonnet > demo/demo-config.json
#+end_example

One may now go back to the section above [[Run]] where we ran this configuration through the ~cogs-demo~ application.