Advanced Usage

Loading Overview

The loading of configuration is done in the following steps.

  1. Get the list of files to load.
  2. Load data from each file.
  3. Put the data into result tree object.
  4. Post-process the result object.
  5. Format the result object.

The loading itself is done by configtree.loader.Loader. It performs first four steps. If you use configtree.loader.Loader programmatically, it is probably what you need. The last fifth step is performed by ctdump shell command before printing the result.

Walker

configtree.loader.Walker object is used to get list of files to load. The walker responds for skipping ignored and unsupportable files, and sort the rest of ones by their priority.

To pass walker into configtree.loader.Loader programmatically use:

from configtree import Loader, Walker

load = Loader(walk=Walker())

To specify walker for ctdump shell command, create walk object in loaderconf.py

from configtree import Walker

walk = Walker()

Unsupportable and ignored files

The walker skips file, if its name starts with underscore or dot chat; or its extension is not in configtree.source.map, i.e. there is no loader for the file format.

Environment specific files and directories

Additionally, the walker can skip or include environment specific files. The name of environment specific file (or directory) starts with env- prefix. The rest part of the name is considered as an environment name. To control what to include, pass env argument into configtree.loader.Walker constructor.

For example, here is the directory of configuration:

configs/
    defaults.yaml           # Default configuration
    env-dev/
        defaults.yaml       # Default developing configuration
        env-john.yaml       # John's personal configuration
        env-jane.yaml       # Jane's personal configuration
    env-prod.yaml           # Production configuration

This is how we can get files of different environments:

>>> walk = Walker(env='prod')       # Production configuration
>>> for path in walk('./configs'): print(path)
/full/path/to/configs/defaults.yaml
/full/path/to/configs/env-prod.yaml

>>> walk = Walker(env='dev')        # Default developing configuration
>>> for path in walk('./configs'): print(path)
/full/path/to/configs/defaults.yaml
/full/path/to/configs/env-dev/defaults.yaml

>>> walk = Walker(env='dev.john')   # John's personal developing configuration
>>> for path in walk('./configs'): print(path)
/full/path/to/configs/defaults.yaml
/full/path/to/configs/env-dev/defaults.yaml
/full/path/to/configs/env-dev/env-john.yaml

Final files

If name of a file (or directory) starts with final sting, the file will be placed at the end of result list of files.

The order of files

The result list of files is sorted in the following order:

  1. Regular file, priority 30.
  2. Regular directory, priority 31.
  3. Environment file, priority 50.
  4. Environment directory, priority 51.
  5. Final directory, priority 100.
  6. Final file, priority 101.

Additionally, files are alphabetically sorted within their groups.

For example, we got this configuration directory:

configs/
    defaults.yaml
    common/
        foo.yaml
        bar.yaml
    env-dev/
        defaults.yaml
        env-john.yaml
        env-jane.yaml
    env-dev.yaml
    env-prod.yaml
    final/
        foo.yaml
        bar.yaml
    final-foo.yaml
    final-bar.yaml

If env is equal to dev.jane, the files from the list above will be returned in the following order:

defaults.yaml           # Regular file
common/bar.yaml         # Regular directory.  Regular file bar.yaml goes before foo.yaml,
common/foo.yaml         # because of alphabetical sort.
env-dev.yaml            # Environment file
env-dev/defaults.yaml   # Regular file from environment directory
env-dev/env-jane.yaml   # Environment file the same directory
final/bar.yaml          # Regular file from final directory
final/foo.yaml
final-bar.yaml          # Final file
final-foo.yaml

Extending walker

If you want to add some features to the walker, you can subclass it and add some additional workers to its pipeline (see configtree.loader.Pipeline).

Each worker accepts single argument—configtree.loader.File object, and returns priority for the passed file. None value means, that the worker passes the file to the next worker. -1 value means, that the file must be skipped. Other means priority and is used to sort files in the result list.

For example, let’s add support of initial files as opposite of final ones, that should be at the beginning of the result list.

from configtree import Walker, Pipeline

class MyWalker(Walker):

    @Pipeline.worker(20)   # Place worker between ``ignored`` and ``final``
    def initial(self, fileobj):
        if not fileobj.name.startswith('init'):
            return None
        return 11 if fileobj.isdir else 10

Source

Loading data from files is done by configtree.source module. The module provides configtree.source.map that stores map of file extensions to loaders. The map is used by configtree.loader.Loader to load data from files. The following formats are supported out of the box:

The map is filled scanning entry points configtree.source. So that it is extensible by plugins. Ad hoc loader can be also defined within loaderconf.py module. The loader itself should be a callable object, which accepts single argument—opened file, and returns collections.OrderedDict.

Example:

from collections import OrderedDict

def from_xml(data):
    # Do something with ``data`` file
    return OrderedDict(...)

Define plugin within setup.py file:

entry_points="""\
[configtree.source]
.xml = plugin.module.name:from_xml
"""

Or define ad hoc loader within loaderconf.py:

from configtree import source

source.map['.xml'] = from_xml

Updater

configtree.loader.Updater object is used to put loaded data into the result object of configtree.loader.Loader.__call__() method. The updater responds for adding syntactic sugar into regular data that come from YAML, JSON, and other files.

Updating process can be basically illustrated by the following code:

for key, value in loaded_data.items():
    # result_tree[key] = value

    # Instead of simple assignment above, we call updater.
    # So that extending updater, we can change the default behavior.
    updater(result_tree, key, value)

To pass updater into configtree.loader.Loader programmatically use:

from configtree import Loader, Updater

load = Loader(update=Updater())

To specify updater for ctdump shell command, create update object in loaderconf.py

from configtree import Updater

update = Updater()

Built-in syntactic sugar

Out of the box the updater supports the following:

Deferred expressions

Formatting or evaluating value is replaced by configtree.loader.Promise object. The object stores callable object, that should be called after loading process has been done. So that all expressions are calculated on post-processing step.

Extending updater

If you want to add some features to the updater, you can subclass it and add some additional workers to its pipeline (see configtree.loader.Pipeline).

Each worker accepts single argument—configtree.loader.UpdateAction object. Workers can transform configtree.loader.UpdateAction.key, configtree.loader.UpdateAction.value, or configtree.loader.UpdateAction.update attributes to change default updating behavior.

For example, let’s add support of some template language.

from configtree.loader import Updater, Pipeline, ResolverProxy

class MyUpdater(Updater):

    @Pipeline.worker(75)   # Place worker after ``eval_value`` and ``required_value``
    def template_value(self, action):
        if not isinstance(action.value, string) or \
           not action.value.startswith('template>> '):
            return
        value = action.value[len('template>> '):].strip()
        action.value = action.promise(
            lambda: template(value, ResolverProxy(action.tree, action.source))
        )

Here we wrapped configtree.tree.Tree object by configtree.loader.ResolverProxy. The proxy is helper object that resolves configtree.loader.Promise objects on fly. So that the expression could use other deferred expressions.

We also create configtree.loader.Promise object using configtree.loader.UpdateAction.promise(). Because the method wraps original expression by exception handler that adds useful debug info into raised exceptions.

Post-processor

configtree.loader.PostProcessor object is used to finalize configtree.tree.Tree object returned by configtree.loader.Loader. The post-processor responds for resolving deferred expressions (configtree.loader.Promise) and check for undefined required keys (configtree.loader.Updater.required_value()). It is a good place for custom validators, see Extending post-processor.

To pass post-processor into configtree.loader.Loader programmatically use:

from configtree import Loader, PostProcessor

load = Loader(postprocess=PostProcessor())

To specify post-processor for ctdump shell command, create postprocess object in loaderconf.py

from configtree import PostProcessor

postprocess = PostProcessor()

Extending post-processor

If you want to add some features to the post-processor, you can subclass it and add some additional workers to its pipeline (see configtree.loader.Pipeline).

Each worker accepts three arguments: configtree.tree.Tree object, current processing key, and value. It should return None, or error message as a string (or as an object that has human readable string representation). These message will be accumulated and thrown within single configtree.loader.ProcessingError exception at the end of processing.

For example, let’s add validator of port number values. If key ends with .port, it must be int value greater than zero.

from configtree import PostProcessor, Pipeline

class MyPostProcessor(PostProcessor):

    @Pipeline.worker(100)   # Place worker after ``check_required``
    def validate_port(self, tree, key, value):
        if not key.endswith('.port'):
            return None
        try:
            value = int(value)
        except ValueError:
            return (
                '%s: type ``int`` is expected, but %r of type ``%s`` is given'
                % (key, value, type(value).__name__)
            )
        if value < 0:
            return '%s: port number should be greater than zero, but %r is given' % value
        tree[key] = value

Formatter

Formatting of configtree.tree.Tree objects is done by configtree.formatter module. The module provides configtree.formatter.map that stores map of format names to formatters. This formatters are used by ctdump shell command to print result. The following formats are supported out of the box:

The map is filled scanning entry points configtree.formatters. So that it is extensible by plugins. Ad hoc formatter can be also defined within loaderconf.py module. The formatter itself should be a callable object, which accepts single argument—configtree.tree.Tree object, and returns string. Optional keyword arguments are possible too. However, if you want to specify these arguments via ctdump shell command, you should use decorator configtree.formatter.option().

Example:

from configtree import formatter

@formatter.option(
    'indent', default=None, type=int, metavar='<indent>',
    help='indent size (default: %(default)s)'
)
def to_xml(tree, indent=None):
    # See ``demo/loaderconf.py`` for complete working code of the formatter
    result = ...  #  Do something with tree
    return result

Define plugin within setup.py file:

entry_points="""\
[configtree.formatter]
xml = plugin.module.name:to_xml
"""

Or define ad hoc formatter within loaderconf.py:

from configtree import formatter

formatter.map['xml'] = to_xml

ctdump shell command

Command line utility to load configtree.tree.Tree objects and dump them using available formatters.

You can use it to build JSON files, that can be loaded by progams written in any programming language, that supports parsing JSON.

# Somewhere in your build script
ctdump json --path path/to/config/sources > path/to/build/config.json

You can build only a part of configuration specifying branch:

ctdump json --path path/to/config/sources --branch app.http > path/to/build/server.json
ctdump json --path path/to/config/sources --branch app.db > path/to/build/database.json

The special formatter for shell scripts helps to use configuration within Bash scripts. For example, you want to use database credentials:

backup_db() {
    eval "$( ctdump shell --branch app.db --shell-prefix 'local ' )"
    # Output of ctdump will look like this:
    #   local username='dbuser'
    #   local password='qwerty'
    #   local database='mydata'

    mysqldump --user="$username" --password="$password" "$database" > dump.sql
}

To get full help of the command run:

ctdump --help

loaderconf.py

The module is used to specify arguments for configtree.loader.Loader. It is placed at the root of configuration files, and usually used by ctdump shell command to create its loader by configtree.loader.Loader.fromconf().

It is also a good place for ad hoc source loaders and formatters.

Here is an example of the module:

import os

from configtree import Walker, Updater
from configtree import formatter

# Create ``walk`` and ``update`` that will be used by ``Loader``.
update = Updater(namespace={'os': os})
walk = Walker(env=os.environ['ENV_NAME'])


# Create ad hoc formatter
@formatter.option(
    'indent', default=None, type=int, metavar='<indent>',
    help='indent size (default: %(default)s)'
)
def to_xml(tree, indent=None):
    """ Dummy XML formatter """

    def get_indent(level):
        if indent is None:
            return ''
        else:
            return ' ' * indent * level

    result = ['<configtree>']
    for key, value in tree.items():
        result.append('%s<item>' % get_indent(1))
        result.append('%s<key>%s</key>' % (get_indent(2), key))
        result.append(
            '%s<value type="%s">%s</value>' % (
                get_indent(2),
                type(value).__name__,
                value,
            )
        )
        result.append('%s</item>' % get_indent(1))
    result.append('</configtree>')
    if indent is None:
        return ''.join(result)
    else:
        return os.linesep.join(result)

formatter.map['xml'] = to_xml