configtree.loader

The module provides utility functions to load tree object from files

class configtree.loader.Loader(walk=None, update=None, postprocess=None, tree=None)

Configuration tree loader

Parameters:
  • walk (Walker) – Walk actor that generates list of files to load
  • update (Updater) – Update actor that implements syntactic sugar
  • postprocess (PostProcessor) – Result tree post processor
  • tree (Tree) – Tree object that should contain result of loading
classmethod fromconf(path)

Creates loader using configuration module loaderconf

Parameters:path (str) – Path to a directory that contains loaderconf
Returns:Ready to use loader object
Return type:Loader
__call__(path)

Loads configuration

Parameters:path (str) – Path to a directory that contains configuration files.
Returns:Result tree object
Return type:Tree

Utilities

class configtree.loader.Pipeline

Utility class that helps to build pipelines

__pipeline__

List of workers that includes each method of the class that marked by worker() decorator. The list is sorted by worker priority. Inactive workers are not included in the list.

static worker(priority, enabled=True)

Decorator that marks method as a worker

Parameters:
  • priority (int) – Priority of the worker
  • enabled (bool) – Whether worker is active or not

Walker

class configtree.loader.Walker(**params)

File walker is used by Loader to get list of files to load.

params

Dictionary that contains all keyword arguments that are passed into constructor. The dictionary is copied into each File object into File.params attribute. This attribute can be used by workers from __pipeline__ to make decisions about the file priority.

Only the env parameter makes sense for environment() worker. All other parameters are simply ignored, but could be used in extensions.

__pipeline__

File processing pipeline. Each File object is passed through the following methods until some method returns int value. The special value -1 means that the passed file should be ignored. Other values mean file priority. For instance, regular files (see regular()) have priorities equal to 30 or 31, and final ones (see final()) have 100 or 101. That means that final files will be at the end of result list of files, and regular files will be at the start of the list.

The list of workers is: [ignored(), final(), environment(), regular()]

__call__(path)

Walks over the path and yields files to load

Parameters:path (str) – Path to walk over
walk(current)

Processes current traversing file

If current is regular file, it will be yielded as is. If it is directory, the list of its files will be prioritized using __pipeline__. Then the list will be sorted using given priorities and each file will be processed using this method recursively.

The method is low level implementation of __call__() and should not be used directly.

Parameters:current (File) – Current traversing file
ignored(fileobj)

Worker that filters out ignored files and directories

The file will be ignored, if its name starts with dot char or underscore, or the file format is not supported by loader.

Parameters:fileobj (File) – Current traversing file
Returns:
  • -1 when the file is ignored one;
  • None when the file is not ignored one.
__priority__ = 10

Examples:

.hidden             # returns -1 (file name starts with dot char)
_ignored            # returns -1 (file name starts with underscore)
unsupported.txt     # returns -1 (txt files are not supported)
other.yaml          # returns None
final(fileobj)

Worker that checks whether current traversing file is final or not.

Final files are processed at the end of current list of files. If the file name starts with “final”, it will be treated as final one.

Parameters:fileobj (File) – Current traversing file
Returns:
  • 100 when the file is final one and it is directory;
  • 101 when the file is final one and it is regular file;
  • None when the file is not final one.
__priority__ = 30

Examples:

final/           # returns 100
final-dir/       # returns 100
other-dir/       # returns None
final.yaml       # returns 101
final-file.json  # returns 101
other.yaml       # returns None
environment(fileobj)

Worker that checks whether current traversing file is environment specific or not.

The file will be treated as environment specific, if its name starts with “env-” string. The rest part of the name (without extension) is treated as environment name. If the environment name does not match to env parameter (see params), then the file will be ignored.

Parameters:fileobj (File) – Current traversing file
Returns:
  • -1 when the file is environment specific, but environment name is not match;
  • 50 when the file is environment specific and it is regular file;
  • 51 when the file is environment specific and it is directory;
  • None when the file is not environment specific one.
__priority__ = 50

Examples:

# params['env'] == "foo.bar"

env-foo.yaml      # returns 50
env-bar.yaml      # returns -1 (environment name is not match)
env-foo/          # returns 51
    env-bar.yaml  # returns 50
    env-baz.yaml  # returns -1
    other.yaml    # returns None
regular(fileobj)

Worker that treats any file as a regular one. The worker should be the last in the __pipeline__, because it does not make any check.

Parameters:fileobj (File) – Current traversing file
Returns:
  • 30 when the file is regular file;
  • 31 when the file is directory.
__priority__ = 1000
class configtree.loader.File(path, name, params)

Represents current traversing file within Walker routine

path

Path of parent directory containing the file

name

File name itself

params

The copy of Walker.params that could be used and transformed by workers from Walker.__pipeline__. See Walker.environment().

fullpath

Full path to the file

isdir

Boolean value that means whether the file is directory or not

isfile

Boolean value that means whether the file is regular file or not

ext

Extension of the file (with leading dot char)

cleanname

Name of the file without its extension

Updater

class configtree.loader.Updater(**params)

Updater is used by Loader to set up key-value pairs into updating tree object. The object extends default updating mechanism adding some syntactic sugar.

params

Dictionary that contains all keyword arguments that are passed into constructor. The attribute can be used by workers.

Only the namespace parameter makes sense for eval_value() worker. All other parameters are simply ignored, but could be used in extensions.

__pipeline__

Transforms UpdateAction object that created by __call__().

Each UpdateAction object is passed through the following methods. Each method can transform UpdateAction.key, UpdateAction.value, or UpdateAction.update, attributes. So that the default behavior of UpdateAction can be changed.

The list of workers is: [set_default(), call_method(), format_value(), printf_value(), eval_value(), required_value()]

__call__(tree, key, value, source)

Updates tree

It creates UpdateAction object. Then pass the object through the __pipeline__. And finally calls the action.

Parameters:
  • tree (Tree) – Updating tree object
  • key (str) – Setting up key
  • value – Setting up value
  • source (str) – Full path to a source file
set_default(action)

Worker that changes default UpdateAction.update from __setitem__ to setdefault if key ends with ”?” char.

It also transforms key, i.e. strips the last char.

Parameters:action (UpdateAction) – Current update action object
__priority__ = 20

Example:

x: 1
x?: 2           # x == 1
y?: 3           # y == 3
call_method(action)

Worker that changes default UpdateAction.update if key contains “#” char.

It splits UpdateAction.key by the char. The left part is set up as the key itself. The right part is used as a method name. It gets value from UpdateAction.tree by the new key and call its method using UpdateAction.value as an argument. If any of the values is instance of Promise, then it will be wrapped by another Promise object. See PostProcessor.resolve_promise().

Parameters:action (UpdateAction) – Current update action object
__priority__ = 30

Example:

foo: [1, 2]
bar: ">>> self['foo'][:]"        # Get copy of the latest `foo`
bar#extend: [5, 6]               # bar == [1, 2, 3, 4, 5, 6]
foo#extend: [3, 4]               # foo == [1, 2, 3, 4]
format_value(action)

Worker that transforms UpdateAction.value that starts with "$>> " (with trailing space char) into formatting expression and wraps it into Promise. See PostProcessor.resolve_promise().

The expression uses str.format(). Current tree and current branch are passed as self and branch names into template. Both are wrapped by ResolverProxy.

Parameters:action (UpdateAction) – Current update action object
__priority__ = 50

Example:

a: "foo"
b:
    x: "bar"
    y: "a = {self[a]!r}, b.x = {branch[x]!r}"
       # == "a = 'foo', b.x = 'bar'"
printf_value(action)

Worker that transform UpdateAction.value that starts with "%>> " (with trailing space char) into formatting expression and wraps it into Promise. See PostProcessor.resolve_promise().

The expression uses printf style, i.e. % operator. UpdateAction.tree wrapped by ResolverProxy is used as a formatting value.

Parameters:action (UpdateAction) – Current update action object
__priority__ = 60

Example:

name: "World"
hello: "%>> Hello %(name)s"     # == "Hello World"
eval_value(action)

Worker that transform UpdateAction.value that starts with ">>> " (with trailing space char) into expression and wraps it into Promise. See PostProcessor.resolve_promise().

The expression uses built-in function eval(). The value of namespace key from params is passed as gloabls argument of eval. UpdateAction.tree is passed as self and UpdateAction.branch is passed as branch names via locals argument of eval. Both are wrapped by ResolverProxy.

Parameters:action (UpdateAction) – Current update action object
__priority__ = 70

Example:

>>> from math import floor
>>> update = Updater(namespace={'floor': floor})
a: ">>> 1 + 2"                         # == 3
b:
    x: 3
    y: ">>> self['a'] * branch['x']"   # == 9
    z: ">>> floor(3.0 / 2)"            # == 1
required_value(action)

Worker that transform UpdateAction.value that starts with "!!!" into an instance of Required. See PostProcessor.check_required().

Parameters:action (UpdateAction) – Current update action object
__priority__ = 80

Example:

foo: "!!!"                              # without comment
bar: "!!! This should be redefined"     # with comment
class configtree.loader.UpdateAction(tree, key, value, source)

Helper object that is used within Updater routine. It represents current update context.

tree

Current updating configtree.tree.Tree object

branch

Property that is used to get current branch from the tree

>>> tree = Tree({'a.x': 1, 'a.y': 2})
>>> action = UpdateAction(tree, 'a.z', 3, '/path/to/src.yaml')
>>> action.branch == tree['a']
True
key

Current setting up key

value

Current setting up value

source

Path to a file processing by Loader. Is used as a part of debug information.

>>> UpdateAction(Tree(), 'foo', 'bar', '/path/to/src.yaml')
<tree['foo'] = 'bar' from '/path/to/src.yaml'>
update

Callable object that represent current update action. By default is equal to default_update().

__call__()

Calls update, i.e. performs update action

promise(deferred)

Helper method that wraps deferred callable by try-except block. It adds self as a first argument to any exception that might be raised from deferred. So that the exception will contain information of what expression from which file is caused it.

Parameters:deferred (callable) – Callable object that should be wrapped by Promise
static default_update(action)

Default value of update. Literally performs:

action.tree[action.key] = action.value
Parameters:action (UpdateAction) – Current action object
class configtree.loader.Promise(deferred)

Represents deferred expression that should be calculated at the end of loading process. See resolve(), Updater.eval_value(), Updater.format_value(), Updater.printf_value(), and PostProcessor.resolve_promise().

Parameters:deferred (callable) – Deferred expression
__call__()

Resolves deferred value, i.e. calls it and returns its result

static resolve(value)

Helper method that resolves passed promises and returns their results. Other values are returned as is.

Parameters:value – Value to resolve
Returns:Resolved promise or value as it is.
>>> Promise.resolve(Promise(lambda: 1))
1
>>> Promise.resolve(2)
2
class configtree.loader.ResolverProxy(tree, source=None)

Helper object that wraps configtree.tree.Tree objects.

It pass each extracted value through resolve(), so that one deferred expression (see Promise) can use another.

If source argument is not None, there will be __file__ and __dir__ keys available.

Parameters:
  • tree (Tree) – Tree object to wrap
  • source (str) – Path to source file
>>> tree = Tree()
>>> proxy = ResolverProxy(tree, '/path/to/src.yaml')
>>> tree['foo'] = Promise(lambda: 1)
>>> tree['bar'] = Promise(lambda: proxy['foo'] + 1)
>>> proxy['foo']
1
>>> proxy['bar']
2
>>> proxy['__file__']
'/path/to/src.yaml'
>>> proxy['__dir__']
'/path/to'
class configtree.loader.Required(key, comment='')

Helper object that indicates undefined required key.

Values of the type are set up by Updater.required_value() and treated as error by PostProcessor.check_required().

Post processor

class configtree.loader.PostProcessor

Post processor is used by Loader to perform final transformations of its result tree object after loading process is finished.

Post processor iterates over passed configtree.tree.Tree object and pass its keys and values through __pipeline__. If any worker of the pipeline returns non None value, this value will be treated as an error. Such errors are accumulated and raised within ProcessingError exception at the end of processing.

__pipeline__

The list of workers is: [resolve_promise(), check_required()]

__call__(tree)

Runs post processor

Parameters:tree (Tree) – A tree object to process
resolve_promise(tree, key, value)

Worker that resolves Promise objects.

Any exception raised within promise expression will not be caught.

Parameters:
  • tree (Tree) – Current processing tree
  • key (str) – Current traversing key
  • value – Current traversing value
__priority__ = 30
check_required(tree, key, value)

Worker that checks tree for raw Required values.

Parameters:
  • tree (Tree) – Current processing tree
  • key (str) – Current traversing key
  • value – Current traversing value
Returns:

  • passed value, if it is an an instance of Required;
  • None for other values.

__priority__ = 50
class configtree.loader.ProcessingError

Exception that will be raised, if post processor gets any error

Deprecated features

configtree.loader.load(path, walk=None, update=None, postprocess=None, tree=None)

Warning

Deprecated in favor of Loader.

Loads configtree.tree.Tree object from files.

A path argument should be a path to the directory containing files to load.

A walk argument, if provided should be a callable, which accepts path argument and returns an iterator over the files to load. By default, a function constructed by make_walk() is used.

An update argument, if provided should be a callable, which accepts three arguments tree, key, value and performs update of tree using key and value pair. By default, a function constructed by make_update() is used.

A postprocess argument, if provided should be a callable, which accepts single argument tree. The loaded tree will be passed here. By default no post processing is done. However, it’s a good place to validate a result tree.

A tree argument, if provided should be a tree-like object, which will be updated during the load process. By default, an empty instance of configtree.tree.Tree is used. It is a right place to put some initial data or use derived class of configtree.tree.Tree.

configtree.loader.loaderconf(path)

Warning

Deprecated in favor of Loader.fromconf().

Reads loader configuration from module loaderconf.

If file of the module does not exists, it will return an empty dictionary. Otherwise, the result will contain walk, update, postprocess, and tree keys.

Usage:

config = load(path, **loaderconf(path))
configtree.loader.make_walk(env='')

Warning

Deprecated in favor of Walker.

Constructs walk function, which will be used by load() one.

The walk function recursively iterates over the directory and yields files to load. It will skip:

  • file, if its extension is not contained in the map of configtree.source.
  • file or directory, if its name starts with “_” underscore or ”.” dot char (hidden one);
  • file or directory, if its name starts with “env-” and the rest part of the name does not match environment name specified by the env argument.

The files are emitted in the following order:

  1. Top level common files.
  2. Common files contained in the nested directories.
  3. Environment specific files.
  4. Files contained in the environment specific directories.
  5. Files contained in the directories prefixed by “final-”.
  6. Top level files prefixed by “final-”.

All files are also sorted using natural sort within their groups.

The environment is specified by env argument. It supports tree-like environment configuration. For instance, you have the following environments: dev (developer’s one), test.staging (for staging server), test.stress (for stress testing), and prod (for production usage). The configuration files may be organized in the following way:

config/
    common/
        common-config.yaml
    env-dev
        dev-config.yaml
    env-prod
        prod-config.yaml
    env-test
        common-test-config.yaml
        env-stress/
            stress-test-config.yaml
        env-staging/
            staging-server-config.yaml
    final-common/
        common-config.yaml

So that, specifying env argument as test.staging, will emit the following files in the exact order:

config/common/common-config.yaml
config/env-test/common-test-config.yaml
config/env-test/env-staging/staging-server-config.yaml
config/final-common/common-config.yaml
configtree.loader.make_update(namespace=None)

Warning

Deprecated in favor of Updater.

Constructs update function, which will be used by load() one.

The update function adds a pinch of syntactic sugar to loading configtree.tree.Tree object from files:

x:
    a: 1
    b: 2
y:
    b: 3

    # If a string value starts with ">>> ", it will be evaluated
    # as a Python expression. Where ``branch`` name refers to
    # the current tree branch, ``tree`` name refers to the whole
    # tree object.  Additionally, names passed via ``namespace``
    # argument can be used within the expression.
    c: ">>> tree['x.a'] + branch['b']"               # c == 4

    # If a string value starts with "$>> ", it will be used as
    # a template string.  It will be formatted used standard ``format``
    # method.  Only ``branch`` and ``tree`` names are available
    # within formatting.
    d: "$>> {tree[x.a]} + {branch[b]} = {branch[c]}" # d == "1 + 3 = 4"

    # If a key ends with "?", the corresponding value will be set
    # only if the key does not exists in the tree.  It is useful
    # to set default values, which could be already set within
    # another file.
    e: [1, 2]
    e?: []                                           # e == [1, 2]
    f?: []                                           # f == []

    # If a key contains "#", the part after that char will be used
    # as a method name.  This method will be called using the value.
    e#append: 3                                      # e == [1, 2, 3]