Advanced Usage¶
Loading Overview¶
The loading of configuration is done in the following steps.
- Get the list of files to load.
- Load data from each file.
- Put the data into result tree object.
- Post-process the result object.
- Format the result object.
The loading itself is done by configtree.loader.Loader
. It performs
first four steps. If you use configtree.loader.Loader
programmatically,
it is probably what you need. The last fifth step is performed by ctdump shell command
before printing the result.
Walker¶
configtree.loader.Walker
object is used to get list of files to load.
The walker responds for skipping ignored and unsupportable files, and sort
the rest of ones by their priority.
To pass walker into configtree.loader.Loader
programmatically use:
from configtree import Loader, Walker
load = Loader(walk=Walker())
To specify walker for ctdump shell command, create walk
object in loaderconf.py
from configtree import Walker
walk = Walker()
Unsupportable and ignored files¶
The walker skips file, if its name starts with underscore or dot chat;
or its extension is not in configtree.source.map
,
i.e. there is no loader for the file format.
Environment specific files and directories¶
Additionally, the walker can skip or include environment specific files.
The name of environment specific file (or directory) starts with env-
prefix.
The rest part of the name is considered as an environment name. To control
what to include, pass env
argument into configtree.loader.Walker
constructor.
For example, here is the directory of configuration:
configs/
defaults.yaml # Default configuration
env-dev/
defaults.yaml # Default developing configuration
env-john.yaml # John's personal configuration
env-jane.yaml # Jane's personal configuration
env-prod.yaml # Production configuration
This is how we can get files of different environments:
>>> walk = Walker(env='prod') # Production configuration
>>> for path in walk('./configs'): print(path)
/full/path/to/configs/defaults.yaml
/full/path/to/configs/env-prod.yaml
>>> walk = Walker(env='dev') # Default developing configuration
>>> for path in walk('./configs'): print(path)
/full/path/to/configs/defaults.yaml
/full/path/to/configs/env-dev/defaults.yaml
>>> walk = Walker(env='dev.john') # John's personal developing configuration
>>> for path in walk('./configs'): print(path)
/full/path/to/configs/defaults.yaml
/full/path/to/configs/env-dev/defaults.yaml
/full/path/to/configs/env-dev/env-john.yaml
Final files¶
If name of a file (or directory) starts with final
sting, the file will be
placed at the end of result list of files.
The order of files¶
The result list of files is sorted in the following order:
- Regular file, priority
30
. - Regular directory, priority
31
. - Environment file, priority
50
. - Environment directory, priority
51
. - Final directory, priority
100
. - Final file, priority
101
.
Additionally, files are alphabetically sorted within their groups.
For example, we got this configuration directory:
configs/
defaults.yaml
common/
foo.yaml
bar.yaml
env-dev/
defaults.yaml
env-john.yaml
env-jane.yaml
env-dev.yaml
env-prod.yaml
final/
foo.yaml
bar.yaml
final-foo.yaml
final-bar.yaml
If env
is equal to dev.jane
, the files from the list above will be
returned in the following order:
defaults.yaml # Regular file
common/bar.yaml # Regular directory. Regular file bar.yaml goes before foo.yaml,
common/foo.yaml # because of alphabetical sort.
env-dev.yaml # Environment file
env-dev/defaults.yaml # Regular file from environment directory
env-dev/env-jane.yaml # Environment file the same directory
final/bar.yaml # Regular file from final directory
final/foo.yaml
final-bar.yaml # Final file
final-foo.yaml
Extending walker¶
If you want to add some features to the walker, you can subclass it and
add some additional workers to its pipeline (see configtree.loader.Pipeline
).
Each worker accepts single argument—configtree.loader.File
object,
and returns priority for the passed file. None
value means, that the
worker passes the file to the next worker. -1
value means, that the
file must be skipped. Other means priority and is used to sort files in
the result list.
For example, let’s add support of initial files as opposite of final ones, that should be at the beginning of the result list.
from configtree import Walker, Pipeline
class MyWalker(Walker):
@Pipeline.worker(20) # Place worker between ``ignored`` and ``final``
def initial(self, fileobj):
if not fileobj.name.startswith('init'):
return None
return 11 if fileobj.isdir else 10
Source¶
Loading data from files is done by configtree.source
module. The module
provides configtree.source.map
that stores map of file extensions to
loaders. The map is used by configtree.loader.Loader
to load data
from files. The following formats are supported out of the box:
- YAML with extensions
.yaml
and.yml
byconfigtree.source.from_yaml()
; - JSON with extension
.json
byconfigtree.source.from_json()
.
The map is filled scanning entry points configtree.source
. So that it is
extensible by plugins. Ad hoc loader can be also defined within loaderconf.py
module. The loader itself should be a callable object, which accepts single
argument—opened file, and returns collections.OrderedDict
.
Example:
from collections import OrderedDict
def from_xml(data):
# Do something with ``data`` file
return OrderedDict(...)
Define plugin within setup.py
file:
entry_points="""\
[configtree.source]
.xml = plugin.module.name:from_xml
"""
Or define ad hoc loader within loaderconf.py:
from configtree import source
source.map['.xml'] = from_xml
Updater¶
configtree.loader.Updater
object is used to put loaded data into
the result object of configtree.loader.Loader.__call__()
method. The updater
responds for adding syntactic sugar into regular data that come from YAML,
JSON, and other files.
Updating process can be basically illustrated by the following code:
for key, value in loaded_data.items():
# result_tree[key] = value
# Instead of simple assignment above, we call updater.
# So that extending updater, we can change the default behavior.
updater(result_tree, key, value)
To pass updater into configtree.loader.Loader
programmatically use:
from configtree import Loader, Updater
load = Loader(update=Updater())
To specify updater for ctdump shell command, create update
object in loaderconf.py
from configtree import Updater
update = Updater()
Built-in syntactic sugar¶
Out of the box the updater supports the following:
Setup default value, see
configtree.loader.Updater.set_default()
:x: 1 x?: 2 # x == 1 y?: 3 # y == 3
Call specified method of the value, see
configtree.loader.Updater.call_method()
:x: [1, 2, 3] x#append: 4 # x == [1, 2, 3, 4]
Use the value as a template, see
configtree.loader.Updater.format_value()
andconfigtree.loader.Updater.printf_value()
:x: 1 y: foo: 2 # Formatted by ``str.format()`` bar: "$>> {self[x]} {branch[foo]}" # bar == '1 2' # Formatted by ``%`` baz: "%>> %(x)s %(y.foo)s" # baz == '1 2'
Evaluate expressions, see
configtree.loader.Updater.eval_value()
from os import path # Namespace will be passed into expressions update = Updater(namespace={'path': path})
configdir: ">>> self['__dir__']" projectdir: ">>> path.dirname(self['configdir'])"
Setup required values, see
configtree.loader.Updater.required_value()
x: "!!!" y: "!!! Add useful comment here"
Deferred expressions¶
Formatting or evaluating value is replaced by configtree.loader.Promise
object.
The object stores callable object, that should be called after loading process
has been done. So that all expressions are calculated on post-processing step.
Extending updater¶
If you want to add some features to the updater, you can subclass it and
add some additional workers to its pipeline (see configtree.loader.Pipeline
).
Each worker accepts single argument—configtree.loader.UpdateAction
object.
Workers can transform configtree.loader.UpdateAction.key
,
configtree.loader.UpdateAction.value
, or
configtree.loader.UpdateAction.update
attributes to change default
updating behavior.
For example, let’s add support of some template language.
from configtree.loader import Updater, Pipeline, ResolverProxy
class MyUpdater(Updater):
@Pipeline.worker(75) # Place worker after ``eval_value`` and ``required_value``
def template_value(self, action):
if not isinstance(action.value, string) or \
not action.value.startswith('template>> '):
return
value = action.value[len('template>> '):].strip()
action.value = action.promise(
lambda: template(value, ResolverProxy(action.tree, action.source))
)
Here we wrapped configtree.tree.Tree
object by configtree.loader.ResolverProxy
.
The proxy is helper object that resolves configtree.loader.Promise
objects on fly. So that the expression could use other deferred expressions.
We also create configtree.loader.Promise
object using
configtree.loader.UpdateAction.promise()
. Because the method wraps
original expression by exception handler that adds useful debug info into
raised exceptions.
Post-processor¶
configtree.loader.PostProcessor
object is used to finalize
configtree.tree.Tree
object returned by configtree.loader.Loader
.
The post-processor responds for resolving deferred expressions (configtree.loader.Promise
)
and check for undefined required keys (configtree.loader.Updater.required_value()
).
It is a good place for custom validators, see Extending post-processor.
To pass post-processor into configtree.loader.Loader
programmatically use:
from configtree import Loader, PostProcessor
load = Loader(postprocess=PostProcessor())
To specify post-processor for ctdump shell command, create postprocess
object in loaderconf.py
from configtree import PostProcessor
postprocess = PostProcessor()
Extending post-processor¶
If you want to add some features to the post-processor, you can subclass it and
add some additional workers to its pipeline (see configtree.loader.Pipeline
).
Each worker accepts three arguments: configtree.tree.Tree
object,
current processing key
, and value
. It should return None
,
or error message as a string (or as an object that has human readable string representation).
These message will be accumulated and thrown within single configtree.loader.ProcessingError
exception at the end of processing.
For example, let’s add validator of port number values. If key
ends with .port
,
it must be int
value greater than zero.
from configtree import PostProcessor, Pipeline
class MyPostProcessor(PostProcessor):
@Pipeline.worker(100) # Place worker after ``check_required``
def validate_port(self, tree, key, value):
if not key.endswith('.port'):
return None
try:
value = int(value)
except ValueError:
return (
'%s: type ``int`` is expected, but %r of type ``%s`` is given'
% (key, value, type(value).__name__)
)
if value < 0:
return '%s: port number should be greater than zero, but %r is given' % value
tree[key] = value
Formatter¶
Formatting of configtree.tree.Tree
objects is done by configtree.formatter
module.
The module provides configtree.formatter.map
that stores map of format names to
formatters. This formatters are used by ctdump shell command to print result.
The following formats are supported out of the box:
- JSON with name
json
byconfigtree.formatter.to_json()
; - Shell script (Bash) with name
shell
byconfigtree.formatter.to_shell()
.
The map is filled scanning entry points configtree.formatters
. So that it is
extensible by plugins. Ad hoc formatter can be also defined within loaderconf.py
module. The formatter itself should be a callable object, which accepts single
argument—configtree.tree.Tree
object, and returns string. Optional
keyword arguments are possible too. However, if you want to specify
these arguments via ctdump shell command, you should use decorator configtree.formatter.option()
.
Example:
from configtree import formatter
@formatter.option(
'indent', default=None, type=int, metavar='<indent>',
help='indent size (default: %(default)s)'
)
def to_xml(tree, indent=None):
# See ``demo/loaderconf.py`` for complete working code of the formatter
result = ... # Do something with tree
return result
Define plugin within setup.py
file:
entry_points="""\
[configtree.formatter]
xml = plugin.module.name:to_xml
"""
Or define ad hoc formatter within loaderconf.py:
from configtree import formatter
formatter.map['xml'] = to_xml
ctdump
shell command¶
Command line utility to load configtree.tree.Tree
objects and
dump them using available formatters.
You can use it to build JSON files, that can be loaded by progams written in any programming language, that supports parsing JSON.
# Somewhere in your build script
ctdump json --path path/to/config/sources > path/to/build/config.json
You can build only a part of configuration specifying branch:
ctdump json --path path/to/config/sources --branch app.http > path/to/build/server.json
ctdump json --path path/to/config/sources --branch app.db > path/to/build/database.json
The special formatter for shell scripts helps to use configuration within Bash scripts. For example, you want to use database credentials:
backup_db() {
eval "$( ctdump shell --branch app.db --shell-prefix 'local ' )"
# Output of ctdump will look like this:
# local username='dbuser'
# local password='qwerty'
# local database='mydata'
mysqldump --user="$username" --password="$password" "$database" > dump.sql
}
To get full help of the command run:
ctdump --help
loaderconf.py
¶
The module is used to specify arguments for configtree.loader.Loader
.
It is placed at the root of configuration files, and usually used by ctdump shell command
to create its loader by configtree.loader.Loader.fromconf()
.
It is also a good place for ad hoc source loaders and formatters.
Here is an example of the module:
import os
from configtree import Walker, Updater
from configtree import formatter
# Create ``walk`` and ``update`` that will be used by ``Loader``.
update = Updater(namespace={'os': os})
walk = Walker(env=os.environ['ENV_NAME'])
# Create ad hoc formatter
@formatter.option(
'indent', default=None, type=int, metavar='<indent>',
help='indent size (default: %(default)s)'
)
def to_xml(tree, indent=None):
""" Dummy XML formatter """
def get_indent(level):
if indent is None:
return ''
else:
return ' ' * indent * level
result = ['<configtree>']
for key, value in tree.items():
result.append('%s<item>' % get_indent(1))
result.append('%s<key>%s</key>' % (get_indent(2), key))
result.append(
'%s<value type="%s">%s</value>' % (
get_indent(2),
type(value).__name__,
value,
)
)
result.append('%s</item>' % get_indent(1))
result.append('</configtree>')
if indent is None:
return ''.join(result)
else:
return os.linesep.join(result)
formatter.map['xml'] = to_xml