Getting Started

The following tutorial demonstrates basic usage of ConfigTree. It covers all features, but does not explain them in details. It is enough to start using ConfigTree within your project, however more detailed explanation will be given in the next section.

Installation

There is nothing unusual, just use Pip:

$ pip install configtree

Warming up

Let’s create a separate directory, for our experiments:

$ mkdir configs
$ cd configs

Then create some dummy data:

$ touch test.yaml
$ echo "x.y: 1" >> test.yaml
$ echo "x:" >> test.yaml
$ echo "  z: 2" >> test.yaml

The test file should look like this:

x.y: 1
x:
  z: 2

Now, let’s run ctdump shell command to see how it loads our file:

$ ctdump json
{"x.z": 2, "x.y": 1}

Let’s play with its arguments:

$ ctdump json --json-indent=4 --json-sort
{
    "x.y": 1,
    "x.z": 2
}
$ ctdump json --json-indent=4 --json-sort --json-rare
{
    "x": {
        "y": 1,
        "z": 2
    }
}

And finally, let’s load the file within Python code and fiddle with the result:

>>> from configtree import Loader
>>> load = Loader()
>>> tree = load('.')
>>> tree
Tree({'x.y': 1, 'x.z': 2})
>>> tree['x']
BranchProxy('x'): {'z': 2, 'y': 1}
>>> tree['x.y']
1
>>> tree == {'x.y': 1, 'x.z': 2}
True
>>> tree['x'] == {'y': 1, 'z': 2}
True

You can see that:

  • ConfigTree flattens the file on loading, i.e. there is no difference between dot-separated keys and nested mappings:

    # This is identical...
    x.y: 1
    x.z: 2
    
    # ...to this
    x:
        y: 1
        z: 2
    

    See configtree.tree.flatten() for details.

  • ConfigTree uses configtree.tree.Tree to store the result. This class provides dictionary interface and can be used wherever built-in dict is expected. It also provides ability to get branches, i.e. expose intermediate keys. That is why it named “Tree”.

  • configtree.loader.Loader is used to load configtree.tree.Tree object from files. The following tutorial is devoted to its features.

  • ctdump shell command can be used to dump tree into JSON, so it can be useful to build configuration for programs written in other programming languages.

Now remove the test file and move on to the real world example.

Safe defaults

Let’s imagine that we develop a web service, which consists of two web applications: frontend and REST API. First of all, we need simple configurations for development and production environments. These two configurations will have lots of common parameters. So it will be better to create a default configuration, that should be updated by environment-specific options.

However, the default configuration must contain safe default parameters. Because it is always possible that someone forget to override default value in the production environment. Nobody wants to go live with weak cryptographic keys, for instance.

Using ConfigTree it is possible to mark keys as required. So the loader will raise an error, if such keys have not been overridden.

Create default.yaml with the following content:

api:                                            # API configuration
    host: "!!! API host name"
    port: 80
    db:
        driver: "mysql"
        user: "!!!"
        password: "!!!"
        name: "demo_db"
    secret: "!!! Web tokens encryption key"
    logging: "error"
frontend:                                       # Frontend configuration
    host: "!!! Frontend host name"
    port: 80
    js:
        merge: yes
        minify: yes
    css:
        merge: yes
        minify: yes
    templates:
        reload: no
        cache: yes
    logging: "error"

Now let’s test it:

$ ctdump json
configtree [ERROR]: Undefined required key <api.db.password>
configtree [ERROR]: Undefined required key <api.db.user>
configtree [ERROR]: Undefined required key <api.host>: API host name
configtree [ERROR]: Undefined required key <api.secret>: Web tokens encryption key
configtree [ERROR]: Undefined required key <frontend.host>: Frontend host name

As you can see, the loader reports error for each key, marked with ”!!!”. If you run loader programmatically, an exception of configtree.loader.ProcessingError will be raised.

Move on and see how to override the values in the environment-specific configuration.

Loading environment-specific configuration

Let’s create production configuration in file env-prod.yaml with the following content:

api:
    host: "api.example.com"
    db:
        user: "demo_user"
        password: "pa$$w0rd"        # Password must be strong
    secret: "s3cre7"                # As well as cryptographic key :)
frontend:
    host: "www.example.com"

Now we should “say” to the loader to load this file only in the production environment. The part of loader that responds to get list of files to load is Walker. To change its default behavior, we should manually create configtree.loader.Walker object and pass it into configtree.loader.Loader:

>>> from configtree import Loader, Walker
>>> walk = Walker(env='prod')
>>> load = Loader(walk=walk)

To make it work in ctdump shell command, create loaderconf.py file with the following content:

import os

from configtree import Walker

walk = Walker(env=os.environ['ENV_NAME'])

And test it:

$ ENV_NAME=prod ctdump json
{...}

Using hierarchical environments

Now let’s think about development environments. Our imaginable project consists of two parts: API and frontend. So our imaginable team should consist of two sub-teams: API developers and frontend developers.

The frontend team does not care about backend logs, but they want to have debug logging level on frontend. They also work on templates, and want to switch off caching and switch on reloading options, and so on. While the backend team needs slightly different configuration.

So let’s create a directory for development configuration with three files:

env-dev/                    # Development configuration directory
    common.yaml             # Common development options
    env-api.yaml            # API team development options
    env-frontend.yaml       # Frontend team development options

And play with ENV_NAME. Here we use --verbose option of ctdump shell command to get list of loaded files:

$ ENV_NAME=dev ctdump json --verbose
configtree [INFO]: Walking over "/path/to/configs"
configtree [INFO]: Loading "defaults.yaml"
configtree [INFO]: Loading "env-dev/common.yaml"

$ ENV_NAME=dev.api ctdump json --verbose
configtree [INFO]: Walking over "/path/to/configs"
configtree [INFO]: Loading "defaults.yaml"
configtree [INFO]: Loading "env-dev/common.yaml"
configtree [INFO]: Loading "env-dev/env-api.yaml"

$ ENV_NAME=dev.frontend ctdump json --verbose
configtree [INFO]: Walking over "/path/to/configs"
configtree [INFO]: Loading "defaults.yaml"
configtree [INFO]: Loading "env-dev/common.yaml"
configtree [INFO]: Loading "env-dev/env-frontend.yaml"

As you can see, environments can be organized in hierarchy, where the most common configuration options are defined at the root, and the most specific—at the leafs.

Templates and evaluable expressions

Sometimes you need to calculate some values in your configuration.

For example, let’s add some endpoint URLs to the API configuration. Edit default.yaml file and add the following:

api:
    # Previously added API configuration goes here
    endpoints:
        index: "%>> http://%(api.host)s:%(api.port)s"
        login: "%>> %(api.endpoints.index)s/login"
        logout: "%>> %(api.endpoints.index)s/logout"
frontend:
    # Previously added frontend configuration goes here

In the result production configuration it will look like this:

{
    "api": {
        "endpoints": {
            "index": "http://api.example.com:80",
            "login": "http://api.example.com:80/login",
            "logout": "http://api.example.com:80/logout"
        }
    }
}

Such expressions are calculated after whole configuration has been loaded. So you can use values that are defined after the expression, or even defined in another file.

You can also use expressions similar to standard Python console. And even add your own syntactic sugar. See Updater and Post-processor sections of the manual for details.