msrc-appconfig

Type safe composable configuration management in Python

The package orchestrates application configuration from multiple sources: built-in application defaults; .ini, JSON and YAML configuration files; shell environment variables; command line arguments.

All configuration values are checked and converted to proper type allowing for safer use in application code.

The module allows to set up a configuration as a global shared configuration object easily accessible from different parts of application code.

The configuration object can also be serialized and shared across processes locally and in the Cloud.

Getting started

Install the package with pip install msrc-appconfig.

Configuration schema is a class definition with typed attributes and built-in defaults.

import typing

class AppConfig(typing.NamedTuple):
    app_name: str = "Sample"
    repeat: int = 1

To compile configuration object from this schema use gather_config function.

from msrc.appconfig import gather_config

def main(app_config: AppConfig):
    for i in range(app_config.repeat):
        print("Hello from", app_config.app_name)


if __name__ == '__main__':
    app_config = gather_config(AppConfig, arg_aliases=dict(n='app_name'))
    main(app_config)

If the above code is in a file named getting_started.py, then you may already run it:

>python getting_started.py 
INFO:msrc.appconfig:logging level set to INFO.
Hello from Sample

Now try the following commands:

>python sample.py -h
usage: getting_started.py [-h [OPTION]] [-l LEVEL|FILE]
                          [-c CONF_FILE [CONF_FILE ...]] [-e PREFIX]

optional arguments:
  -h [OPTION], --help [OPTION]
                        Prints this help message and optionally description of an     
                        option.
  -l LEVEL|FILE         Either logging level or a path to a logging configuration     
                        file.
  -c CONF_FILE [CONF_FILE ...]
                        Additional configuration files. Allowed formats are JSON or   
                        YAML.
  -e PREFIX             Prefix for shell variables to look at. If environment
                        contains <PREFIX>_<ELEMENT_NAME>=VALUE the VALUE
                        overrides corresponding configuration element. The
                        default prefix is GETTING_STARTED_. A prefix of sole
                        dash disables the environment lookup.
Additionally, you may specify the following options. Use '--help OPTION_NO_DASHES' to 
get help on an option marked (*).
-n STR, --app_name STR, --app-name STR
--repeat INT

>GETTING_STARTED_REPEAT=3 python getting_started.py -l debug -n "another example"
INFO:msrc.appconfig:logging level set to DEBUG.
DEBUG:msrc.appconfig:Examining shell variables starting with GETTING_STARTED_.
DEBUG:msrc.appconfig:Shell variable GETTING_STARTED_repeat=3.
DEBUG:msrc.appconfig:discovered app_name = 'another example' from argv
INFO:msrc.appconfig:final repeat = 3 from env > GETTING_STARTED_repeat
INFO:msrc.appconfig:final app_name = 'another example' from argv
Hello from another example
Hello from another example
Hello from another example

The last command shows you may use environment variables to set configuration values. On Windows the example should be run with two commands. First, set the environment variable with SET GETTING_STARTED_REPEAT=3, and then run python script as shown.

You may also place configuration values in files and specify these file names in code or on command line. For example, create a file named sample.yaml:

app_name: getting started
repeat: 2

and continue the command line session (on Windows, first unset the environment variable with SET GETTING_STARTED_REPEAT=):

>python getting_started.py -l warn -c sample.yaml
Hello from getting started
Hello from getting started

Note also that with -l argument you control the amount of details being logged. The optional function argument gather_config(log_level=logging.WARN,...) has similar effect.

Configuration Schema

A schema is a class definition that serves as a template for application configuration. The package uses introspection to build a list of configuration schema elements and then reads configuration data from different sources.

Out of the box the package accepts configuration schema created with typing.NamedTuple. The example above shows that for each of its elements configuration schema contains a name, a type and a default value. The default value is optional, although you should be cautious mixixng elements with and without default values. For named tuples all elements without default values must come before elements with default values. If an element doesn't have a default, its value must be present in one of configuration sources. Otherwise gather_config() stops running the script.

The package accepts elements of the following types only:

• Simple types: str, int, float, bool, any enumeration.
• Uniform tuples of simple types. These are annotated with typing.Tuple generic type. E.g., Tuple[int, int] is a pair of integers, Tuple[str, ...] is a tuple of strings of any length including empty tuple.
• Other application configuration schema class. This allows composition of schemas.

All other types will raise an error in call to gather_config() and other package functions requiring introspection.

Optional metadata

Available plugins

Appconfig plugins allow alternative mechanisms to declare application configuration schema in addition to named tuples.

Dataclasses

msrc.appconfig.dataclasses is available for python version 3.7 and above. It allows for more flexible configuration declaration using @dataclass decorator. Compared to named tuples this approach allows building inheritance trees of application configuration. It's a good practice to declare application configuration as a "frozen" object. This prevents accidentally changing configuration values along the application run. Here is a sample declaration of an appconfig schema using dataclasses:

from dataclasses import dataclass, field

@dataclass(frozen=True)
class AppConfig:
    no_default: str
    with_default: repeat = 3
    with_help: bool = field(default=False, metadata=dict(help="description of the flag"))
    secret_password: str = attr.id(repr=False)

To use dataclasses schema specify the extra when installing msrc.appconfig:

>pip install msrc-appconfig[dataclasses]

Alternatively, install msrc-appconfig-dataclasses as a separate package.

Attrs

msrc.appconfig.attrs takes additional external dependency on attrs.

This package is a predecessor of dataclasses. It works on all versions of Python and is even more flexible. Here is a way to declare an appconfig schema using attrs:

import attrs

@attr.s(frozen=True, kw_only=True, auto_attribs=True)
class AppConfig:
    no_default: str
    with_default: repeat = 3
    with_help: bool = attr.ib(default=False, metadata=dict(help="description of the flag"))
    secret_password: str = attr.id(repr=False)

To use attrs schema specify the extra when installing msrc.appconfig:

>pip install msrc-appconfig[attrs]

Alternatively, install msrc-appconfig-attrs as a separate package.

Param

msrc.appconfig.param takes additional external dependency on param package. Application configuration classes must inherit from param.Parameterized. For fixed size tuples the param package have support for float type only (ParamNumeric()). Tuples of unbound size are encoded as ParamList(class_=<type>).

To use param schema specify the extra when installing msrc.appconfig:

>pip install msrc-appconfig[param]

Alternatively, install msrc-appconfig-param as a separate package.

Configuration sources

The default configuration source is the schema itself where you define default values. The values are overriden with the following order:

• override_defaults dictionary in gather_config() arguments;
• configuration files;
• shell variables a.k.a. environment variables;
• command line arguments.

Files

The module reads configparser (.ini), JSON (.json) and YAML (.yaml, .yml) files.

The list of files to be read can be specified using config_files optional argument: gather_config(config_files=[...], ...), and with -c command line option which take one or more file paths. Note though that relative paths are resolved against main script directory for config_files= and against current working directory for -c.

You may enable a default configuration file with a utility function: gather_config(config_files=script_config_file(), ...) If you now run python sample.py the default configuration file can be sample.yaml (or .json, or .ini, or .yml) in the same directory as the sample.py script. If you run python -m msrc.example the file can be example.json (or .yaml, or .ini) in current directory.

Another utility function makes it easy to use hierarchical configurations. If you start a script /foo/bar/script.py, and the script calls gather_config(config_files=config_files_in_parents('config.json'), ...), then the function will look for /config.json,/foo/config.json,/foo/bar/config.json and will read these files in this order if they exist.

In any configuration file you may also reference other configuration files using _include element. The value of this element is a path or a list of paths to read. Relative paths are resolved againsth the location of the including file. The included files are read before processing the rest of the parent file, i.e. other elements in the file may override values from those mentioned in _include.

Shell variables

Shell variables with names like <prefix><element> = <value> override a value of a configuration element <element>. If you run python sample.py then the default prefix is SAMPLE_. If you run python -m msrc.example then the prefix is EXAMPLE_. You can also specify another prefix with env_var_prefix= function argument or -e command line option. A single hyphen sign has a special meaning, it disables the use of environment variables.

For tuples shell variable value should contain all tuple values separated by space. E.g. for a script script.py to specify a pair of numbers as a value of a configuration element named limits the environment variable may look like script_limits=-1.5 2.56. If the tuple type is str, and a string value must have a space in it put the value in double quotes, quoting double quotes themselves with '\"' and the backslash with '\\', e.g. script_paths=C:\Windows "\"C:\\Program Files\""

Command line arguments

For floating point values any valid positive number works ok as well as special values nan and inf. Negative numbers start with minus sign, argparse considers them an option rather than a value. A workaround is to place space in front of the value, e.g. python script.py --interval " -1.5" 2.5.

For enums you may supply either a enum name or it value, preference given to names.

For tuples the option expects multiple arguments.

Boolean elements whith False as default value may be used as flags, i.e. if the option is present with no arguments, the value of the element is set to True. In any case you may also supply an argument. Any string that starts with 't' or 'y' is interpreted as True, for example 'true' or 'Yes'. Any string that starts with 'f' or 'n' is interpreted as False, for example 'F' or 'no'.