Skip to content

User's Guide

Introduction

The primary reason for creating the certlib.log library was to make it easier to configure structured logging across various systems created and used by CERT Polska – in a possibly consistent way and with minimal impact on existing code.

However, despite a few opinionated defaults, the library is quite versatile, so it may prove useful for a much broader audience of developers and system administrators. Apart from the structured logging stuff, it also offers a few other features…

Note

certlib.log uses only the Python standard library, i.e., it does not depend on any third-party packages.

How to Install

You can install the certlib.log library by running the following command (typically, you will do this within a Python virtual environment):

python3 -m pip install certlib.log

The library is compatible with Python 3.10 and all newer versions of Python.

Note

The canonical name of the distribution package is certlib-log (with a hyphen), but pip and other tools accept also the certlib.log form (with a dot); the latter may feel more natural, as it is also the importable module’s name (used in Python code).

TL;DR: How to Quickly Enable Structured Logging

  • Does your program already make use of the standard logging facilities and do you want it to start emitting structured JSON-serialized log entries? Just make your configuration of logging include certlib.log.StructuredLogsFormatter as a formatter. To do that easily, you may want to look at one of these examples (please also read the comments included there):

  • The system, component and component_type keys (which you can see in those examples) are intended to be set with the following semantics in mind:

    • system – the name of the entire system or project your script/application is part of;

    • component – the name of a particular script or application being executed (for a CLI script it should be its basename);

    • component_type – a conventional label of the type of that script or application, agreed upon in your organization (such as: "web", "parser", "collector"…).

  • That’s it! Everything else is optional (but probably worth a try, so you might want to read on).

Library Overview

The tools provided by certlib.log are intended for use with the standard logging module’s toolset. Essentially, they enhance that toolset with the following possibilities:

  • to emit structured log entries – each being a dict, hereinafter referred to as output data (serialized in JSON format before actually being emitted);

  • to permanently assign to selected output data keys: not only constant defaults, but also dynamic factories of values, hereinafter referred to as auto-makers; each auto-maker is just an argumentless function (or method, or other callable), automatically called to produce a value for the respective key – whenever a new log record object is created by a logger (which only occurs if the logger is enabled for the specified log level), before the log record is processed by any handlers, filters and formatters;

  • to replace the legacy %-based style of log message formatting with the modern and more convenient {}-based one, or (when what you need to log is just data) to omit passing the text message altogether; both gained by giving a little tweak to each logger method call…

While it is possible to use each of these capabilities independently of the others, the certlib.log’s stuff encourages combining them.

The following sections will discuss the two main tools provided by the library: StructuredLogsFormatter and xm.

Tool: StructuredLogsFormatter

To make the standard logging module’s machinery able to emit structured log entries (each being a JSON-serialized dict), you need to configure it to employ an instance of certlib.log.StructuredLogsFormatter as a formatter.

Note

Directly below it is shown how to do that in an imperative manner. You may prefer, however, a more declarative approach (especially if your program is not just a small script). In that case, please check out at least one of the following subsections (but first read everything above them as well!):

Basic Configuration

Let us start by creating our StructuredLogsFormatter instance (obviously, the specific values used in the following code snippet are just sample ones – they are supposed to be replaced with values appropriate for your program/system):

import itertools
import json
import logging
import sys
from certlib.log import StructuredLogsFormatter

structured_logs_formatter = StructuredLogsFormatter(
    defaults={
        # Each key in this dict should be an *output data* key.
        # Each value specifies the *default value* for that key.
        # For example:
        "system": "MyOwn",
        "component": "Portal",
        "component_type": "web",
        "example_custom_default": 42,
    },
    auto_makers={
        # Each key in this dict should be an *output data* key.
        # Each value should be either some argumentless callable
        # (function) or a *dotted path* to such a callable. In
        # particular, that callable *may* be the `get()` method
        # of some `ContextVar` instance (see
        # https://docs.python.org/3/library/contextvars.html).
        # For example:

        # (here: dotted paths pointing to callables)
        "client_ip": "myown.portal.client_ip_context_var.get",
        "example_nano_time": "time.time_ns",

        # (here: a callable passed directly)
        "example_local_counter": itertools.count(1).__next__,
    },
    # The value of `serializer` should be either a callable (function)
    # that accepts exactly one argument (being a JSON-serializable dict)
    # and returns a str object, or a *dotted path* to such a callable.
    # Note: the following serializer is the default -- so, in fact, you
    # do not need to specify it. But the possibility to define a custom
    # serializer comes in handy when you want to use, e.g., a faster
    # alternative to the standard `json.dumps()` function (or even a
    # tool which serializes data in some other format...).
    serializer=json.dumps,
)

See also

You may also want to look at the reference documentation for the StructuredLogsFormatter class.

Technically, each of the three keyword arguments accepted by the StructuredLogsFormatter constructor is optional. However, when it comes to the defaults and auto_makers ones, you need to consider that:

  • It is required that each of the following keys appears in at least one of those two mappings (in defaults and/or auto_makers):

    • "system" (the name of the entire system or project your script/application is part of; e.g.: "My Lovely System", "MWDB", "n6", etc.),

    • "component" (the name of a particular script or application being executed; for a CLI script it should be its basename),

    • "component_type" (a conventional label of the type of that script or application, agreed upon in your organization; e.g.: "web", "parser", "collector"…).

      • [maybe TBD: suggest a list of valid values of component_type?]

    Note

    If it is OK for you/your organization that some (or all) of the output data items listed above will remain unspecified, you can provide such a defaults mapping in which some (or all) of the aforementioned keys will be mapped to None values. Then, the said requirement will still be met, even though such void items will be automatically omitted from the ultimate defaults collection.

    See also: StructuredLogsFormatter.get_output_keys_required_in_defaults_or_auto_makers (the method which defines the requirement in question).

  • It is recommended (though not enforced) that each of the following keys, if it is relevant to the particular component_type, appears in at least one of those two mappings (in defaults and/or auto_makers, typically in the latter):

    • "client_ip" (the real IP of the client who communicates with us; for this information to be reliable, the way it is obtained needs to follow best practices specific to the protocol being used; for example, when it comes to HTTP, see: https://httptoolkit.com/blog/what-is-x-forwarded-for);

    • "user_id", "request_id", etc.

    • [TBD: more of the recommended output data keys to be suggested here].

Tip

If the presence of some output data key makes sense only in a certain context (e.g., when handling a HTTP request…), just make the respective auto-maker return None in any other contexts. Such void items will be automatically omitted from output data.

The next step is to prepare the root logger, and then add a handler to it with our formatter attached:

# (continuing with the previous example)

root_logger = logging.getLogger()
root_logger.setLevel(logging.INFO)

stderr_handler = logging.StreamHandler(sys.stderr)
stderr_handler.name = "stderr"
stderr_handler.setFormatter(structured_logs_formatter)  # <- Our formatter

root_logger.addHandler(stderr_handler)

See also

You may also want to take a look at the relevant parts of the documentation for the standard logging module.

Basic Usage

OK. Once the stuff is configured, let us emit some structured log entries!

We can do that in the legacy (standard, yet old-fashioned) manner…

import datetime as dt
import logging
import sys

logger = logging.getLogger(__name__)

# [...]

logger.info("Hello world!")

logger.warning("Hello %s!", sys.platform)

logger.error(
    "Here we have %x and %r.", sys.maxsize, sys.byteorder,
    extra={
        "example_stuff": [1, "foo", False],
        "other_example_item": {42: dt.datetime.now()},
    },
    exc_info=True,
)

…or (better!) by making use of the certlib.log.xm tool:

import datetime as dt
import ipaddress
import logging
import sys
from certlib.log import xm

logger = logging.getLogger(__name__)

# [...]

logger.info(xm("Hello world!"))

logger.warning(xm("Hello {}!", sys.platform))

logger.error(xm(
    "Here we have {:x} and {!r}.", sys.maxsize, sys.byteorder,
    example_stuff=[1, "foo", False],
    other_example_item={42: dt.datetime.now()},
    exc_info=True,
))

pure_data_dict = {
    'this': 123,
    'that': ipaddress.IPv4Address('192.168.0.42'),
    'there': 'example.com',
    'then': dt.datetime(2026, 1, 2, 3, 4, 56, tzinfo=dt.timezone.utc),
}
logger.info(xm(pure_data_dict))  # <- No text message at all.

logger.warning(xm(
    "{who} owns {fract:.2%} of all issues of {title!r} magazine.",
    who="John",
    fract=0.87239,
    title="Bajtek",
    first_issue_date=dt.date(1985, 9, 1),
))

See also

To learn more about using the xm tool, see this guide’s section Tool: xm (below).

Regarding the last logger.warning(...) call in the example above, the resultant JSON-serialized output data dict (i.e., the ultimate content of the log entry to be emitted) could look like the following (note that the serialized data presented here contains arbitrary example values for many keys, and – just for visual clarity – we present it here as being sorted by key, and with extra newlines/indentation):

{
    "client_ip": "192.168.0.123",
    "component": "Portal",
    "component_type": "web",
    "example_custom_default": 42,
    "example_local_counter": 6,
    "example_nano_time": 1771629287019638820,
    "first_issue_date": "1985-09-01",
    "fract": 0.87239,
    "func": "<module>",
    "level": "WARNING",
    "levelno": 30,
    "lineno": 253,
    "logger": "myown.portal.example_module",
    "message": "John owns 87.24% of all issues of 'Bajtek' magazine.",
    "message_base": {
        "pattern": "{who} owns {fract:.2%} of all issues of {title!r} magazine."
    },
    "pid": 324485,
    "process_name": "MainProcess",
    "py_ver": "3.14.3.final.0",
    "script_args": [
        "/opt/MyOwn/conf/web/portal.wsgi"
    ],
    "src": "/opt/MyOwn/py/myown/portal/example_module.py",
    "system": "MyOwn",
    "thread_id": 139781835344768,
    "thread_name": "MainThread",
    "timestamp": "2026-02-20 23:14:47.019574Z",
    "title": "Bajtek",
    "who": "John"
}

Note

As you can see, the dt.date instance provided as first_issue_date, before becoming an output data value, was converted to a string – thanks to an automatic invocation of the prepare_value method (before the actual data serialization).

All output data values are subject to preparation by that method (which processes them differenttly depending on their types…). By extending/overriding it in your StructuredLogsFormatter subclass you can gain full control over that preparation.

Nevertheless, you can get quite well just by sticking with the default implementation of that method.

logging.config.dictConfig-Style Configuration Example

import logging.config

logging_configuration_dict = {
    "formatters": {
        "structured": {
            "()": "certlib.log.StructuredLogsFormatter",
            "defaults": {
                # Each key in this dict should be an *output data* key.
                # Each value specifies the *default value* for that key.
                # For example:
                "system": "MyOwn",
                "component": "Portal",
                "component_type": "web",
                "example_custom_default": 42
                # ^ Important: by default, each of the "system", "component"
                #   and "component_type" keys is *required* to be included
                #   either *here* or in "auto_makers" (below). Note: *here*
                #   each of them can be assigned None -- if excluding this
                #   key from *output data* is OK for you/your organization.
            },
            "auto_makers": {
                # Each key in this dict should be an *output data* key.
                # Each value should be either some argumentless callable
                # (function) or a *dotted path* to such a callable. In
                # particular, that callable *may* be the `get()` method
                # of some `ContextVar` instance (see
                # https://docs.python.org/3/library/contextvars.html).
                # For example:
                "client_ip": "myown.portal.client_ip_context_var.get",
                "example_nano_time": "time.time_ns"
            },
            # The value of "serializer", if specified, should be either
            # a callable (function) which accepts exactly one argument
            # (being a JSON-serializable dict) and returns a str object,
            # or a *dotted path* to such a callable. If "serializer" is
            # not specified, the standard `json.dumps()` function will
            # be used.
            "serializer": "some_package.faster_replacement_for_json_dumps"
        }
    },
    "handlers": {
        "stderr": {
            "class": "logging.StreamHandler",
            "formatter": "structured",
            "stream": "ext://sys.stderr"
        }
    },
    "root": {
        "level": "INFO",
        "handlers": ["stderr"]
    },
    "disable_existing_loggers": False,
    "version": 1
}

logging.config.dictConfig(logging_configuration_dict)

Tip

Typically, applications load such a configuration dict from some file (usually in TOML, YAML or JSON format).

See also

You can learn more about the logging.config.dictConfig-specific configuration dict schema by referring to the relevant section of the documentation for the standard logging.config module.

logging.config.fileConfig-Style Configuration Example

[loggers]
keys = root

[handlers]
keys = stderr

[formatters]
keys = structured

[logger_root]
level = INFO
handlers = stderr

[handler_stderr]
class = StreamHandler
formatter = structured
args = (sys.stderr,)

[formatter_structured]
class = certlib.log.StructuredLogsFormatter
format = {
    "defaults": {
        # Each key in this dict should be an *output data* key.
        # Each value specifies the *default value* for that key.
        # For example:
        "system": "MyOwn",
        "component": "Portal",
        "component_type": "web",
        "example_custom_default": 42,
        # ^ Important: by default, each of the "system", "component"
        #   and "component_type" keys is *required* to be included
        #   either *here* or in "auto_makers" (below). Note: *here*
        #   each of them can be assigned None -- if excluding this
        #   key from *output data* is OK for you/your organization.
    },
    "auto_makers": {
        # Each key in this dict should be an *output data* key.
        # Each value should be a *dotted path* to some argumentless
        # callable (function). In particular, that callable *may* be
        # the `get()` method of some `ContextVar` instance (see
        # https://docs.python.org/3/library/contextvars.html).
        # For example:
        "client_ip": "myown.portal.client_ip_context_var.get",
        "example_nano_time": "time.time_ns",
    },
    # The value of "serializer", if specified, should be a *dotted path*
    # to a callable (function) which accepts exactly one argument (being
    # a JSON-serializable dict) and returns a str object. If "serializer"
    # is not specified, the standard `json.dumps()` function will be used.
    "serializer": "some_package.faster_replacement_for_json_dumps",
  }
# ^ *Note:* all non-comment and non-blank continuation lines, *including*
#   the one with the closing `}`, *must be indented* (by at least 1 space).

Tip

If logging.config.fileConfig is called by your code (rather than automatically by some framework/library…), you may want to set the disable_existing_loggers argument to False (because if its default value, True, is in effect, then some loggers created before that call may be turned off, which is usually not what you want). This is a general advice (not specific to certlib.log).

See also

You can learn more about the logging.config.fileConfig-specific configuration format by referring to the relevant section of the documentation for the standard logging.config module.

Tool: xm

Essentially, the purpose of xm is two-fold:

  • to make it more convenient to emit structured log entries (each being representable as a dict), especially if a StructuredLogsFormatter is in use;

  • if you choose the traditional text-message-focused style of logging (rather than a pure-data-focused, message-less one) – to easily replace the legacy %-based log message formatting style with the modern and more convenient {}-based one (regardless of what formatter is in use).

Note

xm is just a convenience alias of ExtendedMessage (the latter is the actual name of the class, but the former is definitely more handy when you want to log a message or data).

Let examples speak…

Dealing with Pure Data

Below – a couple of examples of logging just some data (without the need to specify any text message).

import logging
from certlib.log import xm

logger = logging.getLogger(__name__)

# Logging pure data:
logger.info(xm(
    some_key=["example", "data"],
    another=lambda: 42,  # (<- function/method: to be called by formatter)
    yet_another={"abc": 1.0, "qwerty": [True, False]},
))

# Same as above, but here we pass our data just *as one dict*:
my_data = {
    "some_key": ["example", "data"],
    "another": lambda: 42,  # (<- function/method: to be called by formatter)
    "yet_another": {"abc": 1.0, "qwerty": [True, False]},
}
logger.info(xm(my_data))

Tip

Regarding the "another" item in the above examples as well as some of the items/arguments that appear in the next subsection’s examples: if you pass a function/method (also a lambda expression) instead of a plain value, it will be automatically called to obtain the actual value (at most once per xm use, by a formatter of any type, that is, not necessarily by a StructuredLogsFormatter).

In practice, this feature is useful if the creation of a certain value is costly – then you may prefer it to be created when (and if) the log entry is actually about to be formatted and emitted.

Note

By default, the mechanism is applied only if you pass a function or method object – not just an arbitrary callable (as that could lead to inadvertent calls).

Modern Formatting Style

Below there are a few examples of traditional text-message-focused logging, but – what using the xm tool makes possible – with the modern and convenient {}-based style of message formatting (rather than the legacy, less convenient and less powerful, %-based one).

Note

What we are discussing here concerns the formatting of text messages themselves (i.e., the contents of log records’ message), rather than the formatting of entire log entries (where message is just a field). Note that the latter is completely orthogonal to the former. Whereas the standard tools provided by the logging module do support the {}-based formatting style for the latter, they do not support it for the former.

import datetime as dt
import logging
from certlib.log import xm

logger = logging.getLogger(__name__)

some_name = "foo"
some_value = "Bar"

logger.info(xm(
    "Note: {} is {!r} (in {:%Y-%m})",
    some_name, some_value,
    dt.date.today,  # (<- function/method: to be called by formatter)
))

The resultant message will be: "Note: foo is 'Bar' (in 2026-02)" (assuming that, for this particular example, the dt.date.today class method would return an instance of dt.date representing a February 2026 date, e.g., one equal to dt.date(2026, 2, 21)).

What that means if the logging system is configured to employ a StructuredLogsFormatter, is that:

  • the formatted message will appear in the JSON-serialized output data as the item: "message": "Note: foo is 'Bar' (in 2026-02)",
  • and the raw message pattern will also be included, like this: "message_base": {"pattern": "Note: {} is {!r} (in {:%Y-%m})"}.

Info

When you use xm, you still benefit from the standard mechanism of deferring message formatting until the log entry really needs to be emitted (regardless of what formatter is in use).

The code in the next example does the same as above; the only difference is that here the replacement fields in the message pattern are explicitly numbered:

logger.info(xm(
    "Note: {0} is {1!r} (in {2:%Y-%m})",
    some_name, some_value,
    dt.date.today,  # (<- function/method: to be called by formatter)
))

Below there is an example similar to the previous two, but with some of the replacement fields being named (and, therefore, with the corresponding keyword arguments specifying the values to be interpolated):

logger.info(xm(
    "Note: {} is {val!r} (in {today:%Y-%m})",
    some_name,
    val=some_value,
    today=dt.date.today,  # (<- function/method: to be called by formatter)
))

It is worth noting that if a StructuredLogsFormatter is in use, then any keyword arguments (named ones) passed to xm, apart from being used to fill in the respective replacement fields, are also included as output data items. For example, output data resulting from the logger.info(...) call in the last example will contain, among others, the following items:

  • "message": "Note: foo is 'Bar' (in 2026-02)",
  • "message_base": {"pattern": "Note: {} is {val!r} (in {today:%Y-%m})"},
  • "val": "Bar",
  • "today": "2026-02-21".

And below there is almost the same call as previously, but with a couple of extra keyword arguments (conveying some additional data, unrelated to message formatting):

logger.info(xm(
    "Note: {} is {val!r} (in {today:%Y-%m})",
    some_name,
    val=some_value,
    today=dt.date.today,          # (<- function/method: to be called...)
    something=lambda: 123456789,  # (<- function/method: to be called...)
    something_more=(1, 2, 3, 4, True, None, {5: [6789, 10]}),
))

In this case, the resultant output data generated by the StructuredLogsFormatter’s machinery will contain, among others, the following items:

  • "message": "Note: foo is 'Bar' (in 2026-02)",
  • "message_base": {"pattern": "Note: {} is {val!r} (in {today:%Y-%m})"},
  • "val": "Bar",
  • "today": "2026-02-21",
  • "something": 123456789,
  • "something_more": [1, 2, 3, 4, true, null, {"5": [6789, 10]}].

The entire resultant JSON-serialized output data (i.e., the ultimate content of the log entry to be emitted) could look like the following (note that the serialized data presented here contains arbitrary example values for many keys, and – just for visual clarity – we present it here as being sorted by key, and with extra newlines/indentation):

{
    "client_ip": "192.168.0.123",
    "component": "Portal",
    "component_type": "web",
    "example_custom_default": 42,
    "example_local_counter": 4,
    "example_nano_time": 1771631594315719605,
    "func": "<module>",
    "level": "INFO",
    "levelno": 20,
    "lineno": 179,
    "logger": "myown.portal.another_example_module",
    "message": "Note: foo is 'Bar' (in 2026-02)",
    "message_base": {
        "pattern": "Note: {} is {val!r} (in {today:%Y-%m})"
    },
    "pid": 327578,
    "process_name": "MainProcess",
    "py_ver": "3.14.3.final.0",
    "script_args": [
        "/opt/MyOwn/conf/web/portal.wsgi"
    ],
    "something": 123456789,
    "something_more": [
        1, 2, 3, 4, true, null, {
            "5": [
                6789, 10
            ]
        }
    ],
    "src": "/opt/MyOwn/py/myown/portal/another_example_module.py",
    "system": "MyOwn",
    "thread_id": 140062429502336,
    "thread_name": "MainThread",
    "timestamp": "2026-02-20 23:53:14.315296Z",
    "today": "2026-02-21",
    "val": "Bar"
}

See also

You may also want to look at the reference documentation for the ExtendedMessage class (among other things, you will find there information about three special arguments you can also pass to xm – namely: exc_info, stack_info and stacklevel).

Advanced Topics and Finer Points

More About StructuredLogsFormatter (Including Subclassing)

If you have not read the reference documentation for the StructuredLogsFormatter class yet, you are strongly encouraged to do so. Among other things, you will find there a list of hook methods that can be extended/overridden in your subclasses. Apart from that, the documentation in question includes (also in the individual descriptions of those hook methods) valuable information about other elements of the StructuredLogsFormatter’s interface and behavior.

Other Stuff Provided by certlib.log

Besides StructuredLogsFormatter and xm (ExtendedMessage), the certlib.log module provides the following public stuff:

Roadmap Outline

Future ideas under consideration include:

  • StructuredLogsFormatter: add the ability to specify keys related to sensitive data – so that values assigned to them in output data will be automatically masked/anonymized.

  • xm: add dedicated suport for pattern of type string.templatelib.Template – instances of which can be created by evaluating t-strings (available in Python 3.14 and newer). Additionaly, to support passing such t-string-made template objects directly to logger methods, add an opt-in mechanism that will automatically wrap such templates in xm objects (so that, e.g., you could just do: logger.info(t"2n is {2 * n}, not {sys.maxsize:x}").