API Reference

Warning

The certlib.log library is currently in the beta stage of development. This means, in particular, that backward incompatible changes to the public API are possible (and they sometimes do happen) in non-major pre-release versions – until the final 1.0.0 version is released.

General Remarks

The certlib.log library is compatible with Python 3.10 and all newer versions of Python.

Important definition: whenever this document refers to undefined behavior, this should be understood to mean: the API makes no guarantees about what will happen – an exception or a malfunction is likely.

Unless otherwise specified, using the library in a way that contravenes the documented API will result in undefined behavior.

Interface exclusion

The following elements/features are not part of the API (so, in particular, they may change – or be removed if applicable – in minor or patch versions of the library):

• any elements not documented in this API reference as well as elements that appear only in source code excerpts (available via <> Source code... drop-down widgets), e.g., specific exception messages;

• specific runtime types of any objects bound to an element of the API (a variable, attribute, parameter or call result), provided that they remain correct with respect to the element’s type annotation, according to the static typing rules;

• specific behaviors in cases where – according to the documentation – undefined behavior is expected (see the definition above);

• unofficial support for Python 3.9.

---

StructuredLogsFormatter

StructuredLogsFormatter(
    *,
    defaults: Mapping[str, object] | None = None,
    auto_makers: Mapping[str, ValueProvider[object] | DottedPath] | None = None,
    serializer: OutputSerializer | DottedPath = json.dumps
)

StructuredLogsFormatter(
    mapping_of_kwargs_compatible_with_main_signature: (
        Mapping[Literal["defaults", "auto_makers", "serializer"], Any]
        | KwargsMappingAsLiteralEvaluableString
    ),
    /,
    datefmt: None = None,
    style: Literal["%"] = "%",
    validate: Literal[True] = True,
)

StructuredLogsFormatter(
    fmt: None = None,
    datefmt: None = None,
    style: Literal["%"] = "%",
    validate: Literal[True] = True,
    *,
    defaults: Mapping[str, object] | None = None,
    auto_makers: Mapping[str, ValueProvider[object] | DottedPath] | None = None,
    serializer: OutputSerializer | DottedPath = json.dumps
)

A subclass of logging.Formatter to form structured log entries.

Tip

If the three call signatures defined by the StructuredLogsFormatter constructor seem overwhelming, do not worry. In most cases, you will only really be interested in the first one (the main signature). The details are provided below.

See also

For extra information about StructuredLogsFormatter, including a bunch of usage examples and configuration tips, see the Tool: StructuredLogsFormatter section of the User’s Guide.

Constructor arguments (all keyword-only, all optional):

• defaults (a dict or other mapping; default: {}): maps output data keys to values each of which specifies the default value for the respective key (see also the make_base_defaults method…).

• auto_makers (a dict or other mapping; default: {}): maps output data keys to respective auto-makers (argumentless factories of output data values – to be automatically called whenever a log entry is prepared). Each auto-maker can be specified either directly or as a string being a dotted path (importable dotted name) that points to an auto-maker (see also the make_base_auto_makers method…).

• serializer (a function or other callable; default: json.dumps): a callable that takes one argument being an output data dict and returns a string (presumably, a JSON-serialized form of that dict, even though you may decide to use some other serialization format, if this is OK for you/your organization). Alternatively, serializer can be a dotted path string (importable dotted name) that points to such a callable.

Note

The type of every output data dict (taken by serializer) is annotated as dict[str, OutputValue] – where, essentially, the OutputValue element denotes all types of values that might be returned by the actual implementation of the prepare_value method. In other words, that method is what determines those types.

Note that the default implementation of that method always returns values of json.dumps-compatible types.

Interface restriction

While serializer is allowed to add, remove or replace top-level items in an output data dict it takes, it should never mutate any object inside that dict (regardless of the level of nesting). If some data needs to be modified, completely new data object(s) should be created – to entirely replace the respective top-level value in the dict (without mutating the original object(s) being replaced). Doing otherwise will result in undefined behavior.

Alternatively, a mapping (especially a dict) of keyword arguments compatible with the main signature described above, or an ast.literal_eval-evaluable string representing such a dict, can be passed to the StructuredLogsFormatter constructor as the first positional argument.

Moreover, extra arguments that match – by position or by name – any non-keyword-only parameters defined by the logging.Formatter base class are accepted but ignored by the StructuredLogsFormatter constructor, provided that the value of each (if given) is the default value of the respective parameter; that is:

• the first or fmt argument – needs to be None (except that it is fine for the first argument to be a mapping or a string representing a mapping, as described above…);

• the second or datefmt argument – needs to be None;

• the third or style argument – needs to be the "%" string (remember, it will be ignored anyway);

• the fourth or validate argument – needs to be True.

If any of them does not comply, TypeError is raised.

Info

Passing any unexpected (surplus) arguments also causes TypeError.

Note

Thanks to the interface extensions described above, you can configure a StructuredLogsFormatter even if you are using the logging.config.fileConfig-specific configuration format (which, despite its limitations, is still quite popular).

See the formatter_structured section of the fileConfig-style configuration example in the User’s Guide.

This class defines the following extendable/overridable hook methods:

• get_output_keys_required_in_defaults_or_auto_makers
• make_base_defaults
• make_base_auto_makers
• make_base_record_attr_to_output_key
• format_timestamp
• get_prepared_output_data
• prepare_value
• prepare_submapping_key
• serialize_prepared_output_data

In some of the individual descriptions of these methods, several other elements of the StructuredLogsFormatter’s interface and behavior are also discussed – in particular, the following instance attributes:

• defaults
• auto_makers
• auto_made_record_attr_prefix
• record_attr_to_output_key
• serializer

Interface restriction

Once an instance of StructuredLogsFormatter is initialized, the instance attributes listed above should be treated as read-only and immutable ones (together with all their contents, regardless of the level of nesting, if any nested data is present). Doing otherwise will result in undefined behavior.

When it comes to customizing the format of log entry timestamps, the related attributes defined by the logging.Formatter base class (namely: converter, default_time_format and default_msec_format) are ignored by the machinery of StructuredLogsFormatter.

To learn how to actually customize timestamp formatting, please refer to the description of the StructuredLogsFormatter’s format_timestamp method.

Additional requirement

Regarding the initialization of a StructuredLogsFormatter instance, it is also required that every output data key (just key, as we are not talking about output data values here) appearing in any of the mappings listed below – be a string and not exceed 200 characters (otherwise, respectively, TypeError or ValueError will be raised by the constructor). The mappings covered by this requirement are:

• that returned by the make_base_defaults method,

• the defaults argument to the constructor (if actually passed),

• that returned by the make_base_auto_makers method,

• the auto_makers argument to the constructor (if actually passed),

• that returned by the make_base_record_attr_to_output_key method (note that the requirement in question applies to output data keys – which, when it comes to this mapping, are its values, not its keys; and note that this mapping’s values are also allowed to be None).

Source code in src/certlib/log.py

def __init__(self, /, *args: Any, **kwargs: Any):
    arguments = self._resolve_init_arguments(args, *args, **kwargs)

    given_defaults = arguments.pop('defaults', None) or {}
    given_auto_makers = arguments.pop('auto_makers', None) or {}
    given_serializer = arguments.pop('serializer', json.dumps)

    if arguments:
        raise TypeError(
            f'{type(self).__init__.__qualname__}() got unexpected '
            f'keyword argument(s): {", ".join(map(ascii, arguments))}'
        )

    super().__init__()

    unfiltered_defaults = self._get_unfiltered_defaults(given_defaults)
    unprefixed_auto_makers = self._get_unprefixed_auto_makers(given_auto_makers)
    self._check_output_keys_required_in_defaults_or_auto_makers(
        unfiltered_defaults,
        unprefixed_auto_makers,
    )

    actual_defaults = self._get_actual_defaults(unfiltered_defaults)
    auto_made_record_attr_prefix = self._get_auto_made_record_attr_prefix()
    actual_auto_makers = self._get_actual_auto_makers(
        auto_made_record_attr_prefix,
        unprefixed_auto_makers,
    )
    self.defaults = actual_defaults
    self.auto_makers = actual_auto_makers
    self.auto_made_record_attr_prefix = auto_made_record_attr_prefix

    self.record_attr_to_output_key = self._get_record_attr_to_output_key()
    self.serializer = self._get_actual_serializer(given_serializer)

    for rec_attr, auto_maker in self.auto_makers.items():
        register_log_record_attr_auto_maker(rec_attr, auto_maker)

defaults instance-attribute

defaults: Final[Mapping[str, OutputValue]]

auto_makers instance-attribute

auto_makers: Final[Mapping[str, ValueProvider[object]]]

auto_made_record_attr_prefix instance-attribute

auto_made_record_attr_prefix: Final[str]

record_attr_to_output_key instance-attribute

record_attr_to_output_key: Final[Mapping[str, str | None]]

serializer instance-attribute

serializer: Final[OutputSerializer]

FORMAT_TIMESTAMP_DEFAULT_KWARGS class-attribute

FORMAT_TIMESTAMP_DEFAULT_KWARGS: Final[Mapping[str, Any]]

Default values of all StructuredLogsFormatter.format_timestamp’s keyword-only parameters (this mapping may come in handy when you extend that method in a subclass…).

PREPARE_VALUE_DEFAULT_KWARGS class-attribute

PREPARE_VALUE_DEFAULT_KWARGS: Final[Mapping[str, Any]]

Default values of all StructuredLogsFormatter.prepare_value’s keyword-only parameters (this mapping may come in handy when you extend that method in a subclass…).

unregister_auto_makers

unregister_auto_makers() -> None

A rarely useful method: you may want to invoke it on an instance of StructuredLogsFormatter only when you need to stop using that instance but continue using any logging stuff during further program execution (this does not seem to be a common case).

Source code in src/certlib/log.py

def unregister_auto_makers(self) -> None:
    """
    A rarely useful method: you may want to invoke it on an instance
    of `StructuredLogsFormatter` *only when* you need to stop using
    that instance but continue using any `logging` stuff during
    further program execution (this does not seem to be a common
    case).
    """
    for rec_attr in self.auto_makers.keys():
        unregister_log_record_attr_auto_maker(rec_attr)

format

format(record: logging.LogRecord) -> str

Overrides the logging.Formatter’s implementation with a StructuredLogsFormatter-specific one.

In some respects, the StructuredLogsFormatter’s implementation of this method is similar to the logging.Formatter’s original. In particular, it makes use of the usesTime, formatTime, formatMessage and formatException methods in a similar way, and assigns values to the same log record attributes: message, asctime and exc_text, making doing so subject to the same conditions (where applicable). However, it differs from the original in the following ways:

• of the methods mentioned above, the formatMessage one is always invoked last (in particular, after the log record’s exc_text attribute is possibly set to a value returned by formatException);

• if the log record’s exc_info attribute is a (None, None, None) tuple, then it is treated as if it were falsy (the formatException method if not invoked, and the log record’s exc_text attribute is not set);

• the string returned by formatMessage becomes the return value of this method (so this method never appends to that string any formatted traceback or formatted stack information, and it does not invoke formatStack either); that string is supposed to represent the output data dict after serialization (therefore, it should already include, among others, any exception/stack information, if such stuff was requested and obtained);

• regarding how the target value of the log record’s message attribute is determined: if the msg attribute of the given log record is an instance of ExtendedMessage (xm), then that instance’s get_message_value method is invoked (directly), instead of the log record’s method getMessage;

Source code in src/certlib/log.py

def format(self, record: logging.LogRecord) -> str:
    """
    Overrides the [`logging.Formatter`'s
    implementation][logging.Formatter.format] with
    a `StructuredLogsFormatter`-specific one.

    In *some* respects, the `StructuredLogsFormatter`'s implementation
    of this method is similar to the `logging.Formatter`'s original. In
    particular, it makes use of the [`usesTime`][], [`formatTime`][],
    [`formatMessage`][] and [`formatException`][logging.Formatter.formatException]
    methods in a similar way, and assigns values to the same log record
    attributes: [`message`, `asctime` and `exc_text`](https://docs.python.org/3/library/logging.html#logrecord-attributes),
    making doing so subject to the same conditions (where applicable).
    However, it differs from the original in the following ways:

    * of the methods mentioned above, the `formatMessage` one is
      always invoked last (in particular, *after* the log record's
      `exc_text` attribute is possibly set to a value returned by
      `formatException`);

    * if the log record's `exc_info` attribute is a `(None, None,
      None)` tuple, then it is treated as if it were *falsy* (the
      `formatException` method if not invoked, and the log record's
      `exc_text` attribute is *not* set);

    * the string returned by `formatMessage` becomes the return value
      of *this* method (so this method *never* appends to that string
      any *formatted traceback* or *formatted stack information*, and
      it does *not* invoke [`formatStack`][logging.Formatter.formatStack]
      either); that string is supposed to represent the *output data*
      dict after serialization (therefore, it should already include,
      among others, any exception/stack information, if such stuff was
      requested and obtained);

    * regarding how the target value of the log record's `message`
      attribute is determined: if the `msg` attribute of the given
      log record is an instance of [`ExtendedMessage`][] ([`xm`][]),
      then that instance's [`get_message_value`][ExtendedMessage.get_message_value]
      method is invoked (directly), *instead* of the log record's
      method [`getMessage`][logging.LogRecord.getMessage];
    """
    # (Compare to the source code of `logging.Formatter.format()`...)
    msg = getattr(record, 'msg', None)
    if isinstance(msg, ExtendedMessage):
        if args := getattr(record, 'args', None):
            args_repr = self._get_record_args_repr(args)
            raise TypeError(
                f"the specified log message base is an instance "
                f"of {ExtendedMessage.__qualname__} ({msg=!a}); "
                f"in such a case, any positional arguments to "
                f"format the log message should have been passed "
                f"to the `{ExtendedMessage.__qualname__}(...)` "
                f"(or `xm(...)`) call, not to the logger method "
                f"call itself (*args obtained by it: {args_repr})"
            )
        record.message = msg.get_message_value()
    else:
        record.message = record.getMessage()
    if self.usesTime():
        record.asctime = self.formatTime(record, self.datefmt)
    if (record.exc_info
          and record.exc_info != (None, None, None)
          and not record.exc_text):
        record.exc_text = self.formatException(record.exc_info)
    return self.formatMessage(record)

usesTime

usesTime() -> bool

Overrides the logging.Formatter’s implementation with one that always returns True.

Source code in src/certlib/log.py

def usesTime(self) -> bool:
    """
    Overrides the [`logging.Formatter`][]'s implementation with one
    that always returns [`True`][].
    """
    return True

formatTime

formatTime(record: logging.LogRecord, datefmt: str | None = None) -> str

Overrides the logging.Formatter’s implementation with one that delegates its entire job to the format_timestamp method (a StructuredLogsFormatter-specific one), but first checks if the datefmt argument is None (if it is anything else, TypeError is raised – which then, typically, is suppressed and, possibly, printed to sys.stderr by logging.Handler.handleError…).

Source code in src/certlib/log.py

def formatTime(
    self,
    record: logging.LogRecord,
    datefmt: str | None = None,
) -> str:
    """
    Overrides the [`logging.Formatter`'s
    implementation][logging.Formatter.formatTime] with one that
    delegates its entire job to the [`format_timestamp`][] method
    (a `StructuredLogsFormatter`-specific one), but first checks
    if the **`datefmt`** argument is [`None`][] (if it is anything
    else, [`TypeError`][] is raised -- which then, typically, is
    suppressed and, possibly, printed to [`sys.stderr`][] by
    [`logging.Handler.handleError`][]...).
    """
    if datefmt is not None:
        slf = StructuredLogsFormatter.__qualname__
        slf_format_timestamp = f'{StructuredLogsFormatter.format_timestamp.__qualname__}()'
        slf_formatTime = f'{StructuredLogsFormatter.formatTime.__qualname__}()'  # noqa
        raise TypeError(
            f"{datefmt=!a}, whereas for a `{__name__}.{slf}`-derived "
            f"formatter it should be None. To customize timestamp "
            f"formatting in your logs, instead of trying to set "
            f"`datefmt` or other `logging.Formatter`-specific stuff "
            f"(*not* used by the `{slf}`'s machinery!), you should "
            f"rather extend/override (in your custom subclass) the "
            f"`{slf_format_timestamp}` method. (Alternatively, instead "
            f"of that, you might decide to completely override the "
            f"`{slf_formatTime}` method, providing an implementation "
            f"which, for example, would use the `logging.Formatter`'s "
            f"legacy timestamp-formatting-related stuff, without any "
            f"use of the `{slf_format_timestamp}` method...)"
        )
    return self.format_timestamp(record)

formatMessage

formatMessage(record: logging.LogRecord) -> str

Overrides the logging.Formatter’s implementation with one that:

• obtains a ready output data dict by applying the get_prepared_output_data method to the given log record;

• applies the serialize_prepared_output_data method to the obtained output data dict, and returns the result.

Note

The formatMessage name may be slightly misleading. Let us emphasize that the job of this method is always – also in the case of the original logging.Formatter class – to format the crux of the entire log entry, not just the value of the log record’s message attribute. Formatting the latter is the job of the log record’s getMessage method, or – when the machinery of StructuredLogsFormatter deals with an ExtendedMessage (xm) instance – of the get_message_value method of that instance.

Source code in src/certlib/log.py

def formatMessage(self, record: logging.LogRecord) -> str:
    """
    Overrides the [`logging.Formatter`][]'s implementation with
    one that:

    * obtains a ready *output data* dict by applying the
      [`get_prepared_output_data`][] method to the given
      log record;

    * applies the [`serialize_prepared_output_data`][]
      method to the obtained *output data* dict, and
      returns the result.

    !!! note

        The **`formatMessage`** name may be slightly misleading. Let
        us emphasize that the job of this method is *always* -- also
        in the case of the original **`logging.Formatter`** class --
        to format the crux of the _**entire**_ log entry, _**not**_
        just the value of the log record's **`message`** attribute.
        Formatting the latter is the job of the log record's
        **[`getMessage`][logging.LogRecord.getMessage]** method,
        or -- when the machinery of **`StructuredLogsFormatter`**
        deals with an **[`ExtendedMessage`][]** (**[`xm`][]**)
        instance -- of the **[`get_message_value`][ExtendedMessage.get_message_value]**
        method of that instance.
    """
    output_data = self.get_prepared_output_data(record)
    return self.serialize_prepared_output_data(output_data)

get_output_keys_required_in_defaults_or_auto_makers

get_output_keys_required_in_defaults_or_auto_makers() -> Set[str]

A hook method: extend it in a subclass to impose (more) output data keys required to be specified for the purpose of setting defaults and auto_makers.

Note

Obviously, you can also extend/override this method to define fewer required keys than by default (perhaps even no one) if this is OK for you/your organization.

To be more precise: this method’s return value defines the set of keys required to be included in at least one of the following mappings:

• that returned by the make_base_defaults method,

• the defaults argument to the constructor (if actually passed),

• that returned by the make_base_auto_makers method,

• the auto_makers argument to the constructor (if actually passed).

Whether this requirement is satisfied is checked during the formatter initialization. If the check fails, KeyError is raised.

The default implementation of this method just uses the set of keys defined as COMMONLY_EXPECTED_NON_STANDARD_OUTPUT_KEYS (which means that, when invoking the StructuredLogsFormatter constructor, it is required to specify default values and/or auto-makers for the keys: "system", "component" and "component_type").

Info

The requirement in question is considered satisfied also if some (or all) of the required items are provided only as defaults (i.e., only by the make_base_defaults’s result or the defaults constructor argument) and some (or all) of them define such default values that – after being transformed by the prepare_value method – are void values, e.g., None (despite the fact that such void values are always excluded from the ultimate collection of default values – see the Related interfaces note in the make_base_defaults method’s description).

my_formatter = StructuredLogsFormatter(
    # This is OK (the requirement is satisfied):
    defaults={
        "system": None,
        "component": None,
        "component_type": None,
    },
)

In other words, the interface does not prevent you from effectively omitting from output data the keys specified by this method’s result, but you must explicitly state that you really want this (so that it can be safely assumed that this is OK for you/your organization).

Source code in src/certlib/log.py

def get_output_keys_required_in_defaults_or_auto_makers(self) -> Set[str]:
    """
    A hook method: extend it in a subclass to impose (more)
    *output data* keys *required to be specified* for the
    purpose of setting [`defaults`][] and [`auto_makers`][].

    !!! note

        Obviously, you can also extend/override this method to define
        *fewer* required keys than by default (perhaps even *no* one)
        if this is OK for you/your organization.

    To be more precise: this method's return value defines the set of
    keys *required to be included* in *at least one* of the following
    mappings:

    * that returned by the [`make_base_defaults`][] method,

    * the **`defaults`** argument to the
      [constructor][StructuredLogsFormatter] (if actually passed),

    * that returned by the [`make_base_auto_makers`][] method,

    * the **`auto_makers`** argument to the
      [constructor][StructuredLogsFormatter] (if actually passed).

    Whether this requirement is satisfied is checked during the
    formatter initialization. If the check fails, [`KeyError`][]
    is raised.

    The default implementation of this method just uses the set of
    keys defined as [`COMMONLY_EXPECTED_NON_STANDARD_OUTPUT_KEYS`][]
    (which means that, when invoking the [`StructuredLogsFormatter`][]
    constructor, it is *required* to specify *default values* and/or
    *auto-makers* for the keys: `"system"`, `"component"` and
    `"component_type"`).

    !!! info

        The requirement in question is considered satisfied _**also**_
        if some (or all) of the required items are provided *only as
        defaults* (i.e., only by the **[`make_base_defaults`][]**'s
        result or the **`defaults`** constructor argument) *and* some
        (or all) of them define such *default values* that -- after
        being transformed by the **[`prepare_value`][]** method --
        are *void* values, e.g., [`None`][] (despite the fact that
        such *void* values are always *excluded* from the ultimate
        collection of *default values* -- see the *Related interfaces*
        note in the **[`make_base_defaults`][]** method's description).

        ```python
        my_formatter = StructuredLogsFormatter(
            # This is OK (the requirement is satisfied):
            defaults={
                "system": None,
                "component": None,
                "component_type": None,
            },
        )
        ```

        In other words, the interface does not prevent you from
        effectively omitting from *output data* the keys specified
        by this method's result, but you must explicitly state that
        you really want this (so that it can be safely assumed that
        this is OK for you/your organization).
    """
    assert isinstance(COMMONLY_EXPECTED_NON_STANDARD_OUTPUT_KEYS, frozenset)
    return COMMONLY_EXPECTED_NON_STANDARD_OUTPUT_KEYS

make_base_defaults

make_base_defaults() -> Mapping[str, object]

A hook method: extend it in a subclass to define basic default values for output.

Automatically invoked on the formatter initialization. Each key in the resultant mapping needs to be an output data key, and each value in that mapping needs to be the desired default value for that key.

The default implementation of this method returns an empty mapping.

Related interfaces

For every instance, the defaults mapping (which is supposed to specify all default values for any output data to be generated by the instance) is based on this method’s result, but is then updated with all items from the defaults constructor argument (if given), and adjusted by applying the prepare_value method to each value, and – then – by deleting each key to which a void value is assigned (by void value we mean any falsy value that is not equal to 0, for example: None, "", [] or {} – but not: False, 0, 0.0, etc.).

Source code in src/certlib/log.py

def make_base_defaults(self) -> Mapping[str, object]:
    """
    A hook method: extend it in a subclass to define basic *default
    values* for output.

    Automatically invoked on the formatter initialization. Each key in
    the resultant mapping needs to be an *output data* key, and each
    value in that mapping needs to be the desired *default value* for
    that key.

    The default implementation of this method returns an empty mapping.

    !!! info "Related interfaces"

        For every instance, the **[`defaults`][]** mapping (which is
        supposed to specify all *default values* for any *output data*
        to be generated by the instance) is based on this method's
        result, but is then updated with all items from the **`defaults`**
        [constructor argument][StructuredLogsFormatter] (if given), and
        adjusted by applying the **[`prepare_value`][]** method to each
        value, and -- then -- by deleting each key to which a *void*
        value is assigned (by *void* value we mean any *falsy* value
        that is *not equal* to `0`, for example: [`None`][], `""`,
        `[]` or `{}` -- but _**not:**_ [`False`][], `0`, `0.0`, etc.).
    """
    return {}

make_base_auto_makers

make_base_auto_makers() -> Mapping[str, ValueProvider[object] | DottedPath]

A hook method: extend it in a subclass to define (more) auto-makers for output.

Automatically invoked on the formatter initialization. Each key in the resultant mapping needs to be an output data key, and each value in that mapping needs to be either an auto-maker or a dotted path (importable dotted name) that points to an auto-maker. Each auto-maker is supposed to be an argumentless function (or a callable object of some other type) returning – whenever it is called – a candidate for an output data value (to be assigned to the respective output data key).

See also

You may want to learn more about auto-makers themselves by referring to the description of the register_log_record_attr_auto_maker function (the machinery of StructuredLogsFormatter automatically makes use of that function, as appropriate).

It should also be noted, given how the default implementation of the get_prepared_output_data method works, that every candidate for an output data value – including those produced by auto-makers – will be transformed by applying the prepare_value method to it. Furthermore, whenever the result of this transformation is a void value (by which we mean any falsy value not equal to 0, for example: None, "", [] or {} – but not False, 0, 0.0, etc.), then the respective key will not be included in the output data dict (even if a default value is defined for that key; and even if that key was among those included in the set returned by the get_output_keys_required_in_defaults_or_auto_makers method when the formatter instance was initialized).

The default implementation of this method provides a couple of auto-makers which acquire some basic information about the execution environment (e.g., the Python version).

Related interfaces

For every instance, the auto_makers mapping (which is supposed to specify all auto-makers related to the instance) is based on this method’s result, but is then updated with all items from the auto_makers constructor argument (if given), and – then – adjusted by prefixing each key with the value of the auto_made_record_attr_prefix attribute (which is an automatically generated opaque string, created separately for each instance of StructuredLogsFormatter, guaranteed to be unique within a Python interpreter run).

(Therefore – concerning the stuff produced by a particular formatter instance’s auto-makers – the respective output data items will be obtained by picking those log record object’s attributes whose names are prefixed with the formatter’s auto_made_record_attr_prefix and using those names with that prefix removed as the corresponding output data keys. On the other hand, the formatter will ignore any record attribute names prefixed with auto_made_record_attr_prefix of any other formatter instances, as if such attributes did not exist. Thanks to that, more than one StructuredLogsFormatter can be used – and they will work independently of each other, handling just one’s own auto-made stuff.)

Source code in src/certlib/log.py

def make_base_auto_makers(self) -> Mapping[str, ValueProvider[object] | DottedPath]:
    """
    A hook method: extend it in a subclass to define (more)
    *auto-makers* for output.

    Automatically invoked on the formatter initialization. Each key
    in the resultant mapping needs to be an *output data* key, and
    each value in that mapping needs to be either an *auto-maker*
    or a *dotted path* (*importable dotted name*) that points to an
    *auto-maker*. Each *auto-maker* is supposed to be an argumentless
    function (or a callable object of some other type) returning --
    whenever it is called -- a candidate for an *output data* value
    (to be assigned to the respective *output data* key).

    !!! info "See also"

        You may want to learn more about *auto-makers*
        themselves by referring to the description of the
        **[`register_log_record_attr_auto_maker`][]** function
        (the machinery of **`StructuredLogsFormatter`**
        automatically makes use of that function, as appropriate).

    It should also be noted, given how the default implementation of the
    [`get_prepared_output_data`][] method works, that every candidate for
    an *output data* value -- including those produced by *auto-makers*
    -- will be transformed by applying the [`prepare_value`][] method
    to it. Furthermore, whenever the result of this transformation
    is a *void* value (by which we mean any *falsy* value *not equal*
    to `0`, for example: [`None`][], `""`, `[]` or `{}` -- but
    _**not**_ [`False`][], `0`, `0.0`, etc.), then the respective
    key will *not* be included in the *output data* dict (*even*
    if a *default value* is defined for that key; and *even* if
    that key was among those included in the set returned by the
    [`get_output_keys_required_in_defaults_or_auto_makers`][] method
    when the formatter instance was initialized).

    The default implementation of this method provides a couple
    of *auto-makers* which acquire some basic information about
    the execution environment (e.g., the Python version).

    !!! info "Related interfaces"

        For every instance, the **[`auto_makers`][]** mapping
        (which is supposed to specify all *auto-makers* related
        to the instance) is based on this method's result, but
        is then updated with all items from the **`auto_makers`**
        [constructor argument][StructuredLogsFormatter] (if given),
        and -- then -- adjusted by prefixing each key with the value
        of the **[`auto_made_record_attr_prefix`][]** attribute
        (which is an automatically generated opaque string, created
        separately for each instance of **`StructuredLogsFormatter`**,
        guaranteed to be unique within a Python interpreter run).

        (Therefore -- concerning the stuff produced by a particular
        formatter instance's *auto-makers* -- the respective
        *output data* items will be obtained by picking those
        log record object's attributes whose names are prefixed
        with the formatter's **`auto_made_record_attr_prefix`**
        and using those names *with that prefix removed* as the
        corresponding *output data* keys. On the other hand, the
        formatter will *ignore* any record attribute names prefixed
        with **`auto_made_record_attr_prefix`** of any *other*
        formatter instances, as if such attributes did not exist.
        Thanks to that, more than one **`StructuredLogsFormatter`**
        can be used -- and they will work independently of each
        other, handling just one's own *auto-made* stuff.)
    """
    return {
        key: make_constant_value_provider(value)
        for key, value in (
            ('py_ver', '.'.join(map(str, (sys.version_info or ())))),
            ('script_args', tuple(sys.argv or ())),
        )
    }

make_base_record_attr_to_output_key

make_base_record_attr_to_output_key() -> Mapping[str, str | None]

A hook method: extend it in a subclass to modify the mapping of log record attribute names to actual output data keys.

Automatically invoked on the formatter initialization. Each key in the resultant mapping needs to be the name of a (perhaps just hypothetic) log record attribute, and each value in that mapping needs to be either the corresponding output data key or None. In the latter case – given how the default implementation of the get_prepared_output_data method works – the attribute will always be omitted whenever output data is generated (note that this does not apply to log record attributes not included in the mapping).

The default implementation of this method returns a mapping that contains all items of STANDARD_RECORD_ATTR_TO_OUTPUT_KEY. In many cases this will be quite sufficient.

Related interfaces

For every instance, the record_attr_to_output_key mapping (which is supposed to specify the ultimate mapping of log record objects’ attribute names to actual keys in output data) is based on this method’s result, but is then updated with items suitably derived from auto_makers (to cause that any log record attribute name prefixed with the instance’s auto_made_record_attr_prefix is mapped to an output data key being just the unprefixed version of that name; see the Related interfaces note in the description of the make_base_auto_makers method).

Source code in src/certlib/log.py

def make_base_record_attr_to_output_key(self) -> Mapping[str, str | None]:
    """
    A hook method: extend it in a subclass to modify the mapping of
    [log record attribute names](https://docs.python.org/3/library/logging.html#logrecord-attributes)
    to actual *output data* keys.

    Automatically invoked on the formatter initialization. Each key
    in the resultant mapping needs to be the name of a (perhaps just
    hypothetic) log record attribute, and each value in that mapping
    needs to be either the corresponding *output data* key or [`None`][].
    In the latter case -- given how the default implementation of the
    [`get_prepared_output_data`][] method works -- the attribute will
    always be omitted whenever *output data* is generated (note that
    this does *not* apply to log record attributes *not included* in
    the mapping).

    The default implementation of this method returns a mapping that
    contains all items of [`STANDARD_RECORD_ATTR_TO_OUTPUT_KEY`][].
    In many cases this will be quite sufficient.

    !!! info "Related interfaces"

        For every instance, the **[`record_attr_to_output_key`][]**
        mapping (which is supposed to specify the ultimate mapping of
        log record objects' attribute names to actual keys in *output
        data*) is based on this method's result, but is then updated
        with items suitably derived from **[`auto_makers`][]** (to
        cause that any log record attribute name prefixed with the
        instance's **[`auto_made_record_attr_prefix`][]** is mapped
        to an *output data* key being just the unprefixed version of
        that name; see the *Related interfaces* note in the description
        of the **[`make_base_auto_makers`][]** method).
    """
    return dict(STANDARD_RECORD_ATTR_TO_OUTPUT_KEY)

format_timestamp

format_timestamp(
    record: logging.LogRecord,
    *,
    timezone: dt.tzinfo | None = dt.timezone.utc,
    timestamp_as_datetime: Callable[
        [float, dt.tzinfo | None], dt.datetime
    ] = dt.datetime.fromtimestamp,
    utc_offset_to_custom_suffix: Mapping[
        dt.timedelta | None, str
    ] = types.MappingProxyType(
        {dt.timedelta(0): "Z", None: " <UNCERTAIN TIMEZONE>"}
    )
) -> str

A hook method: extend/override it in a subclass to modify/redefine how, for each log entry, the formatted timestamp (asctime) is determined.

This method is invoked by the formatTime method, with a log record (typically, an instance of logging.LogRecord) as the sole argument. The log record is expected to have its created attribute already set to a float number representing a Unix timestamp.

What should be returned by this method is a string (presumably, derived somehow from the aforementioned created attribute of the log record) that will later be assigned (by the format method) to the log record’s asctime attribute.

The default implementation of this method should be sufficient in most cases. It converts the value of the given log record’s created attribute to a string being an ISO-8601-compliant date and time representation, with microsecond resolution. If no optional keyword arguments are given (which is how this method is invoked by formatTime), the resultant time representation is a UTC one (with Z, rather than +00:00, as its suffix), e.g.: "2026-03-15 13:48:56.726403Z".

Note

The logging.Formatter-specific attributes related to timestamp formatting (converter, default_time_format and default_msec_format) are ignored.

Tip

When extending this method in a subclass, you may want to make your custom implementation invoke the default one with some keyword arguments specified. In such a case, you may want to reach for their default values defined by the signature of StructuredLogsFormatter.format_timestamp; if so, refer to the StructuredLogsFormatter.FORMAT_TIMESTAMP_DEFAULT_KWARGS mapping. For example:

import datetime as dt
import types
from certlib.log import StructuredLogsFormatter

class EstTimezoneOrientedStructuredLogsFormatter(StructuredLogsFormatter):

    UTC_OFFSET_FOR_EST = dt.timedelta(hours=(-5))

    DEFAULT_TIMEZONE = dt.timezone(UTC_OFFSET_FOR_EST)
    DEFAULT_UTC_OFFSET_TO_CUSTOM_SUFFIX = types.MappingProxyType({

        # Let's use the base class's stuff in a *DRY* manner...
        **StructuredLogsFormatter.FORMAT_TIMESTAMP_DEFAULT_KWARGS[
            'utc_offset_to_custom_suffix'
        ],

        # ...and extend it with this-class-specific stuff:
        **{
            # If the suffix were to be `-05:00`,
            # we want it to be ` EST` instead.
            UTC_OFFSET_FOR_EST: ' EST',
        },
    })

    def format_timestamp(
        self,
        record,
        *,
        timezone=DEFAULT_TIMEZONE,
        utc_offset_to_custom_suffix=DEFAULT_UTC_OFFSET_TO_CUSTOM_SUFFIX,
        **kwargs,
    ):
        return super().format_timestamp(
            record,
            timezone=timezone,
            utc_offset_to_custom_suffix=utc_offset_to_custom_suffix,
            **kwargs,
        )

Source code in src/certlib/log.py

def format_timestamp(
    self,
    record: logging.LogRecord,
    *,
    timezone: dt.tzinfo | None = dt.timezone.utc,
    timestamp_as_datetime: Callable[[float, dt.tzinfo | None], dt.datetime] = (
        dt.datetime.fromtimestamp
    ),
    utc_offset_to_custom_suffix: Mapping[dt.timedelta | None, str] = types.MappingProxyType({
        # By default, if the suffix were to be
        # `+00:00`, we want it to be `Z` instead
        # (as it means the same but is shorter).
        dt.timedelta(0): 'Z',

        # By default, if there is no explicit
        # timezone information, we want to
        # emphasize this in a visible way.
        None: ' <UNCERTAIN TIMEZONE>',
    }),
) -> str:
    """
    A hook method: extend/override it in a subclass to modify/redefine
    how, for each log entry, the *formatted timestamp* (`asctime`) is
    determined.

    This method is invoked by the [`formatTime`][] method, with a log
    record (typically, an instance of [`logging.LogRecord`][]) as the
    sole argument. The log record is expected to have its [`created`
    attribute](https://docs.python.org/3/library/logging.html#logrecord-attributes)
    already set to a [`float`][] number representing a Unix timestamp.

    What should be returned by this method is a string (presumably,
    derived somehow from the aforementioned `created` attribute of
    the log record) that will later be assigned (by the [`format`][]
    method) to the log record's `asctime` attribute.

    The default implementation of this method should be sufficient
    in most cases. It converts the value of the given log record's
    `created` attribute to a string being an *ISO-8601-compliant*
    date and time representation, with *microsecond* resolution.
    If *no optional keyword arguments* are given (which is how
    this method is invoked by `formatTime`), the resultant time
    representation is a *UTC* one (with `Z`, rather than `+00:00`,
    as its suffix), e.g.: `"2026-03-15 13:48:56.726403Z"`.

    !!! note

        The [`logging.Formatter`][]-specific attributes related to
        timestamp formatting (`converter`, `default_time_format` and
        `default_msec_format`) are _**ignored**_.

    !!! tip

        When extending this method in a subclass, you may want to make
        your custom implementation invoke the default one with some
        keyword arguments specified. In such a case, you may want to
        reach for their default values defined by the signature of
        **[`StructuredLogsFormatter.format_timestamp`][]**; if so, refer
        to the **[`StructuredLogsFormatter.FORMAT_TIMESTAMP_DEFAULT_KWARGS`][]**
        mapping. For example:

        ```python
        import datetime as dt
        import types
        from certlib.log import StructuredLogsFormatter

        class EstTimezoneOrientedStructuredLogsFormatter(StructuredLogsFormatter):

            UTC_OFFSET_FOR_EST = dt.timedelta(hours=(-5))

            DEFAULT_TIMEZONE = dt.timezone(UTC_OFFSET_FOR_EST)
            DEFAULT_UTC_OFFSET_TO_CUSTOM_SUFFIX = types.MappingProxyType({

                # Let's use the base class's stuff in a *DRY* manner...
                **StructuredLogsFormatter.FORMAT_TIMESTAMP_DEFAULT_KWARGS[
                    'utc_offset_to_custom_suffix'
                ],

                # ...and extend it with this-class-specific stuff:
                **{
                    # If the suffix were to be `-05:00`,
                    # we want it to be ` EST` instead.
                    UTC_OFFSET_FOR_EST: ' EST',
                },
            })

            def format_timestamp(
                self,
                record,
                *,
                timezone=DEFAULT_TIMEZONE,
                utc_offset_to_custom_suffix=DEFAULT_UTC_OFFSET_TO_CUSTOM_SUFFIX,
                **kwargs,
            ):
                return super().format_timestamp(
                    record,
                    timezone=timezone,
                    utc_offset_to_custom_suffix=utc_offset_to_custom_suffix,
                    **kwargs,
                )
        ```
    """
    dt_timestamp = timestamp_as_datetime(record.created, timezone)
    custom_suffix = utc_offset_to_custom_suffix.get(dt_timestamp.utcoffset())
    if custom_suffix is None:
        return dt_timestamp.isoformat(' ', 'microseconds')
    dt_without_tzinfo = dt_timestamp.replace(tzinfo=None)
    return f"{dt_without_tzinfo.isoformat(' ', 'microseconds')}{custom_suffix}"

get_prepared_output_data

get_prepared_output_data(record: logging.LogRecord) -> dict[str, OutputValue]

A hook method: extend/override it in a subclass to modify/redefine how an output data dict is obtained from a log record object (which, at least typically, is a logging.LogRecord instance).

Subclass behavior restriction

This method should not mutate the given log record or any data it carries (regardless of the level of nesting, if any nested data is present). If some data needs to be modified, a completely new data object(s) should be created.

Moreover, each output data dict returned by this method should be a newly created dict (never the original log record’s __dict__ itself), so that during later stages of processing it will be safe to add, remove or replace any top-level items in that dict.

Doing otherwise will result in undefined behavior.

The default implementation of this method should be sufficient in most cases. To build a new output data dict, it digs into the given log record (and if that log record’s msg attribute is an ExtendedMessage instance – also into that instance…). While doing that, it also looks at the formatter attributes: record_attr_to_output_key (when determining output data keys; see also: make_base_record_attr_to_output_key) and defaults (to suitably complement the extracted output data with default items; see also: make_base_defaults), as well as makes intensive use of the prepare_value method (to ensure that each value in the resultant output data dict will be prepared for serialization). To make this description comprehensive, several details – regarding the resultant output data dict’s top-level keys and values – need to be clarified:

• when those keys and values are being determined based on the log record’s contents, any log record attributes that have been created by auto-makers belonging to some other instances of StructuredLogsFormatter (i.e., not belonging to self) are excluded from consideration, meaning no output data items are created from them (for certain low-level details, see the Related interfaces note in the description of the make_base_auto_makers method…);

• all existing log record attributes whose names are mapped in record_attr_to_output_key to some output data keys – are being included in the output data dict; this applies, in particular, to any log record attributes that have been created by auto-makers belonging to this (self) instance of StructuredLogsFormatter (see the Related interfaces note in the description of the make_base_record_attr_to_output_key method…);

• all existing log record attributes whose names are mapped in record_attr_to_output_key to None – are being excluded;

• all existing log record attributes not created by any auto-maker and not included in record_attr_to_output_key – are being included (!) in the output data dict, using each record attribute name as the corresponding output data key (as if it was mapped in record_attr_to_output_key to itself);

• only keys that are instances of str are ever included (meaning that any non-string keys, even if they appeared at some stage of processing, are always excluded), and every key is truncated to a maximum length of 200 characters (if it was longer); compare this with the treatment of nested keys (see the description of the prepare_submapping_key method…);

• when it comes to transforming every value by applying the aforementioned prepare_value method to it, if the result of this transformation turns out to be a void value (by which we mean any falsy value not equal to 0, for example: None, "", [] or {} – but not: False, 0, 0.0, etc.), then the respective key is excluded (even if it should be included according to any other rule described above; and even if some default value is defined for that key!); note that nested values, even if void, are never subject to such an exclusion (at least if the default implementations of prepare_value and prepare_submapping_key are used);

• potential item collisions (which might occur, for example, when some key is present both in the ExtendedMessage’s data mapping and among other data obtained from the log record’s content, and the value to be assigned to that key varies depending on which of those two sources of information is checked) – are avoided by suffixing problematic keys with one or more underscore character(s), as needed to prevent key duplication; such cases are expected to be rare.

Note

The said key truncation occurs before the said key deduplication – so it is possible, although very rare in practice, that appending underscore(s) to certain keys (as described above) will result in some keys ending up a little longer than 200 characters.

Source code in src/certlib/log.py

def get_prepared_output_data(self, record: logging.LogRecord) -> dict[str, OutputValue]:
    """
    A hook method: extend/override it in a subclass to modify/redefine
    how an *output data* dict is obtained from a log record object
    (which, at least typically, is a [`logging.LogRecord`][] instance).

    !!! warning "Subclass behavior restriction"

        This method should *not* mutate the given log record or any
        data it carries (regardless of the level of nesting, if any
        nested data is present). If some data needs to be modified,
        a completely *new* data object(s) should be created.

        Moreover, each *output data* dict returned by this method
        should be a *newly created* [`dict`][] (*never* the original
        log record's `__dict__` itself), so that during later stages
        of processing it will be safe to add, remove or replace any
        *top-level* items in that dict.

        Doing otherwise will result in undefined behavior.

    The default implementation of this method should be sufficient
    in most cases. To build a new *output data* dict, it digs into
    the given log record (and if that log record's `msg` attribute is
    an [`ExtendedMessage`][] instance -- also into that instance...).
    While doing that, it also looks at the formatter attributes:
    [`record_attr_to_output_key`][] (when determining *output data*
    keys; see also: [`make_base_record_attr_to_output_key`][]) and
    [`defaults`][] (to suitably complement the extracted *output
    data* with *default items*; see also: [`make_base_defaults`][]),
    as well as makes intensive use of the [`prepare_value`][] method
    (to ensure that each value in the resultant *output data* dict
    will be prepared for serialization). To make this description
    comprehensive, several details -- regarding the resultant *output
    data* dict's **top-level** *keys* and *values* -- need to be
    clarified:

    * when those **keys** and **values** are being determined based
      on the log record's contents, any log record attributes that
      have been created by *auto-makers* belonging to some *other*
      instances of `StructuredLogsFormatter` (i.e., *not* belonging
      to `self`) are *excluded* from consideration, meaning no *output
      data* items are created from them (for certain low-level details,
      see the *Related interfaces* note in the description of the
      [`make_base_auto_makers`][] method...);

    * all existing log record attributes whose names are mapped in
      [`record_attr_to_output_key`][] to some *output data* **keys**
      -- are being *included* in the *output data* dict; this applies,
      in particular, to any log record attributes that have been
      created by *auto-makers* belonging to *this* (`self`) instance
      of `StructuredLogsFormatter` (see the *Related interfaces* note
      in the description of the [`make_base_record_attr_to_output_key`][]
      method...);

    * all existing log record attributes whose names are mapped
      in `record_attr_to_output_key` to [`None`][] -- are being
      *excluded*;

    * all existing log record attributes *not* created by any
      *auto-maker* and *not* included in `record_attr_to_output_key`
      -- are being *included* (!) in the *output data* dict, using
      each record attribute name as the corresponding *output data*
      **key** (as if it was mapped in `record_attr_to_output_key` to
      itself);

    * *only* **keys** that are instances of [`str`][] are ever included
      (meaning that any non-string keys, even if they appeared at some
      stage of processing, are always *excluded*), and every key is
      *truncated* to a maximum length of 200 characters (if it was
      longer); compare this with the treatment of *nested keys* (see
      the description of the [`prepare_submapping_key`][] method...);

    * when it comes to transforming every **value** by applying the
      aforementioned `prepare_value` method to it, if the result of
      this transformation turns out to be a *void* value (by which
      we mean any *falsy* value *not equal* to `0`, for example:
      [`None`][], `""`, `[]` or `{}` -- but _**not:**_ [`False`][],
      `0`, `0.0`, etc.), then the respective **key** is *excluded*
      (*even* if it should be included according to any other rule
      described above; and *even* if some *default value* is defined
      for that key!); note that *nested* values, even if *void*, are
      *never* subject to such an *exclusion* (at least if the default
      implementations of `prepare_value` and `prepare_submapping_key`
      are used);

    * potential *item collisions* (which might occur, for example,
      when some **key** is present *both* in the `ExtendedMessage`'s
      [`data`][ExtendedMessage.data] mapping *and* among other data
      obtained from the log record's content, and the **value** to be
      assigned to that key varies depending on which of those two
      sources of information is checked) -- are avoided by suffixing
      problematic keys with one or more underscore character(s), as
      needed to prevent key duplication; such cases are expected to
      be rare.

    !!! note

        The said key truncation occurs *before* the said key
        deduplication -- so it is possible, although very rare
        in practice, that appending underscore(s) to certain keys
        (as described above) will result in some keys ending up
        a little longer than 200 characters.
    """
    output_data: dict[str, OutputValue] = {}
    actual_defaults = dict(self.defaults)
    handle_output_item = functools.partial(
        self._handle_output_item,
        self._DESIRED_MAX_KEY_LENGTH,
        self.prepare_value,
        actual_defaults,
        output_data,
    )

    xm_instance = getattr(record, 'msg', None)
    if isinstance(xm_instance, ExtendedMessage):
        self._extract_output_from_xm(record, xm_instance, handle_output_item)
    else:
        xm_instance = None

    self._extract_output_from_record(record, xm_instance, handle_output_item)

    for key, value_prepared in actual_defaults.items():
        output_data.setdefault(key, value_prepared)

    return output_data

prepare_value

prepare_value(
    value: object,
    *,
    to_str_types: tuple[type, ...] = (
        dt.date,
        dt.datetime,
        dt.time,
        decimal.Decimal,
        enum.Enum,
        fractions.Fraction,
        ipaddress.IPv4Address,
        ipaddress.IPv4Interface,
        ipaddress.IPv4Network,
        ipaddress.IPv6Address,
        ipaddress.IPv6Interface,
        ipaddress.IPv6Network,
        uuid.UUID,
    ),
    pass_thru_types: tuple[type, ...] = (str, int, float, bool, type(None)),
    exclude_from_seq_types: tuple[type, ...] = (str, bytes, bytearray),
    is_dataclass: Callable[[object], bool] = dataclasses.is_dataclass,
    dataclass_as_dict: Callable[[Any], dict[str, Any]] = dataclasses.asdict,
    last_resort: Callable[[object], str] = repr,
    **kwargs: Any
) -> OutputValue

A hook method: extend/override it in a subclass to modify/redefine how every value in an output data dict is prepared before the actual data serialization.

Subclass behavior restriction

This method should not mutate its argument or anything inside it (regardless of the level of nesting, if any nested data is present). If some data needs to be modified, a completely new value should be created (to be used as the prepared value to be returned), without mutating any existing objects. Doing otherwise will result in undefined behavior.

The default implementation of this method should be sufficient in most cases. It converts any value (even such one that is deeply nested inside sequences/mappings – thanks to recursive calls, always passing all keyword arguments from the parent call…) to a form that can be serialized with json.dumps (and which is – hopefully – short yet still readable, especially regarding instances of such types as: exceptions, dataclasses, typical named tuples, enum.Enum, uuid.UUID as well as the essential types from the datetime and ipaddress modules). When it comes to preparing any keys contained in a value which is a mapping (e.g., a dict) – see the prepare_submapping_key method…

Tip

When extending this method in a subclass, you may want to make your custom implementation invoke the default one with some keyword arguments specified. In such a case, you may want to reach for their default values defined by the signature of StructuredLogsFormatter.prepare_value; if so, refer to the StructuredLogsFormatter.PREPARE_VALUE_DEFAULT_KWARGS mapping. For example:

import array, pprint
import attrs   # <- 3rd party package used just in this example
from certlib.log import StructuredLogsFormatter

_BASE_KWARGS = StructuredLogsFormatter.PREPARE_VALUE_DEFAULT_KWARGS

class MyEnhancedStructuredLogsFormatter(StructuredLogsFormatter):

    @staticmethod
    def default_is_dataclass(obj):
        base_is_dataclass = _BASE_KWARGS['is_dataclass']
        return base_is_dataclass(obj) or attrs.has(type(obj))

    @staticmethod
    def default_dataclass_as_dict(obj):
        base_is_dataclass = _BASE_KWARGS['is_dataclass']
        base_dataclass_as_dict = _BASE_KWARGS['dataclass_as_dict']
        return (base_dataclass_as_dict(obj) if base_is_dataclass(obj)
                else attrs.asdict(obj))

    def prepare_value(
        self,
        value,
        *,
        exclude_from_seq_types = (
            *_BASE_KWARGS['exclude_from_seq_types'],
            memoryview,
            array.array,
        ),
        is_dataclass=default_is_dataclass,
        dataclass_as_dict=default_dataclass_as_dict,
        last_resort=pprint.pformat,
        **kwargs,
    ):
        return super().prepare_value(
            value,
            exclude_from_seq_types=exclude_from_seq_types,
            is_dataclass=is_dataclass,
            dataclass_as_dict=dataclass_as_dict,
            last_resort=last_resort,
            **kwargs,
        )

Source code in src/certlib/log.py

def prepare_value(
    self,
    value: object,
    *,
    to_str_types: tuple[type, ...] = (
        dt.date, dt.datetime, dt.time,
        decimal.Decimal, enum.Enum, fractions.Fraction,
        ipaddress.IPv4Address, ipaddress.IPv4Interface, ipaddress.IPv4Network,
        ipaddress.IPv6Address, ipaddress.IPv6Interface, ipaddress.IPv6Network,
        uuid.UUID,
    ),
    pass_thru_types: tuple[type, ...] = (str, int, float, bool, type(None)),
    exclude_from_seq_types: tuple[type, ...] = (str, bytes, bytearray),
    is_dataclass: Callable[[object], bool] = dataclasses.is_dataclass,
    dataclass_as_dict: Callable[[Any], dict[str, Any]] = dataclasses.asdict,
    last_resort: Callable[[object], str] = repr,
    **kwargs: Any,
) -> OutputValue:
    """
    A hook method: extend/override it in a subclass to modify/redefine
    how every *value* in an *output data* dict is prepared before the
    actual data serialization.

    !!! warning "Subclass behavior restriction"

        This method should *not* mutate its argument or anything
        inside it (regardless of the level of nesting, if any nested
        data is present). If some data needs to be modified, a
        completely *new* value should be created (to be used as
        the prepared value to be returned), *without* mutating any
        existing objects. Doing otherwise will result in undefined
        behavior.

    The default implementation of this method should be sufficient
    in most cases. It converts any *value* (even such one that is
    deeply nested inside sequences/mappings -- thanks to recursive
    calls, always passing all keyword arguments from the parent
    call...) to a form that can be serialized with [`json.dumps`][]
    (and which is -- hopefully -- short yet still readable, especially
    regarding instances of such types as: [*exceptions*][BaseException],
    [*dataclasses*][], typical [*named tuples*][collections.namedtuple],
    [`enum.Enum`][], [`uuid.UUID`][] as well as the essential types
    from the [`datetime`][] and [`ipaddress`][] modules). When it
    comes to preparing any *keys* contained in a *value* which is
    a mapping (e.g., a [`dict`][]) -- see the
    [`prepare_submapping_key`][] method...

    !!! tip

        When extending this method in a subclass, you may want to make
        your custom implementation invoke the default one with some
        keyword arguments specified. In such a case, you may want to
        reach for their default values defined by the signature of
        **[`StructuredLogsFormatter.prepare_value`][]**; if so, refer to
        the **[`StructuredLogsFormatter.PREPARE_VALUE_DEFAULT_KWARGS`][]**
        mapping. For example:

        ```python
        import array, pprint
        import attrs   # <- 3rd party package used just in this example
        from certlib.log import StructuredLogsFormatter

        _BASE_KWARGS = StructuredLogsFormatter.PREPARE_VALUE_DEFAULT_KWARGS

        class MyEnhancedStructuredLogsFormatter(StructuredLogsFormatter):

            @staticmethod
            def default_is_dataclass(obj):
                base_is_dataclass = _BASE_KWARGS['is_dataclass']
                return base_is_dataclass(obj) or attrs.has(type(obj))

            @staticmethod
            def default_dataclass_as_dict(obj):
                base_is_dataclass = _BASE_KWARGS['is_dataclass']
                base_dataclass_as_dict = _BASE_KWARGS['dataclass_as_dict']
                return (base_dataclass_as_dict(obj) if base_is_dataclass(obj)
                        else attrs.asdict(obj))

            def prepare_value(
                self,
                value,
                *,
                exclude_from_seq_types = (
                    *_BASE_KWARGS['exclude_from_seq_types'],
                    memoryview,
                    array.array,
                ),
                is_dataclass=default_is_dataclass,
                dataclass_as_dict=default_dataclass_as_dict,
                last_resort=pprint.pformat,
                **kwargs,
            ):
                return super().prepare_value(
                    value,
                    exclude_from_seq_types=exclude_from_seq_types,
                    is_dataclass=is_dataclass,
                    dataclass_as_dict=dataclass_as_dict,
                    last_resort=last_resort,
                    **kwargs,
                )
        ```
    """
    if isinstance(value, to_str_types):
        return str(value)

    if isinstance(value, pass_thru_types):
        return value

    kwargs.update(
        to_str_types=to_str_types,
        pass_thru_types=pass_thru_types,
        exclude_from_seq_types=exclude_from_seq_types,
        is_dataclass=is_dataclass,
        dataclass_as_dict=dataclass_as_dict,
        last_resort=last_resort,
    )

    if isinstance(value, Mapping):
        # Any *mapping* => convert it to a *dict*.
        prepare_key = self.prepare_submapping_key
        prepare_value = self.prepare_value
        return {
            prepare_key(key): prepare_value(val, **kwargs)
            for key, val in value.items()
        }

    if isinstance(value, type):
        # A runtime *type* (*class*) => convert it to a *str*...
        module = getattr(value, '__module__', '<unknown module>')
        qualname = getattr(value, '__qualname__', '<unknown type>')
        full_qualified_type_name = (
            qualname if module == 'builtins'
            else f'{module}.{qualname}'
        )
        return self.prepare_value(full_qualified_type_name, **kwargs)

    if isinstance(value, BaseException):
        # An *exception instance* => convert it to a *dict* of the
        # crucial exception's components (type, arguments, etc.).
        exc_components = {
            key: val
            for key, val in (
                ('exc_type', type(value)),
                ('args', getattr(value, 'args', None)),
                ('dict', getattr(value, '__dict__', None)),
            )
            if val
        }
        return self.prepare_value(exc_components, **kwargs)

    if isinstance(value, Sequence) and not isinstance(value, exclude_from_seq_types):
        seq = cast(Sequence[object], value)

        if (len(seq) == 3
              and seq[0] is type(seq[1])
              and isinstance(seq[1], BaseException)
              and (seq[2] is None
                   or type(seq[2]).__name__ == 'traceback')):
            # A sequence of 3 items: exception type, that type's
            # instance and traceback (or None) => treat it as if
            # it was just the *exception instance*...
            return self.prepare_value(seq[1], **kwargs)

        if (isinstance(seq, tuple)
              and callable(dict_from_this := getattr(seq, '_asdict', None))):
            # A tuple (presumably, a *named tuple*) with an
            # `_asdict()` method => try to use that method
            # to convert this tuple (presumably, to a *dict*).
            try:
                d = dict_from_this()
            except TypeError:
                pass
            else:
                return self.prepare_value(d, **kwargs)

        # Some other sequence => convert it to a *list*.
        prepare_value = self.prepare_value
        return [
            prepare_value(val, **kwargs)
            for val in seq
        ]

    if is_dataclass(value):
        # A *dataclass instance* (we're sure it's not a type, see the
        # type-dedicated check earlier...) => convert it to a *dict*.
        return self.prepare_value(dataclass_as_dict(value), **kwargs)

    # Any other object...
    return last_resort(value)

prepare_submapping_key

prepare_submapping_key(key: object) -> str

A hook method: extend/override it in a subclass to modify/redefine how to prepare, before the actual data serialization, every key in every mapping (e.g., in a dict) being a value inside an output data dict (possibly deeply nested within it).

The default implementation of this method should be sufficient in most cases. It applies str to the given key (converting it to a string if it was not one already) and truncates the result to a maximum length of 200 characters (if longer).

Note

prepare_value is what invokes this method – so (let us stress that!) this method is not applied to top-level keys in the output data dict, but is applied to each key in every mapping that prepare_value takes as an input value (also, in every dict created by prepare_value as a result of converting an exception, named tuple or dataclass instance…). All of this is true for the default implementation of prepare_value. It is recommended (yet not enforced) that any custom implementations of the prepare_value method make use of this method in a similar way.

Source code in src/certlib/log.py

def prepare_submapping_key(self, key: object) -> str:
    """
    A hook method: extend/override it in a subclass to modify/redefine
    how to prepare, before the actual data serialization, every *key*
    in every mapping (e.g., in a [`dict`][]) being a *value* inside an
    *output data* dict (possibly deeply nested within it).

    The default implementation of this method should be sufficient in
    most cases. It applies [`str`][] to the given key (converting it
    to a string if it was not one already) and truncates the result to
    a maximum length of 200 characters (if longer).

    !!! note

        **[`prepare_value`][]** is what invokes this method -- so (let
        us stress that!) this method is *not* applied to top-level
        keys in the *output data* dict, but *is* applied to *each key*
        in every mapping that **`prepare_value`** takes as an input
        *value* (also, in every dict created by **`prepare_value`**
        as a result of converting an *exception*, *named tuple* or
        *dataclass* instance...). All of this is true for the default
        implementation of **`prepare_value`**. It is recommended
        (yet not enforced) that any custom implementations of the
        **`prepare_value`** method make use of *this* method in a
        similar way.
    """
    key_str = str(key)
    if len(key_str) > self._DESIRED_MAX_KEY_LENGTH:
        key_str = key_str[:self._DESIRED_MAX_KEY_LENGTH]
    return key_str

serialize_prepared_output_data

serialize_prepared_output_data(output_data: dict[str, OutputValue]) -> str

A hook method: extend/override it in a subclass to modify/redefine the output data serialization procedure.

Subclass behavior restriction

While it is OK to add, remove or replace top-level items in the given output data dict, this method should never mutate any object inside it (regardless of the level of nesting). If some data needs to be modified, completely new data object(s) should be created – to entirely replace the respective top-level value in the dict (without mutating the original object(s) being replaced). Doing otherwise will result in undefined behavior.

The default implementation of this method should be sufficient in most cases. It just applies the serializer callable to the given output data dict, and returns the result.

Related interfaces

By default, the serializer attribute is set to the standard json.dumps function, but this can be changed by specifying the serializer argument when invoking the StructuredLogsFormatter constructor.

Source code in src/certlib/log.py

def serialize_prepared_output_data(self, output_data: dict[str, OutputValue]) -> str:
    """
    A hook method: extend/override it in a subclass to modify/redefine
    the *output data serialization* procedure.

    !!! warning "Subclass behavior restriction"

        While it is OK to add, remove or replace *top-level* items
        in the given *output data* dict, this method should *never*
        mutate any object inside it (regardless of the level of
        nesting). If some data needs to be modified, completely *new*
        data object(s) should be created -- to entirely *replace* the
        respective *top-level* value in the dict (*without* mutating
        the original object(s) being replaced). Doing otherwise will
        result in undefined behavior.

    The default implementation of this method should be sufficient in
    most cases. It just applies the [`serializer`][] callable to the
    given *output data* dict, and returns the result.

    !!! info "Related interfaces"

        By default, the **[`serializer`][]** attribute is set to the
        standard [`json.dumps`][] function, but this can be changed
        by specifying the **`serializer`** argument when invoking the
        **[`StructuredLogsFormatter`][]** constructor.
    """
    return self.serializer(output_data)

---

xm: Final = ExtendedMessage module-attribute

xm is a convenience alias of ExtendedMessage.

ExtendedMessage

ExtendedMessage(
    pattern: str = "",
    /,
    *args: object | ValueProvider[object],
    exc_info: Any = None,
    stack_info: bool = False,
    stacklevel: int = 1,
    **data: object | ValueProvider[object],
)

ExtendedMessage(
    data: Mapping[str, object | ValueProvider[object]],
    /,
    *,
    exc_info: Any = None,
    stack_info: bool = False,
    stacklevel: int = 1,
)

ExtendedMessage(
    pattern: object,
    /,
    *args: object | ValueProvider[object],
    exc_info: Any = None,
    stack_info: bool = False,
    stacklevel: int = 1,
    **data: object | ValueProvider[object],
)

A tool thanks to which you can:

• conveniently emit structured log entries, especially if StructuredLogsFormatter is in use;

• use the modern {}-based style of log message formatting, or – if you just need to log pure data – simply omit passing the text message pattern (regardless of what formatter is in use);

• defer the creation of some values (if it is costly) until the log entry is actually about to be emitted (regardless of what formatter is in use).

There is a convenience alias of this class: xm. As being very short, it is simply much more ergonomic than the actual class name – given that this tool is intended to be used every time you log something…

Usage examples:

import datetime as dt, hashlib, ipaddress, logging, sys
from certlib.log import xm

logging.info(xm("Hello {}!", sys.platform))
logging.info(xm("Maxsize is {maxsize:x}", maxsize=sys.maxsize))
logging.warning(xm(
    connection_count=42,
    client_ip=ipaddress.IPv4Address("192.168.0.121"),
    local_time=dt.datetime.now(),
    # Value creation deferred until log entry is about to be emitted:
    payload_hash=lambda: hashlib.sha256(b'...payload...').hexdigest(),
))
some_data_dict = globals()
logging.debug(xm(some_data_dict))

See also

For extra information about ExtendedMessage, including a bunch of usage examples, see the Tool: xm section of the User’s Guide.

Constructor arguments (all optional):

• first positional argument (default: ""): the text message pattern. Expected to be a string, or any truthy object that could be converted to a string by applying str to it. The pattern may contain {}-formatting-style replacement fields (perhaps with a '!'-separated conversion marker, and/or a ':'-separated format spec). The given object is assigned to the pattern attribute intact, unless it is falsy – then it is ignored, and just "" (empty string) is assigned to that attribute.

• extra positional arguments (if any): positional args to format the text message (they need to match positional/numbered replacement fields in the text message pattern – see the first positional argument described above). A tuple of these arguments is assigned to the args attribute.

• exc_info (keyword-only; default: None): its usage and related behavior are nearly identical to those of the same-named argument to logging.Logger’s methods (see the relevant fragment of the documentation for the logging module). This argument is assigned to the exc_info attribute.

• stack_info (keyword-only; default: False): its usage and related behavior are nearly identical to those of the same-named argument to logging.Logger’s methods (see the relevant fragment of the documentation for the logging module). This argument is assigned to the stack_info attribute.

• stacklevel (keyword-only; default: 1): its usage and related behavior are nearly identical to those of the same-named argument to logging.Logger’s methods (see the relevant fragment of the documentation for the logging module). This argument is assigned to the stacklevel attribute.

• extra keyword arguments (if any): all of them become extra data items – to be included in the output data dict if StructuredLogsFormatter is used, or to be appended to the text message (in a form resembling the keyword arguments syntax) if some other formatter is in use. A dict of those extra data items is always stored as the data attribute. Moreover, any items whose names match some named replacement fields in the text message pattern (specified as the first positional argument) will take part in formatting the actual text message (regardless of what formatter is in use).

Alternatively, a mapping (e.g., a dict) of extra data items can be passed to the constructor as the first positional argument. The effect is the same as if each of its items was passed as an extra keyword argument (without passing any positional arguments). The mapping, after conversion to a dict, is assigned to the data attribute.

Interface restriction

If a mapping is passed as the first positional argument, then passing any other arguments except exc_info, stack_info and stacklevel causes TypeError.

When it comes to the arguments exc_info, stack_info and stacklevel, they should not be included in that mapping (doing so will result in undefined behavior). Each of them, if to be specified, should only be specified as a real keyword argument.

Interface restriction

A string.templatelib.Template object (which is typically created by evaluating a t-string) should not be passed as the first positional argument (doing so will result in undefined behavior). This possibility is reserved for future versions of certlib.log, in which dedicated support for such objects may be provided.

Interface restriction

If you pass a stack_info and/or stacklevel argument to the ExtendedMessage (xm) constructor, you should not pass stack_info or stacklevel to the related logger method call (doing so will result in undefined behavior).

# All WRONG (!!!):
logger.info(xm('Foo', stack_info=True), stack_info=True)
logger.info(xm('Foo', stack_info=True), stacklevel=2)
logger.info(xm('Foo', stacklevel=2), stack_info=True)
logger.info(xm('Foo', stacklevel=2), stacklevel=2)
logger.info(xm('Foo', stack_info=True), stack_info=True, stacklevel=2)
logger.info(xm('Foo', stack_info=True, stacklevel=2), stack_info=True)
logger.info(xm('Foo', stack_info=True, stacklevel=2), stack_info=True, stacklevel=2)
# (and similar...)

# OK:
logger.info(xm('Foo', stack_info=True))
logger.info(xm('Foo', stacklevel=2))
logger.info(xm('Foo', stack_info=True, stacklevel=2))

# Also OK:
logger.info(xm('Foo'), stack_info=True)
logger.info(xm('Foo'), stacklevel=2)
logger.info(xm('Foo'), stack_info=True, stacklevel=2)

Whenever a formatter (of any type) processes a log record whose msg attribute (which typically is just what has been passed to the logger method call as the first positional argument) is an ExtendedMessage (xm) instance, that instance’s method get_message_value is invoked: either directly – by the StructuredLogsFormatter’s machinery; or indirectly, via __str__ – by the standard machinery that other formatter types use.

Interface restriction

When you pass an instance of ExtendedMessage as the first positional argument to a logger method call, you should not pass to that call any other positional arguments (doing so will result in undefined behavior).

# WRONG (!!!):
logger.info(xm('{}, {} and {}'), 'Athos', 'Porthos', 'Aramis')

# OK:
logger.info(xm('{}, {} and {}', 'Athos', 'Porthos', 'Aramis'))

Note

If a text message pattern (not a mapping) is passed to the ExtendedMessage (xm) constructor as the first positional argument and no extra positional or keyword arguments are provided (except exc_info, stack_info and stacklevel) – that is, if both of the attributes args and data of the resultant ExtendedMessage instance are empty – then the get_message_value method will not attempt to format the text message with str.format; instead, it will treat the pattern as an already formatted text message.

logger.info(xm('answer: {}', 42))  # message will be 'answer: 42'
logger.info(xm('answer: {}'))      # message will be 'answer: {}'  [sic!]

This mimics how the standard logging machinery handles a log record whose args attribute is empty (when only a text message pattern is specified, without any values to be interpolated).

If any extra positional or keyword arguments to the constructor – except exc_info, stack_info and stacklevel – or any values included in the extra data mapping (if passed to the constructor as the first positional argument) are function or method objects (precisely: instances of any types included in ExtendedMessage.recognized_callable_arg_or_data_item_types), then – as part of processing the ExtendedMessage instance by a formatter – each of those functions/methods will be called to obtain the actual value, which will then replace (respectively, in args or data) the called function/method.

To be precise: all those calls-and-replacements will be triggered when any of the following methods is invoked on the ExtendedMessage instance for the first time: get_message_value, get_record_msg_and_args_equivalent_info, __str__ or iter_str_parts (with the proviso that the last one returns an iterator which, to achieve the effect in question, needs to be iterated over, at least partially). Each of those calls-and-replacements will be made at most once per instance of ExtendedMessage (see the _ensure_callable_args_and_data_items_resolved method’s description…).

Thanks to that mechanism, if the creation of some value is expected to be costly, you can wrap it in a function/method (in particular, in an argumentless lambda) to defer that costly operation until the value becomes necessary (which may never happen if, for example, the specified log level is lower than the configured threshold). Such a function/method is expected to take no arguments (therefore, if it is a method, it should already be bound to some instance or class).

Multithreading-related restriction

None of those functions/methods should acquire any locks that might also be acquired by any code making use of some logging stuff (because, in particular, that could result in a deadlock).

Source code in src/certlib/log.py

def __init__(
    self,
    first_arg: object | Mapping[str, object | ValueProvider[object]] = '',
    /,
    *args: object | ValueProvider[object],
    exc_info: Any = None,
    stack_info: bool = False,
    stacklevel: int = 1,
    **data: object | ValueProvider[object],
):
    # Note: it is not necessary to protect the following 4
    # lines with a lock, because each call to the function
    # `_ensure_internal_record_hook_is_set_up()` is itself
    # **thread-safe** and **idempotent**; so any redundant
    # (even if concurrent) calls to that function are safe.
    if self._setup_of_record_hooks_still_needs_to_be_done:
        _ensure_internal_record_hook_is_set_up(self._exc_info_record_hook)
        _ensure_internal_record_hook_is_set_up(self._stack_stuff_record_hook)
        self.__class__._setup_of_record_hooks_still_needs_to_be_done = False

    pattern: object
    if isinstance(first_arg, Mapping):
        if args or data:
            raise TypeError(
                f"{type(self).__qualname__}'s *extra data* items, "
                f"if any, must be passed to its constructor either "
                f"by keyword arguments or as a mapping being the "
                f"only positional argument (not both)"
            )
        pattern = ''
        data = dict(first_arg)
    else:
        pattern = first_arg

    self.pattern = pattern or ''
    self.args = args
    self.data = data
    self.exc_info = exc_info
    self.stack_info = stack_info
    self.stacklevel = stacklevel

    self._callable_args_and_data_items_already_resolved: bool = False
    self._callable_args_and_data_items_resolving_lock: threading.Lock | None = None
    self._cached_message: str | None = None

recognized_callable_arg_or_data_item_types class-attribute

recognized_callable_arg_or_data_item_types: tuple[
    type[ValueProvider[object]], ...
]

For ExtendedMessage, this tuple contains the runtime types of function and bound method objects – both in the user-defined and built-in variants (precisely: types.FunctionType, types.BuiltinFunctionType, types.MethodType and types.MethodWrapperType). You can override this attribute in your subclass to redefine the runtime types of values in args and data that shall be called to obtain the actual values (see the part of the ExtendedMessage constructor’s description containing a reference to this attribute…).

pattern instance-attribute

pattern: object

args instance-attribute

args: tuple[object | ValueProvider[object], ...]

data instance-attribute

data: dict[str, object | ValueProvider[object]]

exc_info instance-attribute

exc_info: Any

stack_info instance-attribute

stack_info: bool

stacklevel instance-attribute

stacklevel: int

get_message_value

get_message_value() -> str

Automatically invoked by the StructuredLogsFormatter’s machinery to obtain a string to be assigned to the log record’s message attribute.

Interface restriction

Once this method is invoked on an ExtendedMessage instance, any attempts (regarding that instance) to replace/mutate any of the objects assigned to the pattern, args and data attributes or anything inside them (regardless of the level of nesting, if any nested data is present) – are no loger allowed. Doing so will result in undefined behavior.

The default implementation of this method should be sufficient in most cases. It converts pattern to a string, and then – only if args and/or data contain any items – invokes that string’s format method, passing to it all items of args as positional arguments and all items of data as keyword arguments. A string being the result of the above operation(s) is cached (for any further invocations of this method on the same instance) and returned.

Subclass behavior requirement

This method should always invoke the _ensure_callable_args_and_data_items_resolved method before starting the actual work (the default implementation already does that). Failing to do so will result in undefined behavior.

Note

Apart from the aforementioned use by the machinery of StructuredLogsFormatter, this method is also invoked by the ExtendedMessage’s implementation of __str__ (which is important for formatters that are not instances of StructuredLogsFormatter).

Source code in src/certlib/log.py

def get_message_value(self) -> str:
    """
    Automatically invoked by the [`StructuredLogsFormatter`][]'s
    machinery to obtain a string to be assigned to the log record's
    [`message` attribute](https://docs.python.org/3/library/logging.html#logrecord-attributes).

    !!! warning "Interface restriction"

        Once this method is invoked on an **`ExtendedMessage`** instance,
        any attempts (regarding that instance) to replace/mutate any of
        the objects assigned to the **[`pattern`][]**, **[`args`][]** and
        **[`data`][]** attributes or anything inside them (regardless of
        the level of nesting, if any nested data is present) -- are *no
        loger* allowed. Doing so will result in undefined behavior.

    The default implementation of this method should be sufficient
    in most cases. It converts [`pattern`][] to a string, and then
    -- *only* if [`args`][] and/or [`data`][] contain any items --
    invokes that string's [`format`][str.format] method, passing to
    it all items of `args` as *positional arguments* and all items
    of `data` as *keyword arguments*. A string being the result of
    the above operation(s) is cached (for any further invocations
    of this method on the same instance) and returned.

    !!! warning "Subclass behavior requirement"

        This method should *always* invoke the
        **[`_ensure_callable_args_and_data_items_resolved`][]**
        method before starting the actual work (the default
        implementation already does that). Failing to do so
        will result in undefined behavior.

    !!! note

        Apart from the aforementioned use by the machinery
        of **`StructuredLogsFormatter`**, this method is also
        invoked by the **`ExtendedMessage`**'s implementation
        of **[`__str__`][]** (which is important for formatters
        that are *not* instances of **`StructuredLogsFormatter`**).
    """
    self._ensure_callable_args_and_data_items_resolved()

    # (Compare to the source code of `logging.LogRecord.getMessage()`...)
    message = self._cached_message
    if message is None:
        message = str(self.pattern)
        if self.args or self.data:
            message = message.format(*self.args, **self.data)

        # (Such assignments are assumed to be *atomic* operations.)
        self._cached_message = message

    return message

get_record_msg_and_args_equivalent_info

get_record_msg_and_args_equivalent_info(
    *, pattern_result_key: str | None, args_result_key: str | None
) -> Mapping[str, object]

Automatically invoked by the StructuredLogsFormatter’s machinery to get a value to be included in the output data dict under the key corresponding to the log record’s msg attribute. The returned value is supposed to be a mapping that conveys relevant information from the ExtendedMessage instance – to the extent that corresponds to the information typically conveyed by the msg and args attributes of log records when ExtendedMessage is not used.

Interface restriction

Once this method is invoked on an ExtendedMessage instance, any attempts (regarding that instance) to replace/mutate any of the objects assigned to the args and data attributes or anything inside them (regardless of the level of nesting, if any nested data is present) – are no loger allowed. Doing so will result in undefined behavior.

The default implementation should be sufficient in most cases. It returns a mapping containing zero, one or two items. Specifically – each of the following if the key is not None and the value is not falsy:

• the given pattern_result_key – mapped to the value of the pattern attribute,

• the given args_result_key – mapped to the value of the args attribute.

Subclass behavior requirement

This method should always invoke the _ensure_callable_args_and_data_items_resolved method before starting the actual work (the default implementation already does that). Failing to do so will result in undefined behavior.

Source code in src/certlib/log.py

def get_record_msg_and_args_equivalent_info(
    self,
    *,
    pattern_result_key: str | None,
    args_result_key: str | None,
) -> Mapping[str, object]:
    """
    Automatically invoked by the [`StructuredLogsFormatter`][]'s
    machinery to get a value to be included in the *output data*
    dict under the key corresponding to the log record's `msg`
    attribute. The returned value is supposed to be a mapping
    that conveys relevant information from the `ExtendedMessage`
    instance -- to the extent that corresponds to the information
    typically conveyed by the `msg` and `args` attributes of log
    records when `ExtendedMessage` is not used.

    !!! warning "Interface restriction"

        Once this method is invoked on an **`ExtendedMessage`** instance,
        any attempts (regarding that instance) to replace/mutate any of
        the objects assigned to the **[`args`][]** and **[`data`][]**
        attributes or anything inside them (regardless of the level
        of nesting, if any nested data is present) -- are *no loger*
        allowed. Doing so will result in undefined behavior.

    The default implementation should be sufficient in most cases. It
    returns a mapping containing zero, one or two items. Specifically
    -- *each* of the following *if* the key is not [`None`][] and the
    value is not *falsy*:

    * the given **`pattern_result_key`** -- mapped to the value of the
      [`pattern`][] attribute,

    * the given **`args_result_key`** --  mapped to the value of the
      [`args`][] attribute.

    !!! warning "Subclass behavior requirement"

        This method should *always* invoke the
        **[`_ensure_callable_args_and_data_items_resolved`][]**
        method before starting the actual work (the default
        implementation already does that). Failing to do so
        will result in undefined behavior.
    """
    self._ensure_callable_args_and_data_items_resolved()

    return {
        key: val
        for key, val in (
            (pattern_result_key, self.pattern),
            (args_result_key, self.args),
        )
        if (key is not None) and val
    }

__str__

__str__() -> str

Invoked when str is applied to an ExtendedMessage instance. This is done, in particular, by the machinery related to typical non-StructuredLogsFormatter formatters (specifically, by the log record method getMessage) – to obtain a string to be assigned to the message attribute of the log record.

Interface restriction

Once this method is invoked on an ExtendedMessage instance, any attempts (regarding that instance) to replace/mutate any of the objects assigned to the pattern, args and data attributes or anything inside them (regardless of the level of nesting, if any nested data is present) – are no loger allowed. Doing so will result in undefined behavior.

The default implementation of this method should be sufficient in most cases. It invokes the iter_str_parts method (which, in particular, invokes get_message_value…) and concatenates any yielded strings (if more than one) using " | " as the separator.

Subclass behavior requirement

This method should always invoke the _ensure_callable_args_and_data_items_resolved method before starting the actual work (the default implementation already does that). Failing to do so will result in undefined behavior.

Source code in src/certlib/log.py

def __str__(self) -> str:
    """
    Invoked when [`str`][] is applied to an `ExtendedMessage` instance.
    This is done, in particular, by the machinery related to typical
    non-`StructuredLogsFormatter` formatters (specifically, by the
    log record method [`getMessage`][logging.LogRecord.getMessage])
    -- to obtain a string to be assigned to the [`message`
    attribute](https://docs.python.org/3/library/logging.html#logrecord-attributes)
    of the log record.

    !!! warning "Interface restriction"

        Once this method is invoked on an **`ExtendedMessage`** instance,
        any attempts (regarding that instance) to replace/mutate any of
        the objects assigned to the **[`pattern`][]**, **[`args`][]** and
        **[`data`][]** attributes or anything inside them (regardless of
        the level of nesting, if any nested data is present) -- are *no
        loger* allowed. Doing so will result in undefined behavior.

    The default implementation of this method should be sufficient
    in most cases. It invokes the [`iter_str_parts`][] method
    (which, in particular, invokes [`get_message_value`][]...)
    and concatenates any yielded strings (if more than one) using
    `" | "` as the separator.

    !!! warning "Subclass behavior requirement"

        This method should *always* invoke the
        **[`_ensure_callable_args_and_data_items_resolved`][]**
        method before starting the actual work (the default
        implementation already does that). Failing to do so
        will result in undefined behavior.
    """
    self._ensure_callable_args_and_data_items_resolved()

    return ' | '.join(self.iter_str_parts())

__repr__

__repr__() -> str

Invoked when repr is applied to an ExtendedMessage instance (typically, for debug purposes).

The default implementation of this method should be sufficient in most cases. It invokes the iter_argument_reprs method, concatenates any yielded strings (if more than one) using ", " as the separator, adds the parentheses, and prefixes the whole thing with the class name.

Source code in src/certlib/log.py

@reprlib.recursive_repr(fillvalue='<...>')
def __repr__(self) -> str:
    """
    Invoked when [`repr`][] is applied to an `ExtendedMessage`
    instance (typically, for debug purposes).

    The default implementation of this method should be sufficient
    in most cases. It invokes the [`iter_argument_reprs`][] method,
    concatenates any yielded strings (if more than one) using `", "`
    as the separator, adds the parentheses, and prefixes the whole
    thing with the class name.
    """
    type_name = type(self).__qualname__
    arguments_repr = ', '.join(self.iter_argument_reprs())
    return f'{type_name}({arguments_repr})'

iter_str_parts

iter_str_parts() -> Iterator[str]

Invoked by the __str__ method.

Interface restriction

Once this method is invoked on an ExtendedMessage instance, any attempts (regarding that instance) to replace/mutate any of the objects assigned to the pattern, args and data attributes or anything inside them (regardless of the level of nesting, if any nested data is present) – are no loger allowed. Doing so will result in undefined behavior.

The default implementation of this method yields zero, one or two strings. Specifically – each of the following if not empty:

• the result of an invocation of the get_message_value method,

• a representation of the data mapping’s items (formatted in a way that resembles the syntax for specifying keyword arguments, but without the parentheses).

Subclass behavior requirement

This method should always invoke the _ensure_callable_args_and_data_items_resolved method before starting the actual work (the default implementation already does that). Failing to do so will result in undefined behavior.

Source code in src/certlib/log.py

def iter_str_parts(self) -> Iterator[str]:
    """
    Invoked by the [`__str__`][] method.

    !!! warning "Interface restriction"

        Once this method is invoked on an **`ExtendedMessage`** instance,
        any attempts (regarding that instance) to replace/mutate any of
        the objects assigned to the **[`pattern`][]**, **[`args`][]** and
        **[`data`][]** attributes or anything inside them (regardless of
        the level of nesting, if any nested data is present) -- are *no
        loger* allowed. Doing so will result in undefined behavior.

    The default implementation of this method yields zero, one
    or two strings. Specifically -- *each* of the following *if
    not empty*:

    * the result of an invocation of the [`get_message_value`][]
      method,

    * a representation of the [`data`][] mapping's items (formatted
      in a way that resembles the syntax for specifying keyword
      arguments, but without the parentheses).

    !!! warning "Subclass behavior requirement"

        This method should *always* invoke the
        **[`_ensure_callable_args_and_data_items_resolved`][]**
        method before starting the actual work (the default
        implementation already does that). Failing to do so
        will result in undefined behavior.
    """
    self._ensure_callable_args_and_data_items_resolved()

    if formatted_message := self.get_message_value():
        yield formatted_message
    if formatted_data_items := ', '.join(
        f'{key}={val!a}' for key, val in self.data.items()
    ):
        yield formatted_data_items

iter_argument_reprs

iter_argument_reprs() -> Iterator[str]

Invoked by the __repr__ method.

The default implementation of this method yields string representations of the arguments to the ExtendedMessage (xm) constructor which would be needed to create an instance equivalent to this one (self).

Source code in src/certlib/log.py

def iter_argument_reprs(self) -> Iterator[str]:
    """
    Invoked by the [`__repr__`][] method.

    The default implementation of this method yields string
    representations of the arguments to the [`ExtendedMessage`][]
    ([`xm`][]) constructor which would be needed to create an
    instance equivalent to this one (`self`).
    """
    if self.args or self.pattern:
        yield repr(self.pattern)
    if self.args:
        yield from map(repr, self.args)
    if self.exc_info is not None:
        yield f'exc_info={self.exc_info!r}'
    if self.stack_info is not False:  # noqa
        yield f'stack_info={self.stack_info!r}'
    if self.stacklevel != 1 or type(self.stacklevel) is not int:
        yield f'stacklevel={self.stacklevel!r}'
    for key, val in self.data.items():
        yield f'{key}={val!r}'

_ensure_callable_args_and_data_items_resolved

_ensure_callable_args_and_data_items_resolved() -> None

Interface exclusion

This method is not part of the API – except that it is allowed to be invoked by any methods implemented by possible subclasses of ExtendedMessage.

This method processes the items of args and data – by calling each encountered instance of any type included in ExtendedMessage.recognized_callable_arg_or_data_item_types, and then replacing that value with the result of that call. Each of those calls is made without arguments.

This method can be safely invoked multiple times on the same instance, even in the case of concurrent invocations. The implementation guarantees that none of the calls in question will be made more than once per instance of ExtendedMessage.

Interface restriction

Once this method is invoked on an ExtendedMessage instance, any other attempts (regarding this instance) to replace/mutate any of the collections assigned to the args and data attributes or anything inside them (regardless of the level of nesting, if any nested data is present) – are no loger allowed. Doing so will result in undefined behavior.

Source code in src/certlib/log.py

def _ensure_callable_args_and_data_items_resolved(self) -> None:
    """
    !!! exclusion "Interface exclusion"

        This method is _**not**_ part of the API -- _**except that**_
        it is allowed to be invoked by any methods implemented by
        possible subclasses of **`ExtendedMessage`**.

    This method processes the items of [`args`][] and [`data`][] --
    by *calling* each encountered instance of any type included in
    [`ExtendedMessage.recognized_callable_arg_or_data_item_types`][],
    and then *replacing* that value with the result of that call.
    Each of those calls is made without arguments.

    This method can be safely invoked multiple times on the same
    instance, *even* in the case of *concurrent* invocations. The
    implementation guarantees that *none* of the calls in question
    will be made more than *once* per instance of `ExtendedMessage`.

    !!! warning "Interface restriction"

        Once this method is invoked on an **`ExtendedMessage`**
        instance, any other attempts (regarding this instance)
        to replace/mutate any of the collections assigned to the
        **[`args`][]** and **[`data`][]** attributes or anything
        inside them (regardless of the level of nesting, if any
        nested data is present) -- are *no loger* allowed. Doing
        so will result in undefined behavior.
    """
    if self._callable_args_and_data_items_already_resolved:
        # OK, already resolved (fast path).
        return

    with self._callable_args_and_data_items_resolving_meta_lock:
        # Obtain the instance's lock in a thread-safe manner...
        lock = self._callable_args_and_data_items_resolving_lock
        if lock is None:
            # (We want to defer its creation until this moment, so that
            # `ExtendedMessage.__init__()` remains as fast as possible.)
            lock = self._callable_args_and_data_items_resolving_lock = (
                threading.Lock()
            )

    if not lock.acquire(timeout=self._CALLABLE_ARGS_AND_DATA_ITEMS_RESOLVING_LOCK_TIMEOUT):
        raise RuntimeError(
            f'could not acquire the lock that protects the procedure '
            f'of ensuring that relevant callable items of the `args` '
            f'and `data` collections will be resolved'
        )
    try:
        if self._callable_args_and_data_items_already_resolved:
            # OK, already resolved.
            return
        self._resolve_callable_args_and_data_items()

        # (Such assignments are assumed to be *atomic* operations.)
        self._callable_args_and_data_items_already_resolved = True
    finally:
        lock.release()

---

make_constant_value_provider

make_constant_value_provider(value: T) -> ValueProvider[T]

A trivial (yet sometimes useful) helper: given an arbitrary object (value), create an argumentless function that will always return that object (note that such argumentless functions can be used as auto-makers).

Source code in src/certlib/log.py

def make_constant_value_provider(value: T) -> ValueProvider[T]:
    """
    A trivial (yet sometimes useful) helper: given an arbitrary object
    (**`value`**), create an argumentless function that will always
    return that object (note that such argumentless functions can be
    used as [*auto-makers*][register_log_record_attr_auto_maker]).
    """
    return (lambda: value)

---

register_log_record_attr_auto_maker

register_log_record_attr_auto_maker(
    rec_attr: str, auto_maker: ValueProvider[object]
) -> None

For the specified log record attribute name (rec_attr), register the given auto-maker callable (auto_maker).

By calling this function you ensure that, from now on, the specified attribute will be automatically set on every new log record object – to a value returned by the specified auto-maker.

The auto-maker needs to be an argumentless function or any other object that can be called with no arguments (see: ValueProvider). A call to it will be made at most once for each newly created log record (only if the logger is enabled for the respective log level), in the thread in which the current logger method call is being executed (shortly after the log record is created by a logger, yet before any handlers, filters and formatters process that record). Obviously, the returned values are allowed to vary depending on the context (or even with each call).

If, for the specified attribute name, some auto-maker is already registered, this function raises KeyError.

Tip

The get method of a ContextVar may be a good candidate for an auto-maker.

Note

Typically, you do not need to use the register_log_record_attr_auto_maker function directly, because the machinery of the StructuredLogsFormatter class does this for you – at instantiation time (see the descriptions of the StructuredLogsFormatter constructor’s auto_makers argument and the StructuredLogsFormatter’s make_base_auto_makers method).

That machinery will also take care of avoiding record attribute name collisions.

Warning

If you use this function directly, you need to take care of avoiding record attribute name collisions by yourself. When the internal auto-makers machinery attempts to assign an auto-maker-produced value to the respective attribute of a log record but the log record already has that attribute set, then KeyError is raised (which will typically bubble up to the caller of the currently executed logger method). This behavior mimics how the machinery of the standard logging module reacts to collisions between extra items and existing attributes of a log record.

Source code in src/certlib/log.py

def register_log_record_attr_auto_maker(
    rec_attr: str,
    auto_maker: ValueProvider[object],
) -> None:
    """
    For the specified log record attribute name (**`rec_attr`**),
    register the given *auto-maker* callable (**`auto_maker`**).

    By calling this function you ensure that, from now on, the specified
    attribute will be *automatically* set on every *new* [log record][logging.LogRecord]
    object -- to a value returned by the specified *auto-maker*.

    The _**auto-maker**_ needs to be an argumentless function or
    any other object that can be called with no arguments (see:
    [`ValueProvider`][]). A call to it will be made *at most once* for
    each newly created log record (*only* if the logger is enabled for
    the respective log level), in the thread in which the current logger
    method call is being executed (shortly *after* the log record is
    created by a [logger](https://docs.python.org/3/howto/logging.html#loggers),
    yet *before* any [handlers](https://docs.python.org/3/howto/logging.html#handlers),
    [filters](https://docs.python.org/3/library/logging.html#filter) and
    [formatters](https://docs.python.org/3/howto/logging.html#formatters)
    process that record). Obviously, the returned values are allowed to
    vary depending on the context (or even with each call).

    If, for the specified attribute name, some *auto-maker* is already
    registered, this function raises [`KeyError`][].

    !!! tip

        The [`get`][contextvars.ContextVar.get] method of
        a [`ContextVar`][contextvars.ContextVar] may be a
        good candidate for an *auto-maker*.

    !!! note

        Typically, you _**do not need**_ to use the
        **`register_log_record_attr_auto_maker`** function directly,
        because the machinery of the **`StructuredLogsFormatter`** class
        does this for you -- at instantiation time (see the descriptions of the
        [**`StructuredLogsFormatter`** constructor][StructuredLogsFormatter]'s
        **`auto_makers`** argument and the **`StructuredLogsFormatter`**'s
        **[`make_base_auto_makers`][StructuredLogsFormatter.make_base_auto_makers]**
        method).

        That machinery will also take care of avoiding record attribute
        name collisions.

    !!! warning

        If you use this function *directly*, you need to take care of
        avoiding record attribute name collisions by yourself. When
        the internal *auto-makers* machinery attempts to assign an
        *auto-maker*-produced value to the respective attribute of
        a log record but the log record already has that attribute set,
        then [`KeyError`][] is raised (which will typically bubble up
        to the caller of the currently executed logger method). This
        behavior mimics how the machinery of the standard [`logging`][]
        module reacts to collisions between *extra* items and existing
        attributes of a log record.
    """
    with _auto_makers_registry_and_internal_record_hooks_maintenance_lock:
        _ensure_record_factory_with_auto_makers_and_record_hooks_is_set()
        _add_to_auto_makers_registry(rec_attr, auto_maker)

---

unregister_log_record_attr_auto_maker

unregister_log_record_attr_auto_maker(rec_attr: str) -> None

For the given log record attribute name (rec_attr), unregister the previously registered auto-maker.

If, for the specified attribute name, no auto-maker is currently registered, this function raises KeyError.

Source code in src/certlib/log.py

def unregister_log_record_attr_auto_maker(
    rec_attr: str,
) -> None:
    """
    For the given log record attribute name (**`rec_attr`**), unregister
    the previously registered *auto-maker*.

    If, for the specified attribute name, no *auto-maker* is currently
    registered, this function raises [`KeyError`][].
    """
    with _auto_makers_registry_and_internal_record_hooks_maintenance_lock:
        _remove_from_auto_makers_registry(rec_attr)

---

COMMONLY_EXPECTED_NON_STANDARD_OUTPUT_KEYS module-attribute

COMMONLY_EXPECTED_NON_STANDARD_OUTPUT_KEYS: Final[Set[str]] = frozenset(
    {"system", "component", "component_type"}
)

---

STANDARD_RECORD_ATTR_TO_OUTPUT_KEY module-attribute

STANDARD_RECORD_ATTR_TO_OUTPUT_KEY: Final[Mapping[str, str | None]] = (
    types.MappingProxyType(
        {
            "asctime": "timestamp",
            "exc_info": "exc_info",
            "exc_text": "exc_text",
            "funcName": "func",
            "levelname": "level",
            "levelno": "levelno",
            "lineno": "lineno",
            "message": "message",
            "msg": "message_base",
            "name": "logger",
            "pathname": "src",
            "process": "pid",
            "processName": "process_name",
            "stack_info": "stack_info",
            "thread": "thread_id",
            "threadName": "thread_name",
            "taskName": "async_task_name",
            "args": None,
            "created": None,
            "filename": None,
            "module": None,
            "msecs": None,
            "relativeCreated": None,
        }
    )
)

---

Static Typing Helpers

Note

In your day-to-day work with the certlib.log library, you do not need to delve into this stuff.

Interface exclusion

The flavor of any type aliases – i.e., whether they are TypeAlias-annotated ones or type statement-made ones – is not part of the API.

ValueProvider

__call__() -> Value

A protocol which describes any callable object (e.g., a function) that takes no arguments and returns some value (returned values may vary with each call). It is worth noting that, in particular, every auto-maker is supposed to be such a callable object.

Typing details

• In the above __call__() signature, the Value element is a type variable.

• That variable has no upper bound (or, in other words, its upper bound is object).

• The ValueProvider protocol is generic. It is covariant in that variable.

OutputSerializer

__call__(output_data: dict[str, OutputValue], /) -> str

A protocol which describes a callable object (e.g., a function) that takes an output data dict (supposedly, returned by StructuredLogsFormatter.get_prepared_output_data) as the sole positional argument, and returns a string representing that dict in serialized form (typically, but not necessarily, in JSON format).

Interface restriction

While it is OK to add, remove or replace top-level items in an output data dict, a callable used as an OutputSerializer should never mutate any object inside such a dict (regardless of the level of nesting). If some data needs to be modified, completely new data object(s) should be created – to entirely replace the respective top-level value in the output data dict (without mutating the original object(s) being replaced).

Typing details

In the above __call__() signature, as everywhere else, the OutputValue element is just an alias for Any (see below).

OutputValue module-attribute

OutputValue = Any

A type alias which is used to annotate top-level values in output data dicts.

Required preparation–serialization compatibility

Typically, an output data dict (annotated as dict[str, OutputValue]) is:

• created by StructuredLogsFormatter.get_prepared_output_data – with each value obtained by invoking StructuredLogsFormatter.prepare_value (whose return type annotation is OutputValue);

• passed to StructuredLogsFormatter.serialize_prepared_output_data – and then passed by it to StructuredLogsFormatter.serializer (which is annotated as OutputSerializer).

So every serializer is required to be capable of serializing any dict that maps strings to values whose types, generally referred to as OutputValue, are decided by the (default or customized) implementation of prepare_value (and/or by customized implementations of get_prepared_output_data and/or serialize_prepared_output_data, if they change something in this regard).

Note

The json.dumps function, which is the default serializer, satisfies this requirement for the default implementations of prepare_value, get_prepared_output_data and serialize_prepared_output_data.

Typing details

Accurately expressing the above requirement using static types would be difficult (at least without making things overly complicated), so that approach is not taken. Instead, OutputValue is defined just as an alias for the Any special type.

DottedPath module-attribute

DottedPath = str

A type alias which is used to annotate strings being a dotted path (importable dotted name).

KwargsMappingAsLiteralEvaluableString module-attribute

KwargsMappingAsLiteralEvaluableString = str

A type alias which is used to annotate strings being an ast.literal_eval-evaluable representation of a mapping (dict) of keyword arguments that are compatible with the main (first) signature of the StructuredLogsFormatter constructor.