src.util.logs module

Utilities related to configuration and handling of framework logging.

class src.util.logs.MDTFConsoleHandler(stream=None)

Bases: logging.StreamHandler

Dummy class to designate logging to stdout or stderr from the root logger.

property name

terminator = '\n'

class src.util.logs.StringIOHandler

Bases: logging.StreamHandler

StreamHandler instance that writes log entries to an internal StringIO buffer.

__init__()

Initialize the handler.

If stream is not specified, sys.stderr is used.

reset_buffer()

close()

Tidy up any resources used by the handler.

This version removes the handler from an internal map of handlers, _handlers, which is used for handler lookup by name. Subclasses should ensure that this gets called from overridden close() methods.

buffer_contents()

Return contents of buffer as a string.

property name

terminator = '\n'

class src.util.logs.MultiFlushMemoryHandler(capacity, flushLevel=40, target=None, flushOnClose=True)

Bases: logging.handlers.MemoryHandler

Subclass MemoryHandler to enable flushing the contents of its log buffer to multiple targets. We do this to solve the chicken-and-egg problem of logging any events that happen before the log outputs are configured: those events are captured by an instance of this handler and then transfer()’ed to other handlers once they’re set up. See https://stackoverflow.com/a/12896092.

transfer(target_handler)

Transfer contents of buffer to target_handler.

Parameters

target_handler (Handler) – log handler to transfer contents of buffer to.

transfer_to_non_console(logger)

Transfer contents of buffer to all non-console-based handlers attached to logger (handlers that aren’t MDTFConsoleHandler.)

If no handlers are attached to the logger, a warning is printed and the buffer is transferred to the lastResort handler, i.e. printed to stderr.

Parameters

logger (Logger) – logger to transfer contents of buffer to.

property name

class src.util.logs.HeaderFileHandler(filename, mode='a', encoding=None, delay=False)

Bases: logging.FileHandler

Subclass FileHandler to print system and git repo information in a header at the start of a log file without writing it to other loggers (e.g. the console.)

property name

terminator = '\n'

class src.util.logs.MDTFHeaderFileHandler(filename, mode='a', encoding=None, delay=False)

Bases: src.util.logs.HeaderFileHandler

FileHandler which adds a header to log files with system information, git repo status etc. provided by mdtf_log_header().

property name

terminator = '\n'

class src.util.logs.HangingIndentFormatter(fmt=None, datefmt=None, style='%', tabsize=0, header='', footer='')

Bases: logging.Formatter

Formatter that applies a hanging indent, making it easier to tell where one entry stops and the next starts.

__init__(fmt=None, datefmt=None, style='%', tabsize=0, header='', footer='')

Initialize formatter with extra arguments.

Parameters

• fmt (str) – format string, as in Formatter.

• datefmt (str) – date format string, as in Formatter or strftime.

• style (str) – string templating style, as in Formatter.

• tabsize (int) – Number of spaces to use for hanging indent.

• header (str) – Optional constant string to prepend to each log entry.

• footer (str) – Optional constant string to append to each log entry.

format(record)

Format the specified LogRecord as text, adding indentation and header/footer.

Parameters

record (LogRecord) – Logging record object to be formatted.

Returns

String representation of the log entry.

This essentially repeats the method’s implementation in the python standard library. See comments there and the logging module documentation.

converter()

localtime([seconds]) -> (tm_year,tm_mon,tm_mday,tm_hour,tm_min,

tm_sec,tm_wday,tm_yday,tm_isdst)

Convert seconds since the Epoch to a time tuple expressing local time. When ‘seconds’ is not passed in, convert the current time instead.

default_msec_format = '%s,%03d'

default_time_format = '%Y-%m-%d %H:%M:%S'

class src.util.logs.GeqLevelFilter(name='', level=None)

Bases: src.util.logs._LevelFilterBase

Filter to include only log messages with a severity of level or worse. This is normally done by setting the level attribute on a Handler, but we need to add a filter when transferring records from another logger (the MultiFlushMemoryHandler cache), as shown in https://stackoverflow.com/a/24324246.

filter(record)

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

__init__(name='', level=None)

Initialize a filter.

Initialize with the name of the logger which, together with its children, will have its events allowed through the filter. If no name is specified, allow every event.

class src.util.logs.LtLevelFilter(name='', level=None)

Bases: src.util.logs._LevelFilterBase

Filter to include only log messages with a severity less than level.

filter(record)

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

__init__(name='', level=None)

Initialize a filter.

Initialize with the name of the logger which, together with its children, will have its events allowed through the filter. If no name is specified, allow every event.

class src.util.logs.EqLevelFilter(name='', level=None)

Bases: src.util.logs._LevelFilterBase

Filter to include only log messages with a severity equal to level.

filter(record)

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

__init__(name='', level=None)

Initialize a filter.

Initialize with the name of the logger which, together with its children, will have its events allowed through the filter. If no name is specified, allow every event.

class src.util.logs.NameMatchFilter(name='')

Bases: logging.Filter

Filter that only accepts log events directed to it specifically, rejecting all events coming from child loggers.

Intended to be attached to a handler – the effect of attaching this to a logger is the same as setting propagate = False on it.

__init__(name='')

Initialize a filter.

Initialize with the name of the logger which, together with its children, will have its events allowed through the filter. If no name is specified, allow every event.

filter(record)

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

class src.util.logs.TagMatchFilter(name='', tags=None)

Bases: logging.Filter

Filter which only accepts records having the designated combination of custom ‘tag’ attributes. These are assigned by the methods in MDTFObjectLogger or can be passed via the ‘extra’ kwarg on any logger (see discussion in entry for ~logging.Logger.)

__init__(name='', tags=None)

Initialize a filter.

Initialize with the name of the logger which, together with its children, will have its events allowed through the filter. If no name is specified, allow every event.

filter(record)

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

class src.util.logs.ObjectLogTag(value)

Bases: src.util.basic.MDTFEnum

Standardized values that the MDTF-defined tags attribute on LogRecord objects can take, and that TagMatchFilter can listen for. These specify different destinations for the logging events.

NC_HISTORY = 1

BANNER = 2

IN_FILE = 3

OUT_FILE = 4

class src.util.logs.MDTFObjectLogger(name)

Bases: logging.Logger

This class wraps functionality for use by MDTFObjectLoggerMixin for log record-keeping by objects in the object hierarchy:

• A Logger to record events affecting the parent object only. This logger does not propagate events up the log hierarchy: the module-level logger should be used if that functionality is desired.

• A queue (_exceptions) for holding Exception objects received by the parent object.

__init__(name)

Initialize the logger with a name and an optional level.

log(level, msg, *args, **kw)

Log ‘msg % args’ with the integer severity ‘level’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.log(level, “We have a %s”, “mysterious problem”, exc_info=1)

debug(msg, *args, **kw)

Log ‘msg % args’ with severity ‘DEBUG’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.debug(“Houston, we have a %s”, “thorny problem”, exc_info=1)

info(msg, *args, **kw)

Log ‘msg % args’ with severity ‘INFO’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.info(“Houston, we have a %s”, “interesting problem”, exc_info=1)

warning(msg, *args, **kw)

Log ‘msg % args’ with severity ‘WARNING’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.warning(“Houston, we have a %s”, “bit of a problem”, exc_info=1)

error(msg, *args, **kw)

Log ‘msg % args’ with severity ‘ERROR’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.error(“Houston, we have a %s”, “major problem”, exc_info=1)

critical(msg, *args, **kw)

Log ‘msg % args’ with severity ‘CRITICAL’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.critical(“Houston, we have a %s”, “major disaster”, exc_info=1)

exception(msg, *args, exc_info=True, **kw)

Convenience method for logging an ERROR with exception information.

property has_exceptions

Return boolean corresponding to whether this object has received any exceptions (via store_exception().)

store_exception(exc)

Add an Exception object exc to the internal list.

classmethod get_logger(log_name)

Workaround for setting the logger class, since logger objects have global state (calling getLogger with the same name returns the same object, like a Singleton.)

manager = <logging.Manager object>

root = <RootLogger root (WARNING)>

class src.util.logs.MDTFObjectLoggerMixinBase

Bases: object

Dummy base class acting as a parent for all logging mixin classes for elements of the object hierarchy.

__init__()

Initialize self. See help(type(self)) for accurate signature.

class src.util.logs.MDTFObjectLoggerMixin

Bases: src.util.logs.MDTFObjectLoggerMixinBase

Base class to implement per-object logging for objects in the object hierarchy. Based on https://stackoverflow.com/q/57813096.

This wraps related functionalities:

• A Logger to record events affecting the object only. Log messages are cached in a StringIO buffer.

• A method format() for formatting the contents of the above into a string, along with log messages of any child objects. This is intended for preparing per-POD and per-case log files; logging intended for the console should use the module loggers.

init_log(fmt=None)

Logger initialization. This is a mixin class, so we don’t define a __init__ method for simplicity.

init_extra_log_handlers()

Hook used by child classes to add class-specific log handlers.

property last_exception

Return most recent Exception received by the object.

format_log(children=True)

Return contents of log buffer, as well as that of any child objects in child_objs, as a formatted string.

__init__()

Initialize self. See help(type(self)) for accurate signature.

class src.util.logs.VarlistEntryLoggerMixin

Bases: src.util.logs.MDTFObjectLoggerMixin

Mixin providing per-object logging for VarlistEntry.

init_extra_log_handlers()

Hook used by child classes to add class-specific log handlers.

__init__()

Initialize self. See help(type(self)) for accurate signature.

format_log(children=True)

Return contents of log buffer, as well as that of any child objects in child_objs, as a formatted string.

init_log(fmt=None)

Logger initialization. This is a mixin class, so we don’t define a __init__ method for simplicity.

property last_exception

Return most recent Exception received by the object.

class src.util.logs.PODLoggerMixin

Bases: src.util.logs._CaseAndPODHandlerMixin, src.util.logs.MDTFObjectLoggerMixin

Mixin providing per-object logging for Diagnostic (POD objects.)

__init__()

Initialize self. See help(type(self)) for accurate signature.

format_log(children=True)

Return contents of log buffer, as well as that of any child objects in child_objs, as a formatted string.

init_extra_log_handlers()

Hook used by child classes to add class-specific log handlers.

init_log(fmt=None)

Logger initialization. This is a mixin class, so we don’t define a __init__ method for simplicity.

property last_exception

Return most recent Exception received by the object.

class src.util.logs.CaseLoggerMixin

Bases: src.util.logs._CaseAndPODHandlerMixin, src.util.logs.MDTFObjectLoggerMixinBase

Mixin providing per-object logging for DataSourceBase (case objects, corresponding to experiments.)

init_log(log_dir, fmt=None)

close_log_file(log=True)

__init__()

Initialize self. See help(type(self)) for accurate signature.

init_extra_log_handlers()

src.util.logs.git_info()

Get the current git branch, hash, and list of uncommitted files, if available. Based on NumPy’s implementation: https://stackoverflow.com/a/40170206.

src.util.logs.mdtf_log_header(title)

Returns string of system debug information to use as log file header. Calls git_info() to get repo status.

src.util.logs.signal_logger(caller_name, signum=None, frame=None, log=<Logger>)

Lookup signal name from number and write to log. Taken from https://stackoverflow.com/a/2549950.

Parameters

• caller_name (str) – Calling function name, only used in log message.

• signum – Signal number of the signal we recieved.

• frame – Parameters of the signal we recieved.

src.util.logs.initial_log_config()

Configure the root logger for logging to console and to a cache provided by MultiFlushMemoryHandler.

This is temporary logging configuration, used to solve the chicken-and-egg problem of logging any problems that happen before we’ve set up logging according to the user’s log config files, which requires doing a full parse of the user input. Once this is set up, any messages in the cache are sent to the user-configured logs.

src.util.logs.transfer_log_cache(target_log=None, close=False)

Transfer the contents of the root log cache (MultiFlushMemoryHandler) to logs on newly-configured objects.