src.util.logs module

Utilities related to configuration and handling of framework logging.

class src.util.logs.MDTFConsoleHandler(stream=None)

Bases: logging.StreamHandler

Dummy class to designate logging to stdout or stderr from the root logger.

class src.util.logs.StringIOHandler

Bases: logging.StreamHandler

StreamHandler instance that writes log entries to an internal StringIO buffer.

reset_buffer()

close()

Tidy up any resources used by the handler.

This version removes the handler from an internal map of handlers, _handlers, which is used for handler lookup by name. Subclasses should ensure that this gets called from overridden close() methods.

buffer_contents()

Return contents of buffer as a string.

class src.util.logs.MultiFlushMemoryHandler(capacity, flushLevel=40, target=None, flushOnClose=True)

Bases: logging.handlers.MemoryHandler

Subclass logging.handlers.MemoryHandler to enable flushing the contents of its log buffer to multiple targets. We do this to solve the chicken-and-egg problem of logging any events that happen before the log outputs are configured: those events are captured by an instance of this handler and then transfer()’ed to other handlers once they’re set up. See https://stackoverflow.com/a/12896092.

transfer(target_handler)

Transfer contents of buffer to target_handler.

Parameters

target_handler (logging.Handler) – log handler to transfer contents of buffer to.

transfer_to_non_console(logger)

Transfer contents of buffer to all non-console-based handlers attached to logger (handlers that aren’t MDTFConsoleHandler.)

If no handlers are attached to the logger, a warning is printed and the buffer is transferred to the logging.lastResort handler, i.e. printed to stderr.

Parameters

logger (logging.Logger) – logger to transfer contents of buffer to.

class src.util.logs.HeaderFileHandler(filename, mode='a', encoding=None, delay=False)

Bases: logging.FileHandler

Subclass logging.FileHandler to print system information to start of file without writing it to other loggers.

_log_header()

_open()

Write header information right after we open the log file, then proceed normally.

class src.util.logs.MDTFHeaderFileHandler(filename, mode='a', encoding=None, delay=False)

Bases: src.util.logs.HeaderFileHandler

_log_header()

src.util.logs._hanging_indent(str_, initial_indent, subsequent_indent)

Poor man’s indenter. Easier than using textwrap for this case.

Parameters

• str (str) – String to be indented.

• initial_indent (str) – string to insert as the indent for the first line.

• subsequent_indent (str) – string to insert as the indent for all subsequent lines.

Returns

Indented string.

class src.util.logs.HangingIndentFormatter(fmt=None, datefmt=None, style='%', tabsize=0, header='', footer='')

Bases: logging.Formatter

logging.Formatter that applies a hanging indent, making it easier to tell where one entry stops and the next starts.

format(record)

Format the specified logging.LogReco rd as text, adding indentation and header/footer.

Parameters

record (logging.LogRecord) – logging record to be formatted.

Returns

String representation of the log entry.

This essentially repeats the method’s implementation in the python standard library. See comments there and the logging module documentation.

class src.util.logs._LevelFilterBase(name='', level=None)

Bases: logging.Filter

Base class to implement common features of level-based filters.

class src.util.logs.GeqLevelFilter(name='', level=None)

Bases: src.util.logs._LevelFilterBase

logging.Filter to include only log messages with a severity of level or worse. This is normally done by setting the level attribute on a logging.Handler, but we need to add a filter when transferring records from another logger, as shown in https://stackoverflow.com/a/24324246.

filter(record)

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

class src.util.logs.LtLevelFilter(name='', level=None)

Bases: src.util.logs._LevelFilterBase

logging.Filter to include only log messages with a severity less than level.

filter(record)

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

class src.util.logs.EqLevelFilter(name='', level=None)

Bases: src.util.logs._LevelFilterBase

logging.Filter to include only log messages with a severity equal to level.

filter(record)

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

class src.util.logs.NameMatchFilter(name='')

Bases: logging.Filter

logging.Filter that rejects all log events coming from child loggers.

Intended to be attached to a handler – the effect of attaching this to a logger is the same as setting propagate = False on it.

filter(record)

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

class src.util.logs.TagMatchFilter(name='', tags=None)

Bases: logging.Filter

logging.Filter which only accepts records having the designated combination of ‘tag’ attributes. These are assigned by the methods in MDTFObjectLogger or can be passed via the ‘extra’ kwarg on any logger (see discussion in entry for <https://docs.python.org/3.7/library/logging.html#logging.Logger.debug>__.)

filter(record)

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

class src.util.logs.ObjectLogTag(value)

Bases: src.util.basic.MDTFEnum

Standardized values that the package-defined tags attribute on LogRecord objects can take, and that TagMatchFilter can listen for.

NC_HISTORY = 1

BANNER = 2

IN_FILE = 3

OUT_FILE = 4

class src.util.logs.MDTFObjectLogger(name)

Bases: logging.Logger

This class wraps functionality for use by MDTFObjectLoggerMixin:

• A Logger to record events affecting the parent object

  only. This logger does not propagate events up the log hierarchy: the module-level logger should be passed if that functionality is desired.

• A list for holding Exception objects received by the parent

  object.

log(level, msg, *args, **kw)

Log ‘msg % args’ with the integer severity ‘level’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.log(level, “We have a %s”, “mysterious problem”, exc_info=1)

debug(msg, *args, **kw)

Log ‘msg % args’ with severity ‘DEBUG’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.debug(“Houston, we have a %s”, “thorny problem”, exc_info=1)

info(msg, *args, **kw)

Log ‘msg % args’ with severity ‘INFO’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.info(“Houston, we have a %s”, “interesting problem”, exc_info=1)

warning(msg, *args, **kw)

Log ‘msg % args’ with severity ‘WARNING’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.warning(“Houston, we have a %s”, “bit of a problem”, exc_info=1)

error(msg, *args, **kw)

Log ‘msg % args’ with severity ‘ERROR’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.error(“Houston, we have a %s”, “major problem”, exc_info=1)

critical(msg, *args, **kw)

Log ‘msg % args’ with severity ‘CRITICAL’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.critical(“Houston, we have a %s”, “major disaster”, exc_info=1)

exception(msg, *args, exc_info=True, **kw)

Convenience method for logging an ERROR with exception information.

property has_exceptions

store_exception(exc)

classmethod get_logger(log_name)

class src.util.logs.MDTFObjectLoggerMixinBase

Bases: object

class src.util.logs.MDTFObjectLoggerMixin

Bases: src.util.logs.MDTFObjectLoggerMixinBase

Class to implement per-object logging via inheritance. Used for VarlistEntry and Diagnostic. Based on https://stackoverflow.com/q/57813096.

This wraps related functionalities:

• A Logger to record events affecting the object

  only. Log messages are cached in a StringIO buffer.

• A method format() for formatting the contents of

  the above into a string, along with log messages of any child objects. This is intended for preparing log files (at the per-POD or per-case level); logging intended for the console should use the module loggers.

init_log(fmt=None)

init_extra_log_handlers()

property last_exception

format_log(children=True)

Return contents of log buffer, as well as that of any child objects in child_objs, as a formatted string.

class src.util.logs.VarlistEntryLoggerMixin

Bases: src.util.logs.MDTFObjectLoggerMixin

Mixin configuring log functionality for VarlistEntry.

init_extra_log_handlers()

class src.util.logs._CaseAndPODHandlerMixin

Bases: object

Common methods for PODLoggerMixin, CaseLoggerMixin.

init_extra_log_handlers()

class src.util.logs.PODLoggerMixin

Bases: src.util.logs._CaseAndPODHandlerMixin, src.util.logs.MDTFObjectLoggerMixin

Mixin configuring log functionality for Diagnostic.

class src.util.logs.CaseLoggerMixin

Bases: src.util.logs._CaseAndPODHandlerMixin, src.util.logs.MDTFObjectLoggerMixinBase

Mixin configuring log functionality for DataSourceBase.

init_log(log_dir, fmt=None)

close_log_file(log=True)

src.util.logs.git_info()

Get the current git branch, hash, and list of uncommitted files, if available.

Called by DebugHeaderFileHandler._debug_header(). Based on NumPy’s implementation: https://stackoverflow.com/a/40170206.

src.util.logs.mdtf_log_header(title)

Returns string of system debug information to use as log file header. Calls git_info() to get repo status.

src.util.logs.signal_logger(caller_name, signum=None, frame=None, log=<Logger src.util.logs (WARNING)>)

Lookup signal name from number and write to log.

Taken from https://stackoverflow.com/a/2549950.

Parameters

• caller_name (str) – Calling function name, only used in log message.

• frame (signum,) – parameters of the signal we recieved.

src.util.logs._set_excepthook(root_logger)

Ensure all uncaught exceptions, other than user KeyboardInterrupt, are logged to the root logger.

See https://docs.python.org/3/library/sys.html#sys.excepthook.

src.util.logs._configure_logging_dict(log_d, log_args)

Convert CLI flags (--verbose/--quiet) into log levels. Configure log level and filters on console handlers in a logging config dictionary.

src.util.logs.initial_log_config()

Configure the root logger for logging to console and to a cache provided by MultiFlushMemoryHandler. For debugging purposes we want to get console output set up before we’ve read in the real log config files (which requires doing a full parse of the user input).

src.util.logs.transfer_log_cache(target_log=None, close=False)

Transfer the contents of the root log cache (MultiFlushMemoryHandler) to logs on newly-configured objects.