mlogger Module

mlogger.py

This module defines classes and functions that assist in logging.

There are multiple classes that work together to create our logging system:
  • LogManager - the main supervisor class that manages a collection of Loggers
  • Logger - responsible for saving/writing a log event to some destination, and matching log events to decide whether to handle them
  • LogFilter - class responsible for deciding if a log message should be handled by a Logger; each logger has a list of zero or more LogFilters to check for match
  • LogTarget - destination targets for loggers; each logger has a list of LogTargets to send to on match
How these classes co-exist:
  • A site has a single LogManager; all logging functionality is done through that manager.
  • A LogManager manages a collection of zero or more Loggers
  • A Logger has a collection of zero or more LogFilters and a collection of one or more LogTargets
  • A triggered logmessage(Event class) is sent to the LogManager which sends it to each Logger; if a message passes the filters for a Logger, it triggers the targets for that logger

What are the goals of the logging system?

  • Provide a convenient syntax for adding both simple and complex logging messages

  • Support log messages with useful information like:
    • a dotted id path, to aid in filtering messages (e.g. “authentification.openid.yahoo”)
    • numerical severity level, to aid in filtering messages and deciding who to email them to (e.g. from -100 to +100)
    • a short message “type”, again to aid in filtering (e.g. “error” | “warning”)
    • extras dictionary of arbitrary data to be serialized/stringified
  • Allow a collection of loggers to be configured that process log messages in different ways. Each Logger may:
    • Filter on which log messages it cares about based on various patterns

    • Decide how to handle the log message, with targets that include:
      • Emailing log messages
      • Saving log messages to text files (rotating or otherwise(
      • Storing log messages in database tables
  • Additionally we would like to support things like queuing log messages for deferred handling (useful if we want to log to database but database is not set up yet).

  • Smart throttling if we are being overwhelmed with log messages

  • Smart digesting, if we have a bunch of messages to email we might want to send onlyh one email with all log messages for a given session

Some examples of things we will want to be able to easily do:
  • on severe errors trigger an email to admin
  • log “debug” messages to file only
  • turn off certain log messages with minimal cpu impact
  • discard warning messages when running in production mode
  • log messages of type x|y|z to database tables x, y, z
  • throttle log messages if they start to overwhelm the system
class mewlo.mpacks.core.eventlog.mlogger.MewloLogFilter[source]

Bases: object

MewloLogFilter - class responsible for deciding if a log message should be handled by a Logger; each logger has a list of zero or more LogFilters to check for match

add_andfilter(filter)[source]

Add a chained filter which is treated like an AND.

doesmatch_andchains(logmessage)[source]

Check if logmessages matches any attached “AND” chained filters (we may have none).

doesmatch_full(logmessage)[source]

Check if the logmessage matches our filter (or ALL of them if there are multiple chained with us.

doesmatch_us(logmessage)[source]

This is the exposed public function to check if a logmessage matches the filter. It will normally be implemented by a subclass.

dumps(indent=0)[source]

Return a string (with newlines and indents) that displays some debugging useful information about the object.

startup_prep(stageid, eventlist, mewlosite)[source]

This is invoked by site strtup, for each stage specified in startup_stages_needed() above.

class mewlo.mpacks.core.eventlog.mlogger.MewloLogFormatter(formatstr=None)[source]

Bases: object

MewloLogFormatter - one and only one of these can be used to format a log string before writing it out to some file. A MewloLogFormatter is attached to a MewloLogTarget

format_logmessage_as_string(logmessage)[source]

Return a string which formates the event.

class mewlo.mpacks.core.eventlog.mlogger.MewloLogManager(mewlosite, debugmode)[source]

Bases: mewlo.mpacks.core.manager.manager.MewloManager

MewloLogManager - the main supervisor class that manages a collection of Loggers

add_logger(logger)[source]

Just add a child logger to our collection.

add_pythonlogginghook(pythonlogger, pythonloghandler)[source]

Register a python logging handler, and record it for later removal.

description = 'Log manager supervises all logging operations'
dumps(indent=0)[source]

Return a string (with newlines and indents) that displays some debugging useful information about the object.

hook_pythonlogger(pythonlogger_name, pythonlogger_level=10)[source]

Hook into python logging system to catch python error messages and create events from them for mewlo logging.

process(logmessage)[source]

Process a logmessage(Event), by allowing each of our attached loggers to handle it.

set_debugmode(val)[source]
shutdown()[source]

Shutdown everything, we are about to exit.

startup_prep(stageid, eventlist)[source]

This is invoked by site strtup, for each stage specified in startup_stages_needed() above.

typestr = 'core'
class mewlo.mpacks.core.eventlog.mlogger.MewloLogModifier[source]

Bases: object

MewloLogModifier - runs early and can modify/add stuff to log events before they are sent to a LogTarget.

modify_event(event)[source]

Modify the event in-place.

class mewlo.mpacks.core.eventlog.mlogger.MewloLogTarget(logformatter=None)[source]

Bases: object

MewloLogTarget - targets for loggers; each logger has a list of LogTargets to perform on match

dumps(indent=0)[source]

Return a string (with newlines and indents) that displays some debugging useful information about the object.

flushqueue()[source]
get_isenabled()[source]
get_nicelabel()[source]
get_queuelen()[source]
get_startedup()[source]
get_stopprocessing()[source]
process(logmessage)[source]

Process the target action (write to file, save to database, emailing, etc.). This should be overridden by subclass to do actual work.

process_or_queue(logmessage)[source]

Process a line if we are ready, or queue it if not.

queuelog(logmessage)[source]
readytowrite()[source]

Are we ready to write out stuff?

set_isenabled(flagval)[source]
set_logformatter(logformatter)[source]
set_stopprocessing(val)[source]
shutdown()[source]

Shutdown everything, we are about to exit.

startup_prep(stageid, eventlist, mewlosite)[source]

This is invoked by site strtup, for each stage specified in startup_stages_needed() above.

class mewlo.mpacks.core.eventlog.mlogger.MewloLogger(id)[source]

Bases: object

MewloLogger - responsible for saving/writing a log event to some destination, and matching log events to decide whether to handle them.

add_filter(filter)[source]

Append a filter. Multiple appended filters are treated like OR conditions (you can simulate AND by chaining filters together.

add_target(target)[source]

Append a target. Targets will be run when the filters match. Multiple appended targets will be run in sequence.

doesmatch_filters(logmessage)[source]

Return True if this message matches ANY of the filter(s) for the logger..

dumps(indent=0)[source]

Return a string (with newlines and indents) that displays some debugging useful information about the object.

get_id()[source]
get_stopprocessing()[source]
process(logmessage)[source]

Process a logmessage(Event). This may involve ignoring it if it doesn’t match our filters, or sending it to Targets immediately if it does.

run_targets(logmessage)[source]

Run ALL registered targets on the message.

set_stopprocessing(val)[source]
shutdown()[source]

Shutdown everything, we are about to exit.

startup_prep(stageid, eventlist, mewlosite)[source]

This is invoked by site strtup, for each stage specified in startup_stages_needed() above.

class mewlo.mpacks.core.eventlog.mlogger.PythonLogHandler(logmanager, label)[source]

Bases: logging.Handler

A custom log handler we register with python log handling system, to route python log messages to our log manager.

emit(record)[source]

Table Of Contents

This Page