Python Logging

The only "problem" with python's logging module is that it is so configurable and has so many features that it is easy to lose sight of the forest for the trees. This is my standard method and opinion of how to best work with it. No detailed explanations forthcoming. All the "stab yourself in the eye with a fork" details and alternatives are available in the python docs.

My basic criteria for logging is:

  1. Logs need to write to a file.
  2. Logs need to optionally write to stdout.
  3. Log files need to truncate themselves (i.e. not grow to infinity).
  4. The configuration needs to be external to the application.
  5. The configuration needs to be easy to change and expand.

Fortunately the standard libary logging module handles all of these requirements.

Use The INI File Configuration Option

Using an ini file for logging configuration is the most straightforward in my opinion. It is external to your application which means it can be edited/changed outside the life cycle of your code.

Use the RotatingFileHandler

There is a basic logging FileHandler, but it will add log messages until the cows come home and if no one is watching you can end up with massive log files. Just use a RotatingFileHandler. You can set the limit on how much logging data you ever want on your system. The TimedRotatingFileHandler is similar and worth using if it makes more sense to truncate logs by time periods rather than size.

A Logging Configuration Example

In Your Python Code

Only configure your logging once in your application's main script.

import logging
import logging.config

# all the configuration needed
logging.config.fileConfig('logging.conf')

# get a logger
root_logger = logging.getLogger()  # this will be root
app_logger = logging.getLogger('application')

# make your logging calls
root_logger.debug('root logger to DEBUG called')
root_logger.info('root logger to INFO called')
root_logger.warn('root logger to WARN called')
root_logger.error('root logger to ERROR called')

# Note that this call to debug won't propagate because
# we set the log leve to INFO
app_logger.debug('app logger to DEBUG not logged')
# remaining calls show up
app_logger.info('app logger to INFO will show up')
app_logger.warn('app logger to WARN will show up')
app_logger.error('app logger to ERROR will show up')

You can download this sample here.

Using logging elsewhere in your app just requires:

import logging

logger = logging.getLogger('application')

logger.debug('logging in some other part of your app.')

The Configuration

Here's an example of a basic logging configuration file. I've set this up to handle the root logger as well as a logger named appliation. For simple scripts you may only need the root logger. Larger applications may use external libraries with their own named loggers (e.g. sqlalchemy) that you may or may not want to control from your configuration.

[loggers]
keys=root,application

[formatters]
keys=precise

[handlers]
keys=console,logfile

[logger_root]
level=DEBUG
handlers=console,logfile

[logger_application]
level=INFO
propagate=0
qualname=application
handlers=console,logfile

[formatter_precise]
format=[%(asctime)s][%(levelname)s][%(name)s] %(message)s
datefmt=%Y-%m-%d %H:%M:%S
class=logging.Formatter

[handler_console]
level=DEBUG
class=StreamHandler
formatter=precise
args=(sys.stdout,)

[handler_logfile]
level=DEBUG
class=handlers.RotatingFileHandler
formatter=precise
args=('logging_example.log', 'a', 1000, 2)

You can download this sample here.

Resources

  • Python's logging documentation.
  • If you need to sort out how your logging handlers are doing things, then check out logging_tree.

Tags: python

comments powered by Disqus