Python error logging

I'd like to find a way to log every error that forces the python interpreter to quit to be saved to a file as well as being printed to the screen. The reason I would like to do this is that I want to keep stats on the types of errors I make while writing code, with an eye towards finding ways to avoid mistakes I make commonly in the future.

I've been attempting to do this by writing a wrapper for the python interpreter using the subprocess module. Basically, it runs the python interpreter, captures any output, parse and saves it to a file, prints the output, and use matplotlib to make some summary figures. However, I'm having a problem getting output from my wrapper script in real time. For example, if the script I'm running is:

import os  
import time

for x in range(10):  
    print "testing"  
    time.sleep(10)  

and I'm using subprocess.Popen() with p.communicate(), the wrapper will wait 100 seconds, and then print all of the output. I'd like the wrapper to be as invisible as possible - ideally in this case it would print "testing" once every ten seconds.

If someone could point me towards a good way of doing this, I'd greatly appreciate it.

Thanks!

I believe you can simply replace sys.excepthook with your own function. You can read about it in the Python documentation.

Basically, it allows you to customize what happens when an exception percolates up to the point of forcing the Python interpreter to quit. You use it like this:

import sys

def my_excepthook(type, value, tb):
    # you can log the exception to a file here
    print 'In My Exception Handler'

    # the following line does the default (prints it to err)
    sys.__excepthook__(type, value, tb)

sys.excepthook = my_excepthook

You'll probably also want to look at the traceback module, for formatting the traceback you get.

lots of log4net advocates here so i'm sure this will be ignored, but i'll add my own preference:

system.diagnostics.trace

this includes listeners that listen for your trace() methods, and then write to a log file/output window/event log, ones in the framework that are included are defaulttracelistener, textwritertracelistener and the eventlogtracelistener. it allows you to specify levels (warning,error,info) and categories.

trace class on msdn
writing to the event log in a web application
udptracelistener - write log4net compatible xml messages to a log viewer such as log2console

edit - see bottom of post for more up-to-date, code

figured it out with thanks to ned deily pointing out that smtplib (which sits under smtphandler) requires special treatment. i also found this post demonstrating how to do that, by overloading the smtphandler (in that case to fix a tls problem).

using smtplib.smtp_ssl (see smtplib docs), rather than the straightforward smtplib.smtp, i was able to get the whole system working. this is the utils/logs.py file i use to set up the handlers (which should be a nice example of file, as well as email, handlers):

from your.application.file import app

import smtplib
import logging
from logging.handlers import rotatingfilehandler, smtphandler


# provide a class to allow ssl (not tls) connection for mail handlers by overloading the emit() method
class sslsmtphandler(smtphandler):
    def emit(self, record):
        """
        emit a record.
        """
        try:
            port = self.mailport
            if not port:
                port = smtplib.smtp_port
            smtp = smtplib.smtp_ssl(self.mailhost, port)
            msg = self.format(record)
            if self.username:
                smtp.login(self.username, self.password)
            smtp.sendmail(self.fromaddr, self.toaddrs, msg)
            smtp.quit()
        except (keyboardinterrupt, systemexit):
            raise
        except:
            self.handleerror(record)


# create file handler for error/warning/info/debug logs
file_handler = rotatingfilehandler('logs/app.log', maxbytes=1*1024*1024, backupcount=100)

# apply format to the log messages
formatter = logging.formatter("[%(asctime)s] |  %(levelname)s | {%(pathname)s:%(lineno)d} | %(message)s")
file_handler.setformatter(formatter)

# set the level according to whether we're debugging or not
if app.debug:
    file_handler.setlevel(logging.debug)
else:
    file_handler.setlevel(logging.warn)

# create equivalent mail handler
mail_handler = sslsmtphandler(mailhost=(app.config['mail_server'], app.config['mail_port']),
                           fromaddr=app.config['mail_from_email'],
                           toaddrs='[email protected]',
                           subject='your app died. sad times...',
                           credentials=(app.config['mail_username'], app.config['mail_password']))

# set the email format
mail_handler.setformatter(logging.formatter('''
message type:       %(levelname)s
location:           %(pathname)s:%(lineno)d
module:             %(module)s
function:           %(funcname)s
time:               %(asctime)s

message:

%(message)s
'''))

# only email errors, not warnings
mail_handler.setlevel(logging.error)

this is registered in my application file with:

# register the handlers against all the loggers we have in play
# this is done after app configuration and sqlalchemy initialisation, 
# drop the sqlalchemy if not using - i thought a full example would be helpful.
import logging
from .utils.logs import mail_handler, file_handler
loggers = [app.logger, logging.getlogger('sqlalchemy'), logging.getlogger('werkzeug')]
for logger in loggers:
    logger.addhandler(file_handler)
    # note - i added a boolean configuration parameter, mail_on_error, 
    # to allow direct control over whether to email on errors. 
    # you may wish to use 'if not app.debug' instead.
    if app.config['mail_on_error']:
        logger.addhandler(mail_handler)

edit:

commenter @edugord has had trouble emitting the record correctly. digging deeper, the base smtphandler class is sending messages differently than it was 3+ years ago.

this updated emit() method should get the message to format correctly:

from email.message import emailmessage
import email.utils
class sslsmtphandler(smtphandler):

    def emit(self, record):
        """
        emit a record.
        """
        try:
            port = self.mailport
            if not port:
                port = smtplib.smtp_port
            smtp = smtplib.smtp_ssl(self.mailhost, port)
            msg = emailmessage()
            msg['from'] = self.fromaddr
            msg['to'] = ','.join(self.toaddrs)
            msg['subject'] = self.getsubject(record)
            msg['date'] = email.utils.localtime()
            msg.set_content(self.format(record))
            if self.username:
                smtp.login(self.username, self.password)
            smtp.send_message(msg, self.fromaddr, self.toaddrs)
            smtp.quit()
        except (keyboardinterrupt, systemexit):
            raise
        except:
            self.handleerror(record)

hope this helps somebody!

tl;dr there's nothing wrong with your code

it seems you've followed the linked tutorial correctly, and would probably find your log files in the /home/junsu/sites/superlists-staging.mysite.com/ dir.

regardless, there are a few points to address in your question, i'll try to do that.

loggers and handlers

the settings module you reference above sets up a single logging handler console (streamhandler), and a single django logger which can use that handler.

the root logger does not define any handlers, and "django" will log anything to stderr, and only for level info and above. i ran a quick test, and the root logger also has a streamhandler defined by default.

your authentication.py module currently calls logging.warning which logs to root logger (i.e it does logger = logging.getlogger(); logger.warning('stuff')). however, you may want to define a more specific handler to easier locate the log of your module. this is explained in this section of the referenced tutorial.

gunicorn redirects stderr by default

it seems by default is set up to capture the stderr stream, which you currently redirect to a log file. however, my suggestion is to use your daemonizing app (seems like you're using upstart) to log the stderr/out.

upstart logging

as explained in gunicorn docs, configuring upstart is pretty simple.

if you remove the --error-logfile option in your /etc/init/gunicorn-superlists-staging.mysite.com.conf config, gunicorn will default to logging it's output to stderr which can then be captured by upstart in whatever manner you prefer.

if you are using upstart 1.7 or greater, stdout/err capturing should be enabled by default. if, however, you use an earlier version of upstart, my suggestion is to add a console log option in your config and all output (stdout/stderr) will simply be logged to (i assume) /var/log/upstart/gunicorn-superlists-staging.mysite.com.log


Tags: Python Error Logging