Saved by the log

I have a tendency to obsess a bit about logging.  Sometimes I'm really comfortable with my code that I let my laziness override my judgement, but I generally try to log my code well enough that I can go back and answer just about any question I want to about a system.

One of my dev tasks earlier this year was to write a system that asynchronously archived user data by email.  I'm no stranger to SMTP, but I'm not exactly a mail guru either.  When I wrote the code, I knew we were going to have integration issues.  In a moment of paranoia I decided that instead of just logging the work being done and and the important result codes that I would log every outgoing SMTP session in a special log directory, indexed by time and and event id to make sure I could quickly answer any questions about a transaction.   Dealing with these logs is a bit of a hassle.  I periodically tar them up and ship them out to S3, but as I watch the data grow I've often wondered if maybe I weren't just overdoing it a bit.

The logs have come in handy from time to time, but over the last couple weeks having them around has saved me in some really big ways these last few weeks.  I've had three big issues these last few weeks, ranging from authentication, to mime-type issues to "are we sure we are actually sending anything at all" problems.  In each case, the problem has been in the receiving party's side.  With the extensive logging, it's been trivial to dig up the exact bits we sent out in seconds, while we wait and wait (often for days) for the other guys to figure out what is going on in their own system.

This all has made me wonder if perhaps I'm not being paranoid enough.  There are a lot of integration points in the code I'm working on that really has very poor log coverage.  If a problem did crop up, I wouldn't be able to directly answer a question about what exactly the system did or didn't do, and until now I probably wouldn't have thought much of it.  That's a pretty scary thought, and it's making me rethink my whole approach to logging.