You could turn off the check with -o "StrictHostKeyChecking no" on the command line or in your ~/.ssh/config file, but the warning ssh gives is a good one and not one you really want to turn off.  My approach has been to edit the ~/.ssh/known_hosts file and remove the old key.  SSH is nice enough to tell you the exact line number.  It's inelegant, but it's what I've done for years.

What I didn't realize is that ssh has a better solution.

ssh-keygen -R will update the known_hosts file, removing the old key.  The next connection will prompt you to verify and add the new key.  Much nicer.

OSGi does a really good job of what it does.  It does work.  You can break up your app into very independent modular components and manage them.  It's not a half-baked solution, and given the complexities of this kind of class loading, that says a lot.  There are still gotcha's with things like the context class loader. (OSGi ignores it)

OSGi is a very good fit for servers and services.  If you truly have lots of mostly-independent, dynamic services that need to work together, you will like what OSGi gives you.  It can be used  for webapps, but it is not a great fit.  After working with it, the developers I was working with wanted to move the web server and WAR files out of the OSGi environment.

OSGi does not feel like modern Java.  OSGi was designed before the Java 5 revolution.  The APIs are cumbersome and feel outdated.

You will want to use a some sort of framework with it.   We used the Spring OSGi integration.  I didn't enjoy that, but it seems like the best choice when keeping in mind the big picture.

Working with your own code is wonderful.  Working with other people's code is not so easy.  If someone isn't managing an OSGi bundle for the libraries you want to use, you will feel pain.  Even if they are, you may still have headaches.

Summary: OSGi works, but don't use it unless you really need it.

Of all the things I've encountered so far, the thing that's caused me the most headache has been a dumb monkey patch made by the M2Crypto library.  M2Crypto is a replacement SSL library required by own of our tools.  It apparently provides some functionality not in the standard python SSL code.  That's fine, except that in order to provide it's functionality, M2Crypto does this brilliant patch of urllib:

This apparently works for most urllib use, the the urllib.urlretrieve, which we were using, fails when retrieving https URLs.  It was a bear to track down, but once I realized that the function worked fine in a bare python and failed in the virtualenv python with all our third party libraries, it just took some time single-stepping through in the debugger to figure out where the two environments differed.

Since the open_https method is completely destroyed my the patch, the only option was to switch to urllib2.  However, since urllib2 doesn't seem to have kept the urlretrieve function around, I had to rewrite the urlretrieve functionality on top of it.  

This is exactly the kind of thing I was most afraid of when it came to working with Python - having to deal with other developers who decided it would be great to change the way base system libraries work.  Looking at the big picture, it was really only a minor annoyance - a few lost hours at most.  The real harm is in the feeling of instability it brings.  I want to know exactly what the system is doing and why, and it's hard to do when one library decides to muck around with another libraries internals.

The workaround I found, after digging through the code a bit was to mimic the behavior of AutoField to trick InlineModelAdmin into doing the right thing.  I extended UUIDField like this:

After a bit more searching, I later found this was a known bug.  More interestingly, one of the commenters on the bug had considered the same solution and rejected it.  Take a look at the bug report for his reasoning, but the error condition he points out only occurs when the default value is None.  If the field has a default value, as is the case with this CharField-extended model, the potentially problematic logic doesn't kick in.  I haven't battle-tested the solution yet, but initial testing looks like it performs correctly. 

As a developer, my first question is why I should care open OpenStack.  I'm intrigued, for sure, and my background should leave no doubt that I'm a huge open source proponent.  However, I don't ever expect to run my own cloud.  As a developer, if I run on Amazon EC2, I see a set of Linux boxes (open source) and a public API I can use to manage my instances.   Whether the cloud infrastructure is open source or not, doesn't seem to matter much.  I should be able to spin up a Linux box on the Rackspace cloud just as easily and have nearly the same view of the world.  

The management of instances seems to be the key difference.  Services like Rightscale go a long way to bridging the gaps between the different cloud offerings, but I can certainly see how this would all be easier if the various cloud services had a common management API.  Unfortunately, I don't see much talk about management APIs from OpenStack.  Maybe I've missed it, or maybe it's just too early.  It's not really clear. What exactly is the OpenStack view of management APIs?

More important than the infrastructure APIs is the platform service APIs.  I'm a huge fan of S3 and SQS, for example, both of which are big reasons I go with EC2.  Though technically both are quite accessible outside of EC2, I would only seriously consider moving if equivalent services were available, preferably ones with standard APIs that would give me true cloud service portability.  OpenStack has the start of this with Swift, their version of S3.  However, I'm still a little unclear on the big picture.  Will OpenStack create a core platform of AWS-like cloud services based around standard APIs?  Or, is the goal to create a linux-like environment for the creation of (hopefully compatible) platform packages that can be installed into an OpenStack cloud ad hoc?  It seems to me like the later is closer to the approach, and although that's quite an interesting possibility, I'm not sure how OpenStack then will lead to a simpler, more standardized cloud than we have now.

I'm not down on OpenStack.  I just don't really understand the project.   As an app developer, I'm clearly not the primary audience of OpenStack, yet.  However, cloud platforms exist to serve the developers, and if you are designing an open platform from the ground up, I would hope to have been able to get a little bit better of a perspective on what the world of an OpenStack cloud will look like to a developer.

From my own experience, I find that it takes about 2 years to complete a full cycle on a project.  At first, you come in and are just learning the domain and the code base.  Then there's period of developing fluency until you've completely mastered the codebase.  At that point, you'll be in a position to be able to contribute to project at an architectural level, leaving plenty of time to hang around and learn the lessons from these larger-scale efforts.  

Sometimes this cycle is faster, sometimes it's slower, but a couple years seems to be about right.  After that, something needs to change.  In a larger company, there may be opportunities on teams that are very different from what you are working on.  I've always worked for smaller companies, and often times this change comes in the form of an acquisition, which brings new challenges.  At one company, I transfered to the advanced research division of the acquiring organization.  In another, I tried my hand at evangelism/product management, making a transition back into engineering when I felt I really needed to spend more time coding.  In both of these cases, I ended up staying for 4-5 years.

I'm curious how this compares to other people's experience.  

Despite the fact that I've actually increased the number of technical books I've been reading, I've gotten out of the habit of reviewing them.  I thought I should try and reverse that trend, starting with one of the most helpful books I've read this year, SQL Antipatterns.

I've never been a fan of relational databases.  I started my career working with LDAP, and acceptance of the rigidness relational model didn't come easily.  Over time, I learned to respect relational databases more and become moderately proficient with them.  However, my policy was always that the database is nothing more than an implementation detail.  If I ever actually noticed the database when you were writing code, something was seriously wrong with my dev model.

I won't debate the merits of that attitude, but it led me down a path of accepting many half-baked solutions that ended up causing my nothing but pain, but fortunately I was able to sweep those under the rug of my db abstractions and ORM mapping tools, directing the curses under my breath at that evil SQL database that I had to constantly interrupt my "real" dev work to deal with.

I saw many of those "solutions" in the SQL Antipatterns book, along with a discussion of the problems with each pattern and a pragmatic discussion of the alternatives.  (sometimes the antipattern might actually be better than the alternatives)  I won't discuss the specific antipatterns here.  I'll just repeat that I saw many of my previous design decisions in here along with alternatives I wish I had considered.   If you find relational databases a pain, some of that pain might be caused by the way you are doing things.  SQL Antipatterns may just set you right.   Remember, hating relational databases is not an excuse for mis-using them.   And who knows, when you start using them better you might even find they aren't nearly as ugly as you thought.

After teaching so many training classes back in the JBoss days (until Red Hat took over I was doing at least 1 class a month), being on the other side of the tables was a bit surreal.  At several points during the lab times, I found myself starting to stand up to go check on how everyone and needing to remind myself that I wasn't in charge of the show.   

The Riptano class is a single day class that is split between basic usage and admin/ops topics.  As long as a day is, it's really hard to get from 0 to anywhere interesting in a single day.  If I were redesigning the Riptano class, I would suggest that they record the morning introduction to Riptano concepts as a video and make that a pre-requisite for attending the class.  You'd definitely have some people who show up without doing their homework, but I think you have to target the people who are there to learn first.  Getting straight into Cassandra would have really let us get a lot deeper.

After the introduction to cassandra, we got started directly on some basic usage.  Riptano smartly provides a vmware virtual machine to standardize the learning environment for students.  (I really wish I had the luxury of doing that back when I was leading JBoss training classes.  The lessons and labs largely focussed on using cassandra-clii, the Cassandra shell.  The shell is very limited.  You can only do the most basic operations with it, but anything more would require writing code.  That's not an easy task for a platform for a datastore that aims to serve clients in a variety of languages, each with it's own different API abstractions.    Having a fully-functional query language should definitely be high on the list of future Cassandra improvements I'd like to see.  

The labs were helpful, but we only had time to explore the data model of the twissandra demo app.  It was enough to get a taste of how to navigate related column families.  With Cassandra you have to maintain your own indexing (No join's for you!) and carefully design your model around your access model.  It would have been very enlightening to try and design a usable data model, but that was clearly beyond the scope of the class. 

The rest of the class focussed largely on ops issues.  We played with different replication factors with a multi-node cluster, learning how Cassandra balances data between nodes and how it deals with nodes entering and leaving.  This was really practical information.  As I've only prototyped Cassandra code, I can speak to how well the class prepares you for deploying Cassandra in production, but it was very helpful to me.   The slow part of the afternoon was the hour or so that we spent talking about the JMX MBeans and Java memory management.  (Cassandra is written in Java)  I assume this was helpful to those in the class without a Java background, but I found myself nodding off a bit here. 

The conclusion?  I think the class is a good value for the money, especially since the trainer was an actual Cassandra developer who could answer technical questions without resorting to hand waving.  I would recommend it to anyone new to Cassandra.  

One of my dev tasks earlier this year was to write a system that asynchronously archived user data by email.  I'm no stranger to SMTP, but I'm not exactly a mail guru either.  When I wrote the code, I knew we were going to have integration issues.  In a moment of paranoia I decided that instead of just logging the work being done and and the important result codes that I would log every outgoing SMTP session in a special log directory, indexed by time and and event id to make sure I could quickly answer any questions about a transaction.   Dealing with these logs is a bit of a hassle.  I periodically tar them up and ship them out to S3, but as I watch the data grow I've often wondered if maybe I weren't just overdoing it a bit.

The logs have come in handy from time to time, but over the last couple weeks having them around has saved me in some really big ways these last few weeks.  I've had three big issues these last few weeks, ranging from authentication, to mime-type issues to "are we sure we are actually sending anything at all" problems.  In each case, the problem has been in the receiving party's side.  With the extensive logging, it's been trivial to dig up the exact bits we sent out in seconds, while we wait and wait (often for days) for the other guys to figure out what is going on in their own system.

This all has made me wonder if perhaps I'm not being paranoid enough.  There are a lot of integration points in the code I'm working on that really has very poor log coverage.  If a problem did crop up, I wouldn't be able to directly answer a question about what exactly the system did or didn't do, and until now I probably wouldn't have thought much of it.  That's a pretty scary thought, and it's making me rethink my whole approach to logging.

What go me going was, after going through a number of "this might possibly be interesting troubleshooting steps if the lights on the cable modem were on" checks on my computer, the tech support person asked:

Have you optimized your web browser?

Huh?  I paused for a while lost in thought, trying to figure out what that means.  Optimized my web browser?  Is he asking if I've installed some Time Warner browser plugin to serve ads to me?  Have I updated the latest version of my browser?  What in the world could that mean?  (feel free to pause for a second and take a guess)

It turns out the Time Warner agent was asking trying to ask if I had cleared my browser cache and cookies. Silly me - I thought a cache was supposed to be a performance optimization by reducing the need for reloading pages over the network.  Of course, in some cases that could be a useful troubleshooting step, but would it be possible to actually say that rather than make up nonsense term?

It's funny, but my real dilemma was trying to figure out how to respond.  Should I just hold back and politely say yes?  Should I come up with a snarky response like "No, do I need to charge my flux capacitor first?" I ended up going halfway with the simple "I'm not sure that means anything", but I can't help but feel I missed the chance to give a truly quality response.