Uwsgi & Sqlalchemy Strangeness

A couple of weeks back I migrated our ancient Python 2.5 / Apache2 / Modwsgi deployment setup of mycontrol.aero to Python 2.7 / Nginx / Uwsgi. As I failed to test the setup with a reasonable amount of load before switching it has been a bit of a pain since.

Sqlalchemy ResourceClosedError with Uwsgi

At first the switch lead to the weirdest ResourceClosedError sqlalchemy exceptions. Different code paths would throw this error seemingly at random.

In what was quite the rookie mistake I switched everything on Friday afternoon and ended up working the Uwsgi config Friday night.

Through changing, restarting and waiting I got the config somewhat stable by setting the following values in our uwsgi.ini:

single-interpreter = true
processes = 1

There were still some errors following that change which I couldn't explain, but they didn't require me going into frantic mode.

Moar Processes

As I was finally writing a Puppet config for this particular setup I got the chance today to really test our setup and get to the bottom of our recent problems. After all, I don't really care for limiting myself to a single process for our webapp.

I was able to recreate the errors on my local Virtualbox. I don't know how I missed it the last time, but this time my google-fu lead me to the answer:

http://lists.unbit.it/pipermail/uwsgi/2011-May/002079.html

Basically, it looks like Sqlalchemy has some problems with the way Uwsgi handles threads and processes. What needs to be done is this:

lazy = true

Which ensures that each process really is independent of the other.

So I end up with this config:

processes = 4
lazy = true

And now all my problems are gone. I'm still using the single-interpreter mode of Uwsgi, but I'm pretty sure that it doesn't make a difference at this point (expect that it uses a tad more memory).

So I hope the next time someone googles Uwsgi ResourceClosedError this post can be of help.