some good news :)

Hi all,

Freesound has been struggling to regain stability over the last few weeks but we are happy to announce that we finally made it:) The problems started about a month ago when we realized our DB configuration was completely wrong for our hardware and the kind of traffic we are serving. After fiddling around with it we settled for a setup that seemed to work OK. Even so, problems didn’t stop with the sounds page 404ing at times, resulting in the whole site 404ing. We’ve finally pinned this down to some horribly constructed queries our Django ORM was doing for us…

After we rewrote those queries the load average of the DB server was reduced dramatically as you can see from this beautiful Munin graph which shows the load average before (pre 18:00) and after (post 18:00) we deployed the version with the fixed queries :)

DB load average

 

Essentially these changes made the site more stable but our problems were far from over. Load average on our webserver machine was off the roof, making impossible for any sane sysadmin to sleep… While checking web servers logs we found repeated error of broken connections… This time the culprit seemed to be the flup module that Django uses by default to serve fastcgi. All efforts to isolate this problem proved fruitless and we finally decided to ditch flup and move to Gunicorn which in general is a good idea concerning how things are moving in the Python world. After a period of testing we deployed to production the 3rd of November and are happy to announce that the results are really good! Have a look at the graph below. Load has dropped from peaking at 11! to peaking at 4! and being most of the time at around 1. Which is expected and acceptable for the limited hardware we are running on.

So sorry for the long technical post but we really felt we had to give an explanation for the poor service you’ve seen lately. With this stability we’re back at working on application bugs and even better new features :)

Thanks

Stelios

This entry was posted in Uncategorized. Bookmark the permalink.

5 Responses to some good news :)

  1. Bmaccready says:

    Thanks guys! I was worried.

  2. Peter says:

    Fun and games…. seems good now. :)

  3. Star* says:

    WooHoo Good Job!

  4. Tom Swirly says:

    Thanks, Stelios! Your site is a marvel. You should put the contribution link in the sidebar of this blog to remind people who’re getting this fine service for free…!

  5. John says:

    Glad to find this site at #3 on the google for environmental sounds. Lemme see: After joining the NoButts smoking cessation program and winning at that endeavor; I add to my free time a new interest in meditation and ambient sound, the Halotea sound application creator, joining the freesound community and it has downloadable sounds, a board with a “dare the community” and requests sections, a not too geeky administration, and giggles. It makes sense to be here. I am a happy camper. Good job identifying and correcting the problem, Stelios.