Ouch! That hurts! Thurdsay (yesterday) I finished setting up new DB instances for XLsuite. All was well, and the servers performed admirably well for 24 hours. Then, suddenly, the LVM partition that held /var/lib/mysql went away…

Around 18:08 UTC (11:08:09 PDT), our main MySQL database server went down. Luckily, yesterday (Thurdsay), I had just replaced our whole DB infrastructure to have a replicated master/slave setup. It took us 15 minutes to notice that the sites were down, and another 20 minutes to execute a database failover. By 18:50 UTC (11:50 PDT), things were back to normal.

Fran├žois Beausoleil

Unfortunately, I did not have the scripts in place yet to do the database failover, so it was a manual process. It wasn’t too hard, but I did have to remember one thing:


1 Mysql::Error: The MySQL server is running with the —read-only option so it cannot execute this statement: INSERT INTO sessions (`updated_at`, `sessid`, `data`) VALUES

In High Performance MySQL, the authors recommend, in Chapter 8:

On the slave, we recommend enabling the following configuration options:

skip_slave_start
read_only

Double oops! Anyway, the advice in the book was invaluable to me. If you manage MySQL and don’t already have the book, I highly suggest you buy it.

I’m just setting up Continuous Integration for all of our branches on XLsuite.com. Doing a search for “cruisecontrol.rb” on Google unearthed: Ci For The Web 2.0 Guy Or Gal. I’m running the steps right now on my RimuHosting VPS.

Things are progressing along quite rapidly. I’ll say more when I’m done.

Search

Your Host

A picture of me

I am François Beausoleil, a Ruby on Rails and Scala developer. During the day, I work on Seevibes, a platform to measure social interactions related to TV shows. At night, I am interested many things. Read my biography.

Top Tags

Books I read and recommend

Links

Projects I work on

Projects I worked on