A Single Programmer's Blog

I just pushed a change for my teammates: we’re now using Bundler on our Rails 2.3.5 application. Things went rather well, but our continuous testing server choked a bit on that.

First, I had forgotten to run bundle install after pulling changes. Then I had to ignore our cucumber environment, because it depends on rb-appscript which is a native gem on Mac OS X. Our CT machine is a Linux box, hence the change. Then, bundle install would fail installation because it didn’t have sudo access:

 1 Fetching git://github.com/bloom/rails.git
 2 Fetching git://github.com/francois/active_url
 3 Fetching git://github.com/francois/safariwatir.git
 4 Fetching source index for http://rubygems.org/
 5 sudo: no tty present and no askpass program specified
 6 sudo: no tty present and no askpass program specified
 7 Using rake (0.8.7) 
 8 sudo: no tty present and no askpass program specified
 9 sudo: no tty present and no askpass program specified
10 /usr/lib64/ruby/site_ruby/1.8/rubygems/format.rb:38:in `from_file_by_path': Cannot load gem at [/usr/lib64/ruby/gems/1.8/cache/SystemTimer-1.2.gem] in /home/francois/adgear-admin (Gem::Exception)
11   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/source.rb:77:in `fetch'
12   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/installer.rb:45:in `run'
13   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/spec_set.rb:12:in `each'
14   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/spec_set.rb:12:in `each'
15   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/installer.rb:44:in `run'
16   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/installer.rb:8:in `install'
17   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/cli.rb:185:in `install'
18   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/vendor/thor/task.rb:22:in `send'
19   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/vendor/thor/task.rb:22:in `run'
20   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
21   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/vendor/thor.rb:246:in `dispatch'
22   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/vendor/thor/base.rb:389:in `start'
23   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/bin/bundle:13
24   from /usr/bin/bundle:19:in `load'
25   from /usr/bin/bundle:19
26 

I was busy writing a message to the Ruby Bundler Google Group when I remembered bundle install has an optional parameter which specifies where the bundle should be installed. I changed my CT script to run bundle install --without=cucumber vendor/gems and bundle did it’s magic, and my CT box started running tests.

Much rejoicing ensued.

For the record, here’s our testing script:

 1 #!/bin/bash
 2 cd /home/francois/adgear-admin
 3 
 4 if [ -z $1 ] ; then
 5         echo "Can't proceed if I don't know which branch to test"
 6         exit 1
 7 fi
 8 
 9 TSTAMP=`date +%Y-%m-%d-%H:%M`
10 BRANCH=$1
11 LOG=/home/francois/build-logs/$BRANCH-$TSTAMP.log
12 
13 (
14         git fetch origin                                           && \
15         git reset --hard origin/$BRANCH                            && \
16         echo =======================                               && \
17         git show                                                   && \
18         ( bundle install --without=cucumber vendor/cache || true ) && \
19         nice rake db:adgear:production:update parallel:prepare parallel:test
20 ) > $LOG 2>&1
21 
22 if [ "$?" = "0" ] ; then
23         /usr/bin/curl -s -o /dev/null -d "{\"message\":{\"body\":\"Test Suite Passed -- $BRANCH -- http://ct.local:9001/logs/$BRANCH/$TSTAMP\"}}" -HContent-Type:application/json -u 12345:X http://oursite.campfirenow.com/room/12345/speak.json
24 else
25         /usr/bin/curl -s -o /dev/null -d "{\"message\":{\"body\":\"Test Suite FAILED -- $BRANCH -- http://ct.local:9001/logs/$BRANCH/$TSTAMP\"}}" -HContent-Type:application/json -u 12345:X http://oursite.campfirenow.com/room/12345/speak.json
26 fi
27 
28 # vi: sw=8 ts=8 expandtab

This script is run from cron.

To try and ease my pain, I am refactoring a Rails 2.3.5 application using Hobo from multiple tables to a single table. I’m talking about models which are essentially the same: companies, people and employees. I went from this:

 1 class Company < ActiveRecord::Base
 2   has_many :employees
 3   has_many :people, :through => :employees
 4 end
 5 
 6 class Person < ActiveRecord::Base
 7   has_many :employees
 8   has_many :companies, :through => :employees
 9 end
10 
11 class Employee < ActiveRecord::Base
12   belongs_to :company
13   belongs_to :person
14 end

To an STI-enabled class hierarchy:

 1 class Addressee < ActiveRecord::Base
 2   self.abstract_class = true
 3 end
 4 
 5 class Company < Addressee
 6   has_many :employees
 7   has_many :people, :through => :employees
 8 end
 9 
10 class Person < Addressee
11   has_many :employees
12   has_many :companies, :through => :employees
13 end
14 
15 class Employee < Addressee
16   belongs_to :company
17   belongs_to :person
18 end

Can you spot the mistake? Here’s where it breaks down:

1 $ script/console
2 > PersonAddressee.count
3 SQL (9.9ms)   SELECT count(*) AS count_all FROM "addressees"
4 => 10231

When the classes are defined with abstract_class? set to true, calling Company#count executes the following:

1. ActiveRecord Company#superclass, which answers Addressee
2. ActiveRecord calls Addressee#abstract_class?, which answers true
3. Because Company#base_class == Company#superclass, no STI is involved, and ActiveRecord queries the addressees table with no STI hints (type column).

When removing the abstract class declaration, here’s how it works:

1. ActiveRecord Company#superclass, which answers Addressee
2. ActiveRecord calls Addressee#abstract_class?, which answers false
3. ActiveRecord Addressee#superclass, which answers ActiveRecord::Base
4. Because Company#base_class != Company#superclass, STI is involved and ActiveRecord adds the type hint to the addressees table query.

The documentation is quite clear though:

abstract_class?()

Returns whether this class is a base AR class. If A is a base class and B descends from A, then B.base_class will return B.

The documentation on #base_class is much better though:

base_class()

Returns the base AR subclass that this class descends from. If A extends AR::Base, A.base_class will return A. If B descends from A through some arbitrarily deep hierarchy, B.base_class will return A.

And current HEAD documentation is the best of them all:

base_class()

Returns the base AR subclass that this class descends from. If A extends AR::Base, A.base_class will return A. If B descends from A through some arbitrarily deep hierarchy, B.base_class will return A.

If B < A and C < B and if A is an abstract_class then both B.base_class and C.base_class would return B as the answer since A is an abstract_class.

Luckily, I found this before users got a chance to report bugs. And no, I would never have thought to add a test for this: I’m not in the habit of testing base ActiveRecord functionality.

Damien Caselli, one of the first brave souls to use Mongo Explorer, reported an issue with time travelling in Mongo Explorer. Luckily, it wasn’t anything serious: just a misunderstanding between me, Cocoa and JavaScript.

To celebrate the No Time Travelling Issues, I’ve released version 1.0.2 of Mongo Explorer. Get yours: mongo-explorer-v1.0.2.zip

Yes, I’m aware of Sparkle, and yes, it will eventually make it into Mongo Explorer. That, and a million other little things.

In the past few months, I was looking for an excuse to write a native Mac Cocoa application. We use MongoDB at Bloom Digital to handle our analytics needs. I had tried mongohub, but my itch wasn’t scratched: I wanted something that displayed my data in tables, not as some kind of outline. Thus was born Mongo Explorer.

Click to enlarge

If you tell me “Why didn’t you fork mongohub and add your code”, I’ll tell you I was intimidated by a larger Cocoa application. I wanted something simple from which I could learn everything I wanted: from the grounds up. Of course, once I’m “done” (for some measure of done-ness), the code base will be much larger.

One useful piece of code that might be useful for others is a Mongo Driver for Objective-C. Underlying Mongo Explorer is an abstraction for Mongo connections that’s similar to the Ruby driver. It’s not yet a framework, since I don’t yet know how to do that, but hey, this is a learning exercise for me. One nice thing about this abstraction is that it’s built directly atop the official Mongo C driver: thus it inherits it’s abilities, but some of it’s quirks as well.

What can you expect from Mongo Explorer today? You can open a connection to any local MongoDB instance (manually open an SSH tunnel if you want a remote instance), browse the server’s databases and collections, and list your collection’s documents. For each document, you’ll get what I call the flat view. The flat view is a dot-separated list of keys, followed by the value. It’s useful to view nested objects:

1 obj = {"rp" => {"a" => 1, "b" => {"c" => "d"}}}
2 # Flat view
3 # rp.a   => 1
4 # rp.b.c => d

The flat view was my biggest impetus to write Mongo Explorer.

What’s broken? First, this needs some kind of icon. Also, don’t expect to browse multi-million document collections just yet. Cocoa wants to fully materialize the NSTableView rows, and I don’t know how to handle that yet. I do have an MEArray class that helps somewhat, but it’s not perfect yet. Learning exercise, remember? I’ve successfully viewed collections with slightly less than 20,000 document with no issues.

If you’re interested in following, you can play around on GitHub, or even download the first official release: Mongo Explorer, v1.0.1

UPDATE: Pushed 1.0.1, as 1.0.0 didn’t include it’s frameworks, and was 10.6 only.

I’m writing a Cocoa application to browse and administer a Mongo DB. This is a fun side-project. I’m using the Mongo driver from Objective-C, hence there might be differences for you.

First off, we need to connect to Mongo, and to do that, we need to tell Mongo about the connection information, namely the host and port:

1 mongo_connection_options opts;
2 strcpy(opts.host,
3        [[self.connectionInfo objectForKey:MEHost] cString]);
4 opts.port = [[self.connectionInfo objectForKey:MEPort] intValue];
5 /* mongo doesn't have a default port constant or define */
6 if (0 == opts.port) opts.port = 27017;

connectionInfo above is an instance of NSDictionary. The available keys are declared in NewConnectionController.

After we’ve prepared the connection options, it’s time to connect:

1 connection = (mongo_connection *)malloc(sizeof(mongo_connection));
2 if (!connection) return;
3 
4 mongo_conn_return result = mongo_connect(connection, &opts);
5 if (mongo_conn_success == result) return;

connection above was declared as a pointer to a mongo_connection structure in DatabaseController. If the connection succeeds, I simply continue. If the connection fails, I fall-through and inform the user using an NSAlert, shown as a sheet.

I haven’t yet needed to authenticate to Mongo, but it seems simple to call mongo_cmd_authenticate().

Don’t forget the tests available in the mongo driver itself.

We received a nasty Hoptoad notification today at AdGear :

1 ActiveRecord::StatementInvalid: PGError: ERROR:
2 duplicate key value violates unique constraint "index_placement_rules_on_type_and_placement_id_and_bookable_typ" :
3 INSERT INTO "placement_rules" ("kind", "created_at", "placement_id", "bookable_type", "bookable_id", "updated_at") VALUES(E'PlacementInclusion', '2010-06-02 19:29:24.479983', 2320, E'Site', 162, '2010-06-02 19:29:24.479983') RETURNING "id"

This manifested itself when a customer called and told us he’d found our 500 error page. Ooops.

Our problem turned out to be a misunderstanding of autosave association validations.

Autosave associations are a wonderful beast: with a single #save call, the nested entities are saved within the same transaction, without any intervention on the developer’s part. Unfortunately, validations behave differently than what we expected.

For the sake of argument, let’s use these models:

 1 class Order < ActiveRecord::Base
 2   has_many :items
 3 end
 4 
 5 class OrderItem < ActiveRecord::Base
 6   belongs_to :order
 7 
 8   validates_presence_of :product_number
 9   validates_uniqueness_of :product_number, :scope => :order
10 end

If you build an Order and add two items, you’ll have troubles, unless you wrap your save in a transaction block:

 1 > o = Order.new
 2 > 2.times { o.items << Item.new(:product_number => "A-113") }
 3 > o.save
 4   # Validates entities
 5   Item Load (0.2ms)   SELECT "items".id FROM "items" WHERE ("items"."product_number" = 'A-113' AND "items".order_id IS NULL) LIMIT 1
 6   Item Load (0.1ms)   SELECT "items".id FROM "items" WHERE ("items"."product_number" = 'A-113' AND "items".order_id IS NULL) LIMIT 1
 7   Order Create (0.7ms)   INSERT INTO "orders" ("customer_number", "created_at", "updated_at") VALUES(NULL, '2010-06-02 19:15:57', '2010-06-02 19:15:57')
 8   # Validate 1st entity again
 9   Item Load (0.1ms)   SELECT "items".id FROM "items" WHERE ("items"."product_number" = 'A-113' AND "items".order_id = 1) LIMIT 1
10   Item Create (0.1ms)   INSERT INTO "items" ("created_at", "order_id", "updated_at", "product_number") VALUES('2010-06-02 19:15:57', 1, '2010-06-02 19:15:57', 'A-113')
11   # Validate 2nd entity again, but fails this time around
12   Item Load (0.1ms)   SELECT "items".id FROM "items" WHERE ("items"."product_number" = 'A-113' AND "items".order_id = 1) LIMIT 1
13  => [false, true] 
14 > o.items.map(&:errors).flatten.map(&:full_messages)
15  => [[], ["Product number has already been taken"]] 

A partially saved order is rarely what you’re looking for. Adding the :autosave => true option…

1 class Order < ActiveRecord::Base
2   has_many :items, :autosave => true
3 end

Generates a very different SQL trace:

 1 > o = Order.new
 2 > 2.times { o.items << Item.new(:product_number => "A-113") }
 3 > o.save
 4   # Validate associated entities...
 5   Item Load (0.2ms)   SELECT "items".id FROM "items" WHERE ("items"."product_number" = 'A-113' AND "items".order_id IS NULL) LIMIT 1
 6   Item Load (0.1ms)   SELECT "items".id FROM "items" WHERE ("items"."product_number" = 'A-113' AND "items".order_id IS NULL) LIMIT 1
 7   Order Create (0.4ms)   INSERT INTO "orders" ("customer_number", "created_at", "updated_at") VALUES(NULL, '2010-06-02 20:03:12', '2010-06-02 20:03:12')
 8   # Then save them, even though they're invalid.
 9   Item Create (0.1ms)   INSERT INTO "items" ("created_at", "order_id", "updated_at", "product_number") VALUES('2010-06-02 20:03:12', 2, '2010-06-02 20:03:12', 'A-113')
10   Item Create (0.1ms)   INSERT INTO "items" ("created_at", "order_id", "updated_at", "product_number") VALUES('2010-06-02 20:03:12', 2, '2010-06-02 20:03:12', 'A-113')
11  => true 

This is even worse: instead of a partial save, ActiveRecord told us that everything was good. This is how ActiveRecord behaves:

1 if order.save then
2   order.items.each(&:save) if order.items.all?(&:valid?)
3 end

What we need instead is to hook into the autosave callback chain:

 1 class Order < ActiveRecord::Base
 2   has_many :items, :autosave => true
 3 
 4   private
 5 
 6   # items is the has_many name. If you had posts, it would be
 7   # validate_associated_records_for_posts. If you have multiple
 8   # autosave associations, each can have a method such as this
 9   # one to handle it's validation needs.
10   def validate_associated_records_for_items
11     product_number_errors = false
12     items.group_by(&:product_number).reject {|_, group| group.length <= 1}.each do |product_number, group|
13       product_number_errors = true
14       errors.add_to_base "Product #{product_number} used #{group.length} times on the order"
15     end
16 
17     errors.add_to_base("Each product number can be used at most once per order") if product_number_errors
18   end
19 end

With this validation in place, validations will run on the full collection in a single pass, rather than piecemeal.

1 > o = Order.new
2 > 2.times { o.items << Item.new(:product_number => "A-113") }
3 > o.save # Will run in-memory validations
4  => false 
5 > pp o.errors.full_messages
6 ["Product A-113 used 2 times on the order",
7  "Each product number can be used at most once per order"]

Notice I didn’t say anything about the validates_uniqueness_of validation in Item. I left it in place. It doesn’t harm anything, and if you ever create items without going through the parent model, your code is ready to take care of itself.

I built an application for a non-profit in my region. It allows them to track donations and statistics about their work. One thing that’s very important for them is to know if / when they called a previous donor, to see if they will give again this year. They also need to emit receipts for those people that donated. And they must not emit the same receipt twice, etc. You know the drill. I have a page in the application where they can build such a query:

On this page, they can choose which campaign they’re working on, and different event types that occur during a campaign. They can also choose how to present the result: the date at which the event occured, or the sum of money or quantity they acquired. It’s a large form, since there are almost 15 event types that can occur in a single campaign.

René, my customer, asked me if they could save queries between sessions, so they can go home in the evening, and come back the next day. I said “Sure, I’ll just need a Search model here, and … Hold on a minute: I have a much simpler solution: bookmark the result page.”

You see, I built this application on REST principles. The query itself is idempotent: it either shows events, or not. And since this is idempotent, a GET query is just what the doctor ordered. And browsers are nice enough to allow bookmarking GET requests. Yup: 10 hours of work just disappeared because I used what’s already available. My customer was happy: he didn’t have to wait a minute to get the functionality he needed. I was happy because I had 10 hours of work NOT to do, and I don’t have to maintain that code: it’s in somebody else’s hands. TSTTCPW

This was a real win-win situation.

I spend 70% to 90% of my day in a full-screen iTerm, in Vim. I find it ironic that I use a 3kCAD machine to do what green screen terminals did 30 years ago… My iTerm’s color scheme is green text on a black background. I read somewhere that this color combination was the easiest on eyes, but can’t find the reference right now.

When I’m not in Vim, I’m usually in Safari or Firefox. What does your day look like?

I’m a happy DreamHost customer, for many years. Recently, they moved my data to a new server. That in itself wasn’t a problem, but this blog went down with their “bad_httpd_conf” error. DreamHost support has always been prompt to answer my questions. But more importantly, when a customer asks for support, they have answer right where it matters.

As part of the support process, DreamHost asks what site you have a problem with. Then they run some diagnostics on the network and report those as warnings.

Click to view full size

After they’ve done that, DreamHost asks for what type of problem you have. And the different categories have tips you can apply to try to correct the problem, before you have to wait for DreamHost support.

Click to view full size

Another thing that’s very nice: after logging in, one of the 8 main options is, you guessed it, Support. DreamHost doesn’t hide their support. It’s very visible. I’m very satisfied with the services I use.

I started using Watchr in a couple of projects. An interesting use of Watchr is to check on your tests. I have been using this script with great success so far:

 1 def failed
 2   system("growlnotify --name adgear-reporting-tests --image /Applications/Mail.app/Contents/Resources/Caution.tiff -m 'Oops, tests failed'")
 3   system("say 'failed'")
 4 end
 5 
 6 def succeeded
 7   system("growlnotify --name adgear-reporting-tests -m 'Green tests'")
 8   system("say 'pass'")
 9 end
10 
11 watch('(lib|test)/.+\.(js|rb)') do |_|
12   system("rake")
13   $?.success? ? succeeded : failed
14 end

Notice I’m using Mac OS X application paths, but it works just fine for me. And the regular “pass” and “failed” messages keep me abreast of what’s going on. But this style of continuous testing only works when your tests take a few seconds at most. When my tests take much longer than that, I start fiddling, opening Google Reader, checking news, whatever.

Your mileage may vary.