Please grab cliaws from Rubyforge:


1 $ gem install cliaws

What is Cliaws?

Cliaws is a replacement for the Amazon EC2 API tools, but uses Ruby, and thus does not suffer from a long boot time. Cliaws is also easier to setup:


1 $ AWS_ACCESS_KEY_ID=# Your Access ID
2 $ AWS_SECRET_ACCESS_KEY=# Your secret ID
3 $ # Setup is done, enjoy!
4 $ clis3 list YOUR_BUCKET
5 $ cliec2 launch AMI —keypair KEYNAME

Please view the README for some details. The implementation is still the best place to get information about the options you can pass.

Today sees a new release of Cliaws. This is a simple library to interact with S3 (and other services) through the command-line. This library is very similar to Amazon’s own ec2-* scripts, except this library is written in Ruby.

Interacting with S3 should be no harder than:


1 $ clis3 put my-local-file my-bucket/my-s3-file

Put many files at once:


1 $ clis3 put my-local-file0 my-local-file1 my-bucket/my-s3-directory/

Put an environment variable:


1 $ clis3 put —data $MY_DATA my-bucket/my-env-value

Put stdin too!


1 $ tar czfv /var/cache/mylvmbackup/backup | clis3 put – my-bucket/my-backup/backup-20080825-000000.tar.gz

All of this functionnality is available from with Ruby too:


1 require "rubygems"
2 require "cliaws"
3
4 Cliaws.s3.put("this is the content", "my-bucket/my-s3-file")
5 File.open("my-local-file", "rb") do |io|
6 Cliaws.s3.put(io, "my-bucket/my-s3-file")
7 end

Give the RubyForge gem servers a couple of hours to refresh themselves, and then enjoy!

I just pushed 1.2.0 to GitHub as well as RubyForge. This release has a single major improvement:


1 $ clis3 put A B local/C bucket/dir/
2 A => bucket/dir/A
3 B => bucket/dir/B
4 local/C => bucket/dir/C

put isn’t yet recursive, and note the caveat above: all paths are flattened to their basenames. Enjoy!

Links

Starting from a fresh Rails application (I’m using 2.0.2), install AttachmentFu:


1 script/plugin install http://svn.techno-weenie.net/projects/plugins/attachment_fu/

Edit config/amazon_s3.yml and put this:

config/amazon_s3.yml

1 development:
2 bucket_name: amazon-sqs-development-yourname
3 access_key_id: "your key"
4 secret_access_key: "your secret access key"
5 queue_name: amazon-sqs-development-resizer-yourname

queue_name is new. AttachmentFu does not require this, but we are going to reuse the file from our own code, so better put all configuration in the same place.

Generate a scaffolded Photo model using:


1 $ script/generate scaffold photo filename:string size:integer content_type:string width:integer height:integer parent_id:integer thumbnail:string

Edit app/views/photos/new.erb.html and replace everything with this:

app/views/photos/new.erb.html

1 <h1>New photo</h1>
2
3 <%= error_messages_for :photo >
4
5 < form_for(@photo, :html => {:multipart => true}) do |f| >
6 <p>
7 <label for="photo_uploaded_data">File:</label>
8 <= f.file_field :uploaded_data >
9 </p>
10
11 <p>
12 <= f.submit "Create" >
13 </p>
14 < end >
15
16 <= link_to ‘Back’, photos_path %>


What we did here is simply tell Rails to use a multipart encoded form, and to only provide us with a single file upload field.

Edit app/models/photo.rb and add the AttachmentFu plugin configuration:

app/models/photo.rb

1 class Photo < ActiveRecord::Base
2 has_attachment :content_type => :image, :storage => :s3
3 validates_as_attachment
4 end

Start your server and confirm you can upload a file. No thumbnails were generated as we did not configure any thumbnailing to do. We don’t actually want AttachmentFu to handle that, so we can’t just specify it in the has_attachment call.

To use RightScale’s AWS SQS component, we have to configure it with the access key and secret access key. Add this to the end of the Photo class:

app/models/photo.rb

1 class Photo < ActiveRecord::Base
2 def queue
3 self.class.queue
4 end
5
6 class << self
7 def queue
8 # This creates the queue if it doesn’t exist
9 queue</span> ||= sqs.queue(aws_config[<span class="s"><span class="dl">&quot;</span><span class="k">queue_name</span><span class="dl">&quot;</span></span>]) <span class="no"><strong>10</strong></span> <span class="r">end</span> <span class="no">11</span> <span class="no">12</span> <span class="r">def</span> <span class="fu">sqs</span> <span class="no">13</span> <span class="iv">sqs ||= RightAws::Sqs.new(
14 aws_config["access_key_id"], aws_config["secret_access_key"],
15 :logger => logger)
16 end
17
18 def aws_config
19 return aws_config</span> <span class="r">if</span> <span class="iv">aws_config
20
21 aws_config</span> = <span class="co">YAML</span>.load(<span class="co">File</span>.read(<span class="co">File</span>.join(<span class="co">RAILS_ROOT</span>, <span class="s"><span class="dl">&quot;</span><span class="k">config</span><span class="dl">&quot;</span></span>, <span class="s"><span class="dl">&quot;</span><span class="k">amazon_s3.yml</span><span class="dl">&quot;</span></span>))) <span class="no">22</span> <span class="iv">aws_config = aws_config</span>[<span class="co">RAILS_ENV</span>] <span class="no">23</span> raise <span class="co">ArgumentError</span>, <span class="s"><span class="dl">&quot;</span><span class="k">Missing </span><span class="il"><span class="idl">#{</span><span class="co">RAILS_ENV</span><span class="idl">}</span></span><span class="k"> configuration from config/amazon_s3.yml file.</span><span class="dl">&quot;</span></span> <span class="r">if</span> <span class="iv">aws_config.nil?
24 @aws_config
25 end
26 end
27 end

#aws_config is a method that reads the configuration. #sqs is a method that provides access to an instance of RightScale::Sqs, pre-configured with the correct access keys. #queue uses #sqs to get or create a named queue. There’s also an instance version of #queue, to ease our code later on.

Let’s add the request sending:

app/models/photo.rb

1 class Photo < ActiveRecord::Base
2 def send_resize_request
3 # Don’t send a resize request for thumbnails
4 return true unless self.parent_id.blank?
5
6 params = Hash.new
7 params[:id] = self.id
8 params[:sizes] = Hash.new
9 params[:sizes][:square] = "75×75"
10 params[:sizes][:thumbnail] = "100x"
11
12 begin
13 queue.push(params.to_yaml)
14 rescue
15 logger.warn {"Unable to send resize request. Error: #{$!.message}"}
16 logger.warn {$!.backtrace.join("\n")}
17
18 # Don’t raise the error so the request goes through.
19 # We don’t want the user to see a 500 error because
20 # we can’t talk to Amazon.
21 end
22 end
23 end

Now, this is getting interesting. AttachmentFu knows if the current model is a thumbnail or not by looking at parent_id. If it’s nil, we are the parent, else we are a thumbnail. We do the same thing here.

Then, we setup a couple of parameters to send to the resizer. Notice we send the actual thumbnail sizes in the message itself.

Next, we do the most important part: queue.push. This sends a message string (limited to 256 KiB) to Amazon SQS, and returns. If there is an error, we don’t actually want to prevent the request from completing, so we rescue any exceptions and log them. If you have the ExceptionNotifier plugin installed, this is a good place to log to it.

Now that we have a way to send the resize request, we have to execute it at some point. The controller is not the right place to do it. If you create Photo models from more than one controller, you’re bound to forget to call #send_resize_request. It’s better to do it in an #after_create callback, which we’ll do with a single line:

app/models/photo.rb

1 class Photo < ActiveRecord::Base
2 after_create :send_resize_request
3 end

Next, we have to receive the messages. So, we write a new method in Photo:

app/models/photo.rb

1 class Photo < ActiveRecord::Base
2 class << self
3 def fetch_and_thumbnail
4 messages = queue.receive_messages(20)
5 return if messages.blank?
6
7 logger.debug {"==> Photo\#fetch_and_thumbnail — received #{messages.size} messages"}
8 messages.each do |message|
9 params = YAML.load(message.body)
10 photo = Photo.find_by_id(params[:id])
11 if photo.blank? then
12 # The Photo was deleted before we got a chance to thumbnail it.
13 # We must delete the message, or we’ll always get it afterwards.
14 message.delete
15 next
16 end
17
18 photo.generate_thumbnails(params[:sizes])
19 message.delete
20 end
21 end
22 end
23 end

The first thing we do is see if there are any messages. The call to #queue is the helper method we defined earlier on. We ask to receive up to 20 messages at a time. If there were no messages, we simply return.

Then, for each message, we have to process it, so we iterate over each message, retrieving the original parameters Hash. The important thing to do is to delete the message after we have processed it, or else the message will still be visible next time around.

#generate_thumbnails is important, but uninteresting in this discussion.

I was playing with Amazon SQS yesterday night, using RightScale gems, and I was getting strange errors:


1 <?xml version="1.0"?>
2 <Response>
3 <Errors>
4 <Error>
5 <Code>InvalidParameterValue</Code>
6 <Message/>
7 </Error>
8 </Errors>
9 <RequestID>da0bcbc2-967d-4f26-a7f1-36b87079d2ba</RequestID>
10 </Response>

This looked strange until I thought about using a plainer queue name. I switched from “xlsuite.development.resize” to “xlsuite-development-resize”. Sure enough, the error disappeared. I only found out about the requirements for the queue name this morning:

Name Description Required

QueueName

The name to use for the queue created. The queue name must be unique within the scope of all your queues.

Type: String

Constraints: Maximum 80 characters; alphanumeric characters, hyphens (-), and underscores (_) are allowed.

Yes

CreateQueue in the Amazon Simple Queue Service Developer Guide

RTFM already, Fran├žois. Notwithstanding my own stupidity, RightScale’s gems are very easy to use.

Search

Your Host

A picture of me

I am François Beausoleil, a Ruby on Rails and Scala developer. During the day, I work on Seevibes, a platform to measure social interactions related to TV shows. At night, I am interested many things. Read my biography.

Top Tags

Books I read and recommend

Links

Projects I work on

Projects I worked on