piston update is coming. I have the high-level workflow completed. Conceptually, updating is pretty simple:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
module Piston
  module Commands
    class Update < Piston::Commands::Base
      # +wcdir+ is the working copy we're going to change.
      # +to+ is the new target revision we want to be at after update returns.
      def run(wcdir, to)
        working_copy = Piston::WorkingCopy.guess(wcdir)

        logger.debug {"Recalling previously saved values"}
        values = working_copy.recall

        repository_class = values["repository_class"]
        repository_url = values["repository_url"]
        repository = repository_class.constantize.new(repository_url)
        from_revision = repository.at(values["handler"])
        to_revision = repository.at(to)

        logger.debug {"Validating that #{from_revision} exists and is capable of performing the update"}
        from_revision.validate!

        logger.info {"Updating from #{from_revision} to #{to_revision}"}
        working_copy.apply_differences(from_revision.differences_with(to_revision))
      end
    end
  end
end

Obviously, the devil’s in the details… Notice the last line above:

1
working_copy.apply_differences(from_revision.differences_with(to_revision))
from_revision will calculate a set of differences between itself and to_revision. In Subversion speak, that would probably mean an svn log followed by an svn diff, to get all changes (copies + diffs).

What Piston 1.x does is copy the newer file over the original file, and then applies the changes between the last changed revision of the local files and the working copy. This ensures changes that were made are kept in the new revision.

I know I can do it under Subversion, as I have already done it, but what about git ? I can probably use a combination of git-format-patch and git-apply to get the job done. That would certainly work.

I also thought about finding / using a patch implementation in Ruby. I wonder if that would be another acceptable road ? Anybody out there has / knows about a patch implementation in Ruby ?

James Bowes has a super simple git-rebase explanation: git rebase: keeping your branches current.

If you have any difficulty understanding the concept, or want an easy way to explain to someone, go ahead.

I’m just starting to use git-svn and git-rebase, and the combination rocks.

Capistrano Tip'n'Trick

May 3rd, 2008

If like me you didn’t know, you can do that with Capistrano:

1
2
$ cap HOSTS=web0.myapp.com deploy:update_code
cap ROLES=web,app deploy

Not terribly useful for the tasks demonstrated above, but what about this ?

1
$ cap ROLES=app monit:summary

Incidentally, where are these documented ? I dimly remembered reading this somewhere once, and tried it. Works like a charm!

I just received 2 contributions from Josh Nichols:

  • New—repository-type option on piston import to force the repository backend to use (instead of letting Piston guess), and for cases where Piston is unable to guess: ea958dd
  • Test suite reorganization: 1cef7b6 and 9cfa8f3

Both contributions were accepted and are now part of Piston’s master branch. Thank you very much, Josh, for your work.

If you want to help, do not fear !

1
2
3
4
5
6
7
$ git clone git://github.com/francois/piston.git
$ # make changes
$ git commit
$ # fork piston's repository
$ git remote add github git@github.com:YOURNAME/piston.git
$ git push github master
$ # Send me a pull request

Have you ever wished you could access your Amazon Simple Queue Service from the command line ? Now you can:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ AWS_ACCESS_KEY_ID=<your access key>
$ AWS_SECRET_ACCESS_KEY="<the secret access key>"

$ clisqs create my-queue
Queue my-queue was created.

$ clisqs list
my-queue

$ clisqs push --data "this is the message" my-queue
Pushed 19 bytes to queue my-queue

$ cat README.txt | clisqs push my-queue
Pushed 2687 bytes to queue my-queue

$ clisqs push my-queue README.txt
Pushed 2687 bytes to queue my-queue

$ clisqs size my-queue
3

$ clisqs pop my-queue
this is the message

$ clisqs delete --force my-queue
Queue my-queue was deleted.

Installation

1
$ sudo gem install cliaws

Direct-code access

1
2
3
4
5
6
require "rubygems"
require "cliaws"

Cliaws.sqs.push("my-queue", "the data")
the_size = Cliaws.sqs.size("my-queue")
the_message = Cliaws.sqs.pop("my-queue")

S3 ?

This gem also works with S3. See my prior release announcement: Cliaws: command-line access to S3

I just released a new gem on RubyForge: cliaws. Using it, you have access to nice command line shortcuts:

1
2
3
4
5
6
$ AWS_ACCESS_KEY_ID=<your access key>
$ AWS_SECRET_ACCESS_KEY="<the secret access key>"
$ clis3 list BUCKET/PATH-PREFIX
$ clis3 put BUCKET/FULL/PATH/NAME /path/to/local/file
$ clis3 get BUCKET/FULL/PATH/NAME /path/to/local/file
$ clis3 cat BUCKET/FULL/PATH/NAME

This is not rocket science, but with the kind of work I’m doing at the moment, having quick command-line access to S3 is a boon.

Installation

1
$ gem install cliaws

Direct-code access

1
2
3
4
5
6
7
8
9
10
11
12
13
require "rubygems"
require "cliaws"

# Retrieve and store in a variable
value = Cliaws.s3.get("BUCKET/FULL/PATH/NAME")

# Retrieve and stream to specified file
File.open("/path/to/local/file", "w") do |f|
  Cliaws.s3.get("BUCKET/FULL/PATH/NAME", f)
end

# Returns an Array of full names
contents = Cliaws.s3.list("BUCKET")

Repository

The repository is hosted on GitHub, with a mirror on RubyForge.

The repository’s URLs are:

Go forth and fork !

I just stumbled upon Is Windows a First Class Platform for Ruby? by Peter Cooper and Is Windows a supported platform for Ruby? I guess not by Luis Lavena.

I have to admit, I did use Windows daily for 3 years before I switched permanently to Ubuntu, about 6 months back. Initially, I was using IntelliJ’s IDEA as my development platform, then I switched to the e Text Editor. I never had many problems with gems that didn’t/couldn’t/wouldn’t install. There are many kind souls in the community that keep these gems up to date. I’m talking specifically about Tim Hunter (RMagick), Luis Lavena (Mongrel) and others.

Just before I made the final switch, I was using Linux in a VMware image to run XLsuite. The application was unbearably slow in Windows, but acceptable in a virtual machine. I was using the e Text Editor as my editor, accessing the code through a Samba share.

My own experience was pretty positive. Now that I’m on Ubuntu, I wouldn’t go back though. What made me switch ? Better performance on the same hardware, mostly; the novelty of the experience. I do not dislike Windows, nor do I think Redmond is a bad place. Windows is a fine platform.

Just to contrast, look at the excellent support Java enjoys on Windows. Windows is the 2nd platform for Java (with Solaris being the 1st). We, the Ruby community, should be learning from Sun. There are many, many more Windows machines than Mac or Linux machines out there. There are literally millions of people who could learn to write Ruby, but are on Windows. Dr Nic said it all: … People Use Windows Too. Whether you want to or not, Windows isn’t going away soon.

Personally, I have made the switch. But just on my small team, Windows users outnumber other platforms 1 to 1. Here’s the breakdown:

  • 4 Windows (2 coders, 1 designer, 1 sponsor)
  • 2 Ubuntu (1 coder, 1 designer)
  • 1 Mac (1 ExtJS coder)

Is Windows a good platform for Ruby ? Yes. Is Windows a great platform for Ruby ? No, but there’s no reason why it shouldn’t be. Just look at Why’s work on Hackety Hack. I’m really impressed and happy that I will be able to show Ruby to my daughters. The catch ? Hackety Hack is for Windows only right now.

Please, let’s keep and increase Windows support. Once they’re hooked, they might switch, who knows ?

Instead of using Logger, I’m now using Log4r. This is a departure for me, as I initially gave myself the goal of not depending on too many libraries. But since I’m already depending on Main (which itself has 2 dependencies) and open4, I thought, “why not another one ?”

But this new dependency gives me much greater freedom in logging. I’m not done coding all of this, but—verbose won’t just be a flag. It will represent a level, and the higher the level, the more logging will be done. Obvious, but much more interesting.

Anyway, here’s what 1.9.1 was logging for a simple svn/git pistonization:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
$ ruby -I lib bin/piston import http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
D, [2008-03-25T00:41:13.494826 #13759] DEBUG -- : Piston::Commands::Import with options {:verbose=>false, :force=>false, :quiet=>false, :lock=>false, :dry_run=>false}
D, [2008-03-25T00:41:13.495078 #13759] DEBUG -- : Guessing the repository type of "http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement"
D, [2008-03-25T00:41:13.495386 #13759] DEBUG -- : git ls-remote --heads http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
D, [2008-03-25T00:41:13.495543 #13759] DEBUG -- : > "git ls-remote --heads http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement"
D, [2008-03-25T00:41:13.721569 #13759] DEBUG -- : > #<Process::Status: pid=13760,exited(1)>, success? false, status: 1
D, [2008-03-25T00:41:13.722096 #13759] DEBUG -- : svn info http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
D, [2008-03-25T00:41:19.142407 #13759] DEBUG -- : Path: ssl_requirement
URL: http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
Repository Root: http://dev.rubyonrails.org/svn/rails
Repository UUID: 5ecf4fe2-1ee6-0310-87b1-e25e094e27de
Revision: 9088
Node Kind: directory
Last Changed Author: bitsweat
Last Changed Rev: 8780
Last Changed Date: 2008-02-02 00:16:53 -0500 (Sat, 02 Feb 2008)


D, [2008-03-25T00:41:19.142810 #13759] DEBUG -- : Guessing the working copy type of #<Pathname:repository>
D, [2008-03-25T00:41:19.142950 #13759] DEBUG -- : Asking Piston::Git::WorkingCopy if it understands repository
D, [2008-03-25T00:41:19.143063 #13759] DEBUG -- : git status on repository
D, [2008-03-25T00:41:19.143490 #13759] DEBUG -- : git status on .
D, [2008-03-25T00:41:19.143681 #13759] DEBUG -- : git status
D, [2008-03-25T00:41:19.143848 #13759] DEBUG -- : > "git status"
D, [2008-03-25T00:41:19.166951 #13759] DEBUG -- : > #<Process::Status: pid=13772,exited(1)>, success? false, status: 1
D, [2008-03-25T00:41:19.167193 #13759] DEBUG -- : # On branch my1.9.1
nothing to commit (working directory clean)

D, [2008-03-25T00:41:19.167443 #13759] DEBUG -- : Initialized on repository
D, [2008-03-25T00:41:19.167920 #13759] DEBUG -- : svn checkout --revision HEAD http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement .repository.tmp
D, [2008-03-25T00:41:25.100301 #13759] DEBUG -- : A    .repository.tmp/test
A    .repository.tmp/test/ssl_requirement_test.rb
A    .repository.tmp/lib
A    .repository.tmp/lib/ssl_requirement.rb
A    .repository.tmp/README
Checked out revision 9088.

D, [2008-03-25T00:41:25.100986 #13759] DEBUG -- : svn ls --recursive .repository.tmp
D, [2008-03-25T00:41:30.056625 #13759] DEBUG -- : README
lib/
lib/ssl_requirement.rb
test/
test/ssl_requirement_test.rb

D, [2008-03-25T00:41:30.057107 #13759] DEBUG -- : Copying README to repository/README
D, [2008-03-25T00:41:30.058074 #13759] DEBUG -- : Copying lib/ssl_requirement.rb to repository/lib/ssl_requirement.rb
D, [2008-03-25T00:41:30.058994 #13759] DEBUG -- : Copying test/ssl_requirement_test.rb to repository/test/ssl_requirement_test.rb
D, [2008-03-25T00:41:30.059800 #13759] DEBUG -- : svn info --revision 9088 http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
D, [2008-03-25T00:41:34.750474 #13759] DEBUG -- : Path: ssl_requirement
URL: http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
Repository Root: http://dev.rubyonrails.org/svn/rails
Repository UUID: 5ecf4fe2-1ee6-0310-87b1-e25e094e27de
Revision: 9088
Node Kind: directory
Last Changed Author: bitsweat
Last Changed Rev: 8780
Last Changed Date: 2008-02-02 00:16:53 -0500 (Sat, 02 Feb 2008)


D, [2008-03-25T00:41:34.751037 #13759] DEBUG -- : Remembering {"piston:remote-revision"=>9088, "piston:root"=>"http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement", "piston:uuid"=>"5ecf4fe2-1ee6-0310-87b1-e25e094e27de"}
D, [2008-03-25T00:41:34.752256 #13759] DEBUG -- : Calling #after_remember on repository/.piston.yml
D, [2008-03-25T00:41:34.752475 #13759] DEBUG -- : git add .
D, [2008-03-25T00:41:34.752605 #13759] DEBUG -- : > "git add ."
D, [2008-03-25T00:41:34.758728 #13759] DEBUG -- : > #<Process::Status: pid=13785,exited(0)>, success? true, status: 0
D, [2008-03-25T00:41:34.758993 #13759] DEBUG -- : Removing temporary directory: .repository.tmp

And here’s the current master branch:

1
2
3
4
5
6
$ ruby -I lib bin/piston import http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
 INFO main: Guessing the repository type
 INFO main: Guessing the working copy type
 INFO main: Checking out the repository
 INFO main: Copying from Piston::Revision(http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement@9088)
 INFO main: Checked out "http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement" r9088 to "ssl_requirement"

Obviously, in 1.9.1, I hadn’t configured the logger at all, and everything was logged. Not so anymore !

Well, with much more testing, I’m releasing another preview release of Piston. This release should import Subversion or Git repositories into Subversion or Git working copies just fine. There’s one slight problem, which is when you use piston import URL (without a target directory), it will import into a folder named repository, which isn’t what it’s supposed to do. I’m missing a couple of tests, is all.

How to grab this release ? Two ways:

  • git clone git://github.com/francois/piston.git
  • Grab a tarball

Once you have the code, run rake install_gem to install the gem. Enjoy !

I have new integration tests, and they work just beautifully. I’m missing a couple more, but things are looking very good.

Thanks to Paul Watson for finding and fixing two bugs in the Git/Git case.

Finally, I have faisal which offered looking into adding SVK support for the working copy.

I’m making it official. I’m releasing 1.9.0 today:

http://github.com/francois/piston/tree/1.9.0

I’ll put up a gem tomorrow, but if you want to play with Piston now, the best way is to either clone the repository, or grab a tarball directly from GitHub.

What’s implemented in this release ?

  • piston import

But you can import from either SVN or Git, into either SVN or Git. All Piston metadata is stored in a .piston.yml file in the pistonized dir’s root.

If you have questions, problems, comments, go ahead and comment right here, or use Piston’s tracker

As I outlined in Piston will get Git support, the four cases below are now supported (at least for importing):

Repository Working Copy Strategy
Subversion Subversion Use current strategy of storing data in Subversion properties
Subversion Git Clone the Git repository, copy the files over and store the metadata as Subversion properties. Use Git to handle the merging for Piston (Yay!)
Git Subversion svn export the data and use a hidden YAML file to store the metadata in the pistonized directory
Git Subversion Use Git submodules perhaps ? Or git clone + copy + YAML

I’m not in fact using git submodules, or anything fancy. I’m cloning the repository, and copying manually from there. So, nothing fancy. But adding new repository and working copy handlers is so easy:

samples/import_svn_svn.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
#!/usr/bin/env ruby
#
# Import an SVN repository into an SVN working copy.
require File.dirname(__FILE__) + "/common"

@root = @root + "tmp/git_git"
@root.rmtree rescue nil
@root.mkpath

@tmp = @root + "plugin.tmp"

@plugin = @root + "plugin"
@plugin.mkpath
File.open(@plugin + "README", "wb") {|f| f.puts "Hello World"}
File.open(@plugin + "init.rb", "wb") {|f| f.puts "# Some init code"}
Dir.chdir(@plugin) do
  git :init
  git :add, "."
  git :commit, "-m", "initial commit"
end

@wc = @root + "wc"
@wc.mkpath
File.open(@wc + "README", "wb") {|f| f.puts "My local project"}
Dir.chdir(@wc) do
  git :init
  git :add, "."
  git :commit, "-m", "initial commit"
end

repos = Piston::Git::Repository.new("file://" + @plugin.realpath)
commit = repos.at(:head)
commit.checkout_to(@tmp)

wc = Piston::Git::WorkingCopy.new(@wc + "vendor")
wc.create
wc.copy_from(commit)
wc.remember(commit.remember_values)
wc.finalize

samples/import_git_svn.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#!/usr/bin/env ruby
#
# Import a Git project into a Subversion working copy.
require File.dirname(__FILE__) + "/common"

@root = @root + "tmp/git_svn"
@root.rmtree rescue nil
@root.mkpath

@repos = @root + "repos"
@wc = @root + "wc"

@plugin = @root + "plugin"
@tmp = @root + "plugin.tmp"

svnadmin :create, @repos
svn :checkout, "--quiet", "file://" + @repos.realpath, @wc

@plugin.mkpath
File.open(@plugin + "README", "wb") {|f| f.puts "Hello World"}
File.open(@plugin + "init.rb", "wb") {|f| f.puts "# Some initialization code here"}
Dir.chdir(@plugin) do
  logger.debug {"CWD: #{Dir.getwd}"}
  git :init
  git :add, "."
  git :commit, "-m", "initial commit"
end

repos = Piston::Git::Repository.new("file://" + @plugin.realpath)
commit = repos.at(:head)
commit.checkout_to(@tmp)
wc = Piston::Svn::WorkingCopy.new(@wc + "vendor")
wc.create
wc.copy_from(commit)
wc.remember(commit.remember_values)
wc.finalize

Do you see the differences ? They’re all in the setup code. Once we hit commit.checkout_to, everything else is the same.

I’m almost ready to release a release candidate. This will be 1.9.0, and only support the import subcommand. It will at least expose the code to more testing than just what I have.

Oh, and no more Piston 1.3.3: Now with specifications. This version of Piston was tested right from the start.

Git status exit status ?

March 14th, 2008

1
2
3
4
5
$ git status
# On branch master
nothing to commit (working directory clean)
$ echo $?
1

Shouldn’t a successful git status in a git repository return a status code of 0 ? Doing it in a random folder returns a sensible value:

1
2
3
4
5
$ git status
fatal: Not a git repository
Failed to find a valid git directory.
$ echo $?
128

Can anybody shed some light on this ?

I found Possible bug in ‘git status’ exit code is 1 instead of 0. What is the rational for this ? It goes against everything Unix ?!?

Well, it’s working. Piston 2.0 (aka 1.9.0 in Piston’s GitHub repository) can import from a Subversion repository, into a Subversion working copy. And it’s mostly saving the same information as it was previously, in the same way.

But, I received a suggestion/comment from Faisal:

so i’m wondering if it wouldn’t make sense to have the per-vcs metadata formats replaced with something like config/piston.yml, read by piston. the main advantage to doing so would be easier conversion between format types, especially in cases where one type is checking out from another (e.g. svk or git-svn against an svn back-end).

Faisal N Jawdat in a private E-Mail conversation

Actually, this is not a bad idea. Using this method, a Subversion project, imported in a Git repository, would still be able to use Piston to update the vendor branch. The more the information is accessible, the better it will be.

This means Piston needs to provide an upgrade path. I shall do like Subversion, which silently upgrades the working copy when necessary. And to be on the safe side, I shall also include a format in the piston metadata to enable future upgrades with easier handling.

Recapping, Pistonized directories will now have a new, hidden, file:

1
2
3
4
$ ls -A1 vendor/rails
.piston-metadata.yml
activerecord/
...

And the .piston-metadata.yml file would contain something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ cat vendor/rails/.piston-metadata.yml
# What data can Piston expect from this file ?
format: 1

# Which repository handler must we use ?
repository: svn

# Properties that the handler wanted us to save
handler-metadata:
  # Same as piston:remote-revision
  remote-revision: 9025
  # Same as piston:root
  svn-root: http://dev.rubyonrails.org/svn/rails/trunk
  # Same as piston:uuid
  svn-uuid: 5ecf4fe2-1ee6-0310-87b1-e25e094e27de

If you were aware of how Piston was storing properties, you might not that the piston:local-revision property is gone. Instead of hard-coding which revision we need to import, I’ll instead use the last changed date/time. It’s less precise, but makes interoperability with different repository handlers much easier. No need to map between revision 8920 of Subversion to Git commits.

That looks promising, no ?

Next step ? Implementing the Git backend. After that, it’s testing Svn+Git, Git+Svn, Git+Git and Svn+Svn. Hurray!

Well, I just got the answer I needed: how to clone just the tip of a remote Git repository ? Here’s how:

Sometimes you just want to distribute the source code without its history, and that’s where git-archive comes in.

git archive <tree-ish> > my_new_archive.tar

Kate Rhodes in Getting just the tip of a Git repo

I think I might use this in Piston. I’m not too sure yet, as I haven’t implemented piston-git, but it looks like a promising candidate.

 

Search

A picture of me

I am François Beausoleil, a Ruby on Rails coder. During the day, I work on XLsuite. At night, I am interested many things. Read my biography

Tags

(3) (1) (0) (2) (1) (1) (2) (2) (1) (2) (1) (2) (1) (2) (1) (1) (1) (1) (2) (14) (1) (1) (1) (1) (2) (1) (1) (2) (0) (1) (2) (1) (3) (1) (1) (1) (1) (1) (1) (0) (3) (2) (1) (2) (2) (1) (3) (2) (8) (8) (9) (12) (1) (1) (3) (1) (1) (1) (1) (1) (1) (2) (2) (2) (1) (1) (3) (1) (3) (1) (0) (21) (1) (1) (0) (1) (1) (1) (21) (23) (1) (1) (13) (1) (1) (2) (3) (1) (1) (4) (1) (2) (3) (0) (1) (7) (3) (1) (5) (5) (2) (2) (2) (4) (6) (7) (1) (0) (1) (1) (2) (2) (1) (4) (12) (2) (1) (2) (4) (1) (1) (1) (2) (8) (2) (3) (2) (2) (1) (3) (1) (1)

Links

Projects I work on

Categories

Archives