Piston 2.0.8 is here. This is a minor bugfix release:

  • piston status with no path would not check any status. Thanks to Michael Grosser for the heads up;
  • The ActiveSupport gem deprecated require “activesupport” in favor of “active_support”. Thanks for Michael for reporting this as well;
  • Scott Johnson reported and fixed a problem where a git mv would fail because a parent directory was missing.

Thanks to all contributors!

Installing


1 $ gem install piston

At yesterday’s Montreal.rb I presented Nestor, an autotest-like framework. This is it’s official release announcement.

Nestor is different in that it uses an explicit state machine, namely Aaron Pfeifer‘s StateMachine. Nestor also uses Martin Aumont’s Watchr to listen to filesystem events. But the biggest difference is that the default Rails + Test::Unit is a forking test server. Nestor will load the framework classes—ActiveRecord, ActionController, ActionView, plugins and gems—only once. This saves a lot of time on the aggregate versus running rake everytime.

Nestor's state diagram with events denoting success or failure of a run, and states such as green, running_all or run_focused_pending.
Click for larger version

This release of Nestor is 0.2 quality: it’s not ready for large projects. It only supports Rails + Test::Unit, probably doesn’t run on 1.9 or JRuby, but it’s a solid foundation for going further. In the coming days, I will blog on the internals of Nestor and how StateMachine allowed me to add plugins with very little effort.

Installation


1 $ gem install nestor
2 $ cd railsapp
3 $ # edit config/environments/test.rb to set cache_classes to false
4 $ nestor

I already have a plugin that enables Growl notifications. Install and use:


1 $ gem install nestor_growl
2 $ cd railsapp
3 $ nestor start —require nestor/growl

The —require option is where plugins are loaded. This is an Array of files Nestor will require on startup.

Notes

You must set cache_classes to false in test mode for now. This is a limitation of how Rails boots. With cache_classes set to true, Rails will load the controllers and models when it boots. Since this happens before forking, the code under test would never get reloaded. Did I say it was 0.2 quality?

I just pulled in a couple of patches from outside contributors. These are all minor bug fixes, but are important:

  • Chris Gibson: When forcing the repository type, Piston would break because it called #downcase on a Symbol. 1bcc16bf8
  • Terry Heath: Subversion‘s —non-interactive would prevent OS X’s keychain from kicking in. 93d9a957
  • Florian Aßmann: In certain cases, the revision would be a String, and other times it would be an Integer. Normalize before comparing. 40c0bc4e

All users are advised to upgrade:


1 $ sudo gem install piston

I just release Piston 2.0.4. This is a minor fix to enable Piston to work with more Git versions.

Git 1.6.0.1 accepts checking out a new branch without specifying the remote’s name:


1 $ git checkout b my2-3-stable 2-3-stable
2 # implies origin/2-3-stable

Versions prior to that expect the full name of the remote branch:


1 $ git checkout b my2-3-stable origin/2-3-stable

Installation is the usual incantation: sudo gem install piston

What is Piston?

Piston is a utility that eases vendor branch management. This is similar to svn:externals, except you have a local copy of the files, which you can modify at will. As long as the changes are mergeable, you should have no problems.

If you need to run Piston on Windows, BoxCycle wrote some information about it at: Rails Plugin Updates, SVN, and Piston 2.0.2 on Windows

Wow, without my ever doing anything special, searching for “piston” on Google returns this:

Piston is the #1 search result on Google for the string 'piston'

Thanks to everyone who linked to Piston and made this happen.

Tapajos has been helping me shaping Piston into shape. And I’m working on update now, so expect a 1.9.3 RSN

piston update is coming. I have the high-level workflow completed. Conceptually, updating is pretty simple:


1 module Piston
2 module Commands
3 class Update < Piston::Commands::Base
4 # wcdir is the working copy we’re going to change.
5 # to is the new target revision we want to be at after update returns.
6 def run(wcdir, to)
7 working_copy = Piston::WorkingCopy.guess(wcdir)
8
9 logger.debug {"Recalling previously saved values"}
10 values = working_copy.recall
11
12 repository_class = values["repository_class"]
13 repository_url = values["repository_url"]
14 repository = repository_class.constantize.new(repository_url)
15 from_revision = repository.at(values["handler"])
16 to_revision = repository.at(to)
17
18 logger.debug {"Validating that #{from_revision} exists and is capable of performing the update"}
19 from_revision.validate!
20
21 logger.info {"Updating from #{from_revision} to #{to_revision}"}
22 working_copy.apply_differences(from_revision.differences_with(to_revision))
23 end
24 end
25 end
26 end

Obviously, the devil’s in the details… Notice the last line above:


1 working_copy.apply_differences(from_revision.differences_with(to_revision))

from_revision will calculate a set of differences between itself and to_revision. In Subversion speak, that would probably mean an svn log followed by an svn diff, to get all changes (copies + diffs).

What Piston 1.x does is copy the newer file over the original file, and then applies the changes between the last changed revision of the local files and the working copy. This ensures changes that were made are kept in the new revision.

I know I can do it under Subversion, as I have already done it, but what about git ? I can probably use a combination of git-format-patch and git-apply to get the job done. That would certainly work.

I also thought about finding / using a patch implementation in Ruby. I wonder if that would be another acceptable road ? Anybody out there has / knows about a patch implementation in Ruby ?

I just received 2 contributions from Josh Nichols:

  • New —repository-type option on piston import to force the repository backend to use (instead of letting Piston guess), and for cases where Piston is unable to guess: ea958dd;
  • Test suite reorganization: 1cef7b6 and 9cfa8f3

Both contributions were accepted and are now part of Piston’s master branch. Thank you very much, Josh, for your work.

If you want to help, do not fear !


1 $ git clone git://github.com/francois/piston.git
2 $ # make changes
3 $ git commit
4 $ # fork piston’s repository
5 $ git remote add github git@github.com:YOURNAME/piston.git
6 $ git push github master
7 $ # Send me a pull request

Instead of using Logger, I’m now using Log4r. This is a departure for me, as I initially gave myself the goal of not depending on too many libraries. But since I’m already depending on Main (which itself has 2 dependencies) and open4, I thought, “why not another one ?”

But this new dependency gives me much greater freedom in logging. I’m not done coding all of this, but —verbose won’t just be a flag. It will represent a level, and the higher the level, the more logging will be done. Obvious, but much more interesting.

Anyway, here’s what 1.9.1 was logging for a simple svn/git pistonization:


1 $ ruby I lib bin/piston import http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
2 D, [2008-03-25T00:41:13.494826 #13759] DEBUG -
: Piston::Commands::Import with options {:verbose=>false, :force=>false, :quiet=>false, :lock=>false, :dry_run=>false}
3 D, [2008-03-25T00:41:13.495078 #13759] DEBUG — : Guessing the repository type of "http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement"
4 D, [2008-03-25T00:41:13.495386 #13759] DEBUG — : git ls-remote -heads http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
5 D, [2008-03-25T00:41:13.495543 #13759] DEBUG -
: > "git ls-remote -heads http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement"
6 D, [2008-03-25T00:41:13.721569 #13759] DEBUG -
: > #<Process::Status: pid=13760,exited(1)>, success? false, status: 1
7 D, [2008-03-25T00:41:13.722096 #13759] DEBUG — : svn info http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
8 D, [2008-03-25T00:41:19.142407 #13759] DEBUG — : Path: ssl_requirement
9 URL: http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
10 Repository Root: http://dev.rubyonrails.org/svn/rails
11 Repository UUID: 5ecf4fe2-1ee6-0310-87b1-e25e094e27de
12 Revision: 9088
13 Node Kind: directory
14 Last Changed Author: bitsweat
15 Last Changed Rev: 8780
16 Last Changed Date: 2008-02-02 00:16:53 0500 (Sat, 02 Feb 2008)
17
18
19 D, [2008-03-25T00:41:19.142810 #13759] DEBUG -
: Guessing the working copy type of #<Pathname:repository>
20 D, [2008-03-25T00:41:19.142950 #13759] DEBUG — : Asking Piston::Git::WorkingCopy if it understands repository
21 D, [2008-03-25T00:41:19.143063 #13759] DEBUG — : git status on repository
22 D, [2008-03-25T00:41:19.143490 #13759] DEBUG — : git status on .
23 D, [2008-03-25T00:41:19.143681 #13759] DEBUG — : git status
24 D, [2008-03-25T00:41:19.143848 #13759] DEBUG — : > "git status"
25 D, [2008-03-25T00:41:19.166951 #13759] DEBUG — : > #<Process::Status: pid=13772,exited(1)>, success? false, status: 1
26 D, [2008-03-25T00:41:19.167193 #13759] DEBUG — : # On branch my1.9.1
27 nothing to commit (working directory clean)
28
29 D, [2008-03-25T00:41:19.167443 #13759] DEBUG — : Initialized on repository
30 D, [2008-03-25T00:41:19.167920 #13759] DEBUG — : svn checkout -revision HEAD http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement .repository.tmp
31 D, [2008-03-25T00:41:25.100301 #13759] DEBUG -
: A .repository.tmp/test
32 A .repository.tmp/test/ssl_requirement_test.rb
33 A .repository.tmp/lib
34 A .repository.tmp/lib/ssl_requirement.rb
35 A .repository.tmp/README
36 Checked out revision 9088.
37
38 D, [2008-03-25T00:41:25.100986 #13759] DEBUG — : svn ls -recursive .repository.tmp
39 D, [2008-03-25T00:41:30.056625 #13759] DEBUG -
: README
40 lib/
41 lib/ssl_requirement.rb
42 test/
43 test/ssl_requirement_test.rb
44
45 D, [2008-03-25T00:41:30.057107 #13759] DEBUG — : Copying README to repository/README
46 D, [2008-03-25T00:41:30.058074 #13759] DEBUG — : Copying lib/ssl_requirement.rb to repository/lib/ssl_requirement.rb
47 D, [2008-03-25T00:41:30.058994 #13759] DEBUG — : Copying test/ssl_requirement_test.rb to repository/test/ssl_requirement_test.rb
48 D, [2008-03-25T00:41:30.059800 #13759] DEBUG — : svn info -revision 9088 http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
49 D, [2008-03-25T00:41:34.750474 #13759] DEBUG -
: Path: ssl_requirement
50 URL: http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
51 Repository Root: http://dev.rubyonrails.org/svn/rails
52 Repository UUID: 5ecf4fe2-1ee6-0310-87b1-e25e094e27de
53 Revision: 9088
54 Node Kind: directory
55 Last Changed Author: bitsweat
56 Last Changed Rev: 8780
57 Last Changed Date: 2008-02-02 00:16:53 0500 (Sat, 02 Feb 2008)
58
59
60 D, [2008-03-25T00:41:34.751037 #13759] DEBUG -
: Remembering {"piston:remote-revision"=>9088, "piston:root"=>"http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement", "piston:uuid"=>"5ecf4fe2-1ee6-0310-87b1-e25e094e27de"}
61 D, [2008-03-25T00:41:34.752256 #13759] DEBUG — : Calling #after_remember on repository/.piston.yml
62 D, [2008-03-25T00:41:34.752475 #13759] DEBUG — : git add .
63 D, [2008-03-25T00:41:34.752605 #13759] DEBUG — : > "git add ."
64 D, [2008-03-25T00:41:34.758728 #13759] DEBUG — : > #<Process::Status: pid=13785,exited(0)>, success? true, status: 0
65 D, [2008-03-25T00:41:34.758993 #13759] DEBUG — : Removing temporary directory: .repository.tmp

And here’s the current master branch:


1 $ ruby -I lib bin/piston import http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement
2 INFO main: Guessing the repository type
3 INFO main: Guessing the working copy type
4 INFO main: Checking out the repository
5 INFO main: Copying from Piston::Revision(http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement@9088)
6 INFO main: Checked out "http://dev.rubyonrails.org/svn/rails/plugins/ssl_requirement" r9088 to "ssl_requirement"

Obviously, in 1.9.1, I hadn’t configured the logger at all, and everything was logged. Not so anymore !

Well, with much more testing, I’m releasing another preview release of Piston. This release should import Subversion or Git repositories into Subversion or Git working copies just fine. There’s one slight problem, which is when you use piston import URL (without a target directory), it will import into a folder named repository, which isn’t what it’s supposed to do. I’m missing a couple of tests, is all.

How to grab this release ? Two ways:

  • git clone git://github.com/francois/piston.git
  • Grab a tarball

Once you have the code, run rake install_gem to install the gem. Enjoy !

I have new integration tests, and they work just beautifully. I’m missing a couple more, but things are looking very good.

Thanks to Paul Watson for finding and fixing two bugs in the Git/Git case.

Finally, I have faisal which offered looking into adding SVK support for the working copy.

I’m making it official. I’m releasing 1.9.0 today:

http://github.com/francois/piston/tree/1.9.0

I’ll put up a gem tomorrow, but if you want to play with Piston now, the best way is to either clone the repository, or grab a tarball directly from GitHub.

What’s implemented in this release ?

  • piston import

But you can import from either SVN or Git, into either SVN or Git. All Piston metadata is stored in a .piston.yml file in the pistonized dir’s root.

If you have questions, problems, comments, go ahead and comment right here, or use Piston’s tracker

As I outlined in Piston will get Git support, the four cases below are now supported (at least for importing):

Repository Working Copy Strategy
Subversion Subversion Use current strategy of storing data in Subversion properties
Subversion Git Clone the Git repository, copy the files over and store the metadata as Subversion properties. Use Git to handle the merging for Piston (Yay!)
Git Subversion svn export the data and use a hidden YAML file to store the metadata in the pistonized directory
Git Subversion Use Git submodules perhaps ? Or git clone + copy + YAML

I’m not in fact using git submodules, or anything fancy. I’m cloning the repository, and copying manually from there. So, nothing fancy. But adding new repository and working copy handlers is so easy:

samples/import_svn_svn.rb


1 #!/usr/bin/env ruby
2 #
3 # Import an SVN repository into an SVN working copy.
4 require File.dirname(FILE) + "/common"
5
6 root</span> = <span class="iv">root + "tmp/git_git"
7 root</span>.rmtree <span class="r">rescue</span> <span class="pc">nil</span> <span class="no"> 8</span> <span class="iv">root.mkpath
9
10 tmp</span> = <span class="iv">root + "plugin.tmp"
11
12 plugin</span> = <span class="iv">root + "plugin"
13 plugin</span>.mkpath <span class="no">14</span> <span class="co">File</span>.open(<span class="iv">plugin + "README", "wb") {|f| f.puts "Hello World"}
15 File.open(plugin</span> + <span class="s"><span class="dl">&quot;</span><span class="k">init.rb</span><span class="dl">&quot;</span></span>, <span class="s"><span class="dl">&quot;</span><span class="k">wb</span><span class="dl">&quot;</span></span>) {|f| f.puts <span class="s"><span class="dl">&quot;</span><span class="k"># Some init code</span><span class="dl">&quot;</span></span>} <span class="no">16</span> <span class="co">Dir</span>.chdir(<span class="iv">plugin) do
17 git :init
18 git :add, "."
19 git :commit, "-m", "initial commit"
20 end
21
22 wc</span> = <span class="iv">root + "wc"
23 wc</span>.mkpath <span class="no">24</span> <span class="co">File</span>.open(<span class="iv">wc + "README", "wb") {|f| f.puts "My local project"}
25 Dir.chdir(wc</span>) <span class="r">do</span> <span class="no">26</span> git <span class="sy">:init</span> <span class="no">27</span> git <span class="sy">:add</span>, <span class="s"><span class="dl">&quot;</span><span class="k">.</span><span class="dl">&quot;</span></span> <span class="no">28</span> git <span class="sy">:commit</span>, <span class="s"><span class="dl">&quot;</span><span class="k">-m</span><span class="dl">&quot;</span></span>, <span class="s"><span class="dl">&quot;</span><span class="k">initial commit</span><span class="dl">&quot;</span></span> <span class="no">29</span> <span class="r">end</span> <span class="no"><strong>30</strong></span> <span class="no">31</span> repos = <span class="co">Piston</span>::<span class="co">Git</span>::<span class="co">Repository</span>.new(<span class="s"><span class="dl">&quot;</span><span class="k">file://</span><span class="dl">&quot;</span></span> + <span class="iv">plugin.realpath)
32 commit = repos.at(:head)
33 commit.checkout_to(tmp</span>) <span class="no">34</span> <span class="no"><strong>35</strong></span> wc = <span class="co">Piston</span>::<span class="co">Git</span>::<span class="co">WorkingCopy</span>.new(<span class="iv">wc + "vendor")
36 wc.create
37 wc.copy_from(commit)
38 wc.remember(commit.remember_values)
39 wc.finalize

samples/import_git_svn.rb


1 #!/usr/bin/env ruby
2 #
3 # Import a Git project into a Subversion working copy.
4 require File.dirname(FILE) + "/common"
5
6 root</span> = <span class="iv">root + "tmp/git_svn"
7 root</span>.rmtree <span class="r">rescue</span> <span class="pc">nil</span> <span class="no"> 8</span> <span class="iv">root.mkpath
9
10 repos</span> = <span class="iv">root + "repos"
11 wc</span> = <span class="iv">root + "wc"
12
13 plugin</span> = <span class="iv">root + "plugin"
14 tmp</span> = <span class="iv">root + "plugin.tmp"
15
16 svnadmin :create, repos</span> <span class="no">17</span> svn <span class="sy">:checkout</span>, <span class="s"><span class="dl">&quot;</span><span class="k">--quiet</span><span class="dl">&quot;</span></span>, <span class="s"><span class="dl">&quot;</span><span class="k">file://</span><span class="dl">&quot;</span></span> + <span class="iv">repos.realpath, wc</span> <span class="no">18</span> <span class="no">19</span> <span class="iv">plugin.mkpath
20 File.open(plugin</span> + <span class="s"><span class="dl">&quot;</span><span class="k">README</span><span class="dl">&quot;</span></span>, <span class="s"><span class="dl">&quot;</span><span class="k">wb</span><span class="dl">&quot;</span></span>) {|f| f.puts <span class="s"><span class="dl">&quot;</span><span class="k">Hello World</span><span class="dl">&quot;</span></span>} <span class="no">21</span> <span class="co">File</span>.open(<span class="iv">plugin + "init.rb", "wb") {|f| f.puts "# Some initialization code here"}
22 Dir.chdir(plugin</span>) <span class="r">do</span> <span class="no">23</span> logger.debug {<span class="s"><span class="dl">&quot;</span><span class="k">CWD: </span><span class="il"><span class="idl">#{</span><span class="co">Dir</span>.getwd<span class="idl">}</span></span><span class="dl">&quot;</span></span>} <span class="no">24</span> git <span class="sy">:init</span> <span class="no"><strong>25</strong></span> git <span class="sy">:add</span>, <span class="s"><span class="dl">&quot;</span><span class="k">.</span><span class="dl">&quot;</span></span> <span class="no">26</span> git <span class="sy">:commit</span>, <span class="s"><span class="dl">&quot;</span><span class="k">-m</span><span class="dl">&quot;</span></span>, <span class="s"><span class="dl">&quot;</span><span class="k">initial commit</span><span class="dl">&quot;</span></span> <span class="no">27</span> <span class="r">end</span> <span class="no">28</span> <span class="no">29</span> repos = <span class="co">Piston</span>::<span class="co">Git</span>::<span class="co">Repository</span>.new(<span class="s"><span class="dl">&quot;</span><span class="k">file://</span><span class="dl">&quot;</span></span> + <span class="iv">plugin.realpath)
30 commit = repos.at(:head)
31 commit.checkout_to(tmp</span>) <span class="no">32</span> wc = <span class="co">Piston</span>::<span class="co">Svn</span>::<span class="co">WorkingCopy</span>.new(<span class="iv">wc + "vendor")
33 wc.create
34 wc.copy_from(commit)
35 wc.remember(commit.remember_values)
36 wc.finalize

Do you see the differences ? They’re all in the setup code. Once we hit commit.checkout_to, everything else is the same.

I’m almost ready to release a release candidate. This will be 1.9.0, and only support the import subcommand. It will at least expose the code to more testing than just what I have.

Oh, and no more Piston 1.3.3: Now with specifications. This version of Piston was tested right from the start.

Well, it’s working. Piston 2.0 (aka 1.9.0 in Piston’s GitHub repository) can import from a Subversion repository, into a Subversion working copy. And it’s mostly saving the same information as it was previously, in the same way.

But, I received a suggestion/comment from Faisal:

so i’m wondering if it wouldn’t make sense to have the per-vcs metadata formats replaced with something like config/piston.yml, read by piston. the main advantage to doing so would be easier conversion between format types, especially in cases where one type is checking out from another (e.g. svk or git-svn against an svn back-end).

Faisal N Jawdat in a private E-Mail conversation

Actually, this is not a bad idea. Using this method, a Subversion project, imported in a Git repository, would still be able to use Piston to update the vendor branch. The more the information is accessible, the better it will be.

This means Piston needs to provide an upgrade path. I shall do like Subversion, which silently upgrades the working copy when necessary. And to be on the safe side, I shall also include a format in the piston metadata to enable future upgrades with easier handling.

Recapping, Pistonized directories will now have a new, hidden, file:


1 $ ls -A1 vendor/rails
2 .piston-metadata.yml
3 activerecord/
4

And the .piston-metadata.yml file would contain something like this:


1 $ cat vendor/rails/.piston-metadata.yml
2 # What data can Piston expect from this file ?
3 format: 1
4
5 # Which repository handler must we use ?
6 repository: svn
7
8 # Properties that the handler wanted us to save
9 handler-metadata:
10 # Same as piston:remote-revision
11 remote-revision: 9025
12 # Same as piston:root
13 svn-root: http://dev.rubyonrails.org/svn/rails/trunk
14 # Same as piston:uuid
15 svn-uuid: 5ecf4fe2-1ee6-0310-87b1-e25e094e27de

If you were aware of how Piston was storing properties, you might not that the piston:local-revision property is gone. Instead of hard-coding which revision we need to import, I’ll instead use the last changed date/time. It’s less precise, but makes interoperability with different repository handlers much easier. No need to map between revision 8920 of Subversion to Git commits.

That looks promising, no ?

Next step ? Implementing the Git backend. After that, it’s testing Svn+Git, Git+Svn, Git+Git and Svn+Svn. Hurray!

Well, it is started. If you want to follow Piston 2.0’s progress, head on over to the Piston GitHub Repository.

If you want, you can register to Piston’s Recent Commits on master.

What have I got so far ?

piston/commands/import.rb

1 require "piston/commands/base"
2
3 module Piston
4 module Commands
5 class Import < Piston::Commands::Base
6 attr_reader :options
7
8 def initialize(options={})
9 @options = options
10 logger.debug {"Import with options: #{options.inspect}"}
11 end
12
13 def run(revision, working_copy)
14 tmpdir = working_copy.path.parent + ".#{working_copy.path.basename}.tmp"
15
16 begin
17 debug {"Creating temporary directory: #{tmpdir}"}
18 tmpdir.mkdir
19 revision.checkout_to(tmpdir)
20 working_copy.create
21 working_copy.copy_from(tmpdir)
22 working_copy.remember(revision.remember_values)
23 working_copy.finalize
24 ensure
25 debug {"Removing temporary directory: #{tmpdir}"}
26 tmpdir.rmtree
27 end
28 end
29 end
30 end
31 end

This is the new import command. Everything is expressed in terms of high-level operations, and the different backends will take care of doing the right thing. In the case of Subversion, checking out the repository will involve running “svn checkout”. For Git, this will be “git clone”, and so on.

#remember in the context above is for storing the properties Piston requires. These are the old piston:root, piston:uuid Subversion properties. The different working copy backends will take care of storing this in a format that is suitable: Subversion properties when available, YAML file otherwise.

Also, Piston will be more like Merb: it will be split in multiple Gems. Piston, the main gem, will install piston-core as well as all backends. piston-svn and piston-git are the first two backends I am planning on adding.

I’m having fun reimplementing Piston like this ! No fun == no new features. Fun == many new features.

Jean-François commented on Piston will get Git support:

You might want to check out braid, formerly known as giston.

Jean-François Couture

I knew about Braid. The way I see it, Braid requires Git. Without it, it is useless. What I want to do with Piston instead is to be completely repository and working copy agnostic. Use the best possible solution given the tools at hand: Subversion properties when appropriate, YAML files if I can’t. Use Git to merge if possible, do it using Subversion if I can’t. And so on.

I even had a request for supporting SVK by Faisal. It will be very important for me to allow any combination of server/working copy. Anyway, more code, less talk. Once I have a workinjg release, you’ll be able to judge the quality of my work.

Now that I’ve had a good change to play with Git, I’m ready to implement Git support in Piston. This will involve a couple of refactorings. I can see 4 cases currently:

Repository Working Copy Strategy
Subversion Subversion Use current strategy of storing data in Subversion properties
Subversion Git Clone the Git repository, copy the files over and store the metadata as Subversion properties. Use Git to handle the merging for Piston (Yay!)
Git Subversion svn export the data and use a hidden YAML file to store the metadata in the pistonized directory
Git Git Use Git submodules perhaps ? Or git clone + copy + YAML

I have no idea how git submodules work, so I can’t really say that I will be handling that last case in the most efficient manner. I’m planning on having completed this work by the end of next week (March 14th). Stay tuned for details !

Well, thanks to Graeme Mathieson, Piston finally sports a piston diff subcommand. This command allows you to know what changes you made vs what’s in the remote repository.

As usual, installation is pretty simple:


1 $ gem install piston

What does piston diff report ? Here it is:


1 $ piston help diff
2 diff: Shows the differences between the local repository and the pristine upstream
3 usage: diff [DIR […]]
4
5 This operation has the effect of producing a diff between the pristine upstream
6 (at the last updated revision) and your local version. In other words, it
7 gives you the changes you have made in your repository that have not been
8 incorporated upstream.
9
10 Valid options:
11 v, —verbose Show subversion commands and results as they are executed
12 -q, —quiet Do not output any messages except errors
13 -r, —revision=REVISION
14 -u, —show-updates Query the remote repository for out of dateness information
15 -l, —lock Close down and lock the imported directory from further changes
16 —dry-run Does not actually execute any commands
17 —force Force the command to run, even if Piston thinks it would cause a problem
18
19 $ piston diff vendor/rails
20 Processing ‘vendor/rails’…
21 Fetching remote repository’s latest revision and UUID
22 Checking out repository at revision 8784
23 diff -urN —exclude=.svn vendor/rails.tmp/actionmailer/lib/action_mailer/base.rb vendor/rails/actionmailer/lib/action_mailer/base.rb
24 -
- vendor/rails.tmp/actionmailer/lib/action_mailer/base.rb 2008-02-08 22:38:25.000000000 -0500
25 + vendor/rails/actionmailer/lib/action_mailer/base.rb 2008-01-11 17:05:42.000000000 -0500
26 @ -348,7 +348,7 @
27 # end
28 # end
29 def receive(raw_email)
30 – logger.info "Received mail:\n #{raw_email}" unless logger.nil?
31 + logger.debug "Received mail:\n #{raw_email}" unless logger.nil?
32 mail = TMail::Mail.parse(raw_email)
33 mail.base64_decode
34 new.receive(mail)
35 @ -445,7 +445,7 @
36 # no alternate has been given as the parameter, this will fail.
37 def deliver!(mail = @mail)
38 raise "no mail object available for delivery!" unless mail
39 – logger.info "Sent mail:\n #{mail.encoded}" unless logger.nil?
40 + logger.debug "Sent mail:\n #{mail.encoded}" unless logger.nil?
41
42 begin
43 send("perform_delivery_#{delivery_method}", mail) if perform_deliveries

Caveat: piston diff uses the command line version of diff. This means Windows users will need to install a compatible version, or write a wrapper script around some preferred tool that supports the same features.

What is Piston ?

Piston is a utility that eases vendor branch management. This is similar to svn:externals, except you have a local copy of the files, which you can modify at will. As long as the changes are mergeable, you should have no problems.

Visit Piston’s home page at http://piston.rubyforge.org/

Well, after reading many interesting articles on Git, I moved Piston’s repository to Gitorious. Clone, fix and extend at will. I will maintain the existing Subversion repository and do releases when fixes come in. I will not maintain Piston itself in any significant ways though.

So, for those of you that expressed an interest in maintaining Piston, go ahead ! Get Git, get coding and publish your repository so I can pull from it.

Piston on Gitorious is available at http://gitorious.org/projects/piston. The mainline (or trunk in Subversion terms) is available as http://gitorious.org/projects/piston/repos/mainline

Due to Real Life encroaching on my minimal spare time, I have decided I have no time to maintain Piston. Therefore, as others have done, I am searching for a maintainer.

To qualify, you need to:

  • have a Rubyforge account;
  • be enthusiastic;
  • and have a need to extend Piston.

Piston is my most successful Open Source project so far, and I want to put it in someone’s capable hands.

Please send me mail to francois@teksol.info if you are interested ?

Are you this person ?

At the last Montreal on Rails, I presented Piston. James Golick is taking care of the videos, and I promised I would put the files up. Here they are, at long last. As soon as I have a link for the video, I’ll put it up here.

The files which I used to do the presentation are OpenOffice.org 2:

Marc-André got my only hard-copy of the cheatsheet, authographed by yours truly. Way to go Marc !

The presentation went rather well. Probably due to some parsing issues, the slides didn’t look that good on Keynote, and Carl’s machine had trouble switching slides fast enough for my talk. I was surprised at the many questions from the group. Thanks for your time everyone. I really appreciated seeing you there.

I just received a mail from Walter McGinnis asking:

In other words start a new client project based on an existing open source project that is an entire rails application like Typo or Mephisto and be able to update, modify, and merge accordingly like you would for plugins with piston.

Walter proposed to use the following technique:


1 svn checkout repos_for_new_client_project new_client_project
2 cd new_client_project
3 piston import repos_for_third_party_open_source_rails_app_like_typo/trunk/app app
4 piston import repos_for_third_party_open_source_rails_app_like_typo/trunk/db db

This would work, but as Walter says:

… it doesn’t seem very DRY.

Since this isn’t the first time I am being asked the question, I decided to investigate a solution. I ended up with something that works just fine. The technique boils down to simply not creating trunk/ and using Piston to create that folder. You have to start from a fresh project to use this technique.


1 $ piston version
2 Piston 1.3.3
3
4 $ svn checkout svn://my-server/project/ ~/project/
5 Checked out revision 0.
6
7 $ cd ~/project/
8
9 $ piston import \
10 http://svn.techno-weenie.net/projects/mephisto/trunk/ \
11 trunk/
12 Exported r2856 from ‘http://svn.techno-weenie.net/projects/mephisto/trunk/’ to ‘trunk/’
13
14 $ svn commit —message "Imported Mephito trunk@2756"
15
16 # Some time later…
17 $ cd ~/project/
18 $ piston update .
19 Processing ‘.’…
20 Fetching remote repository’s latest revision and UUID
21 Restoring remote repository to known state at r2756
22 Updating remote repository to r2857
23 Processing adds/deletes
24 Removing temporary files / folders
25 Updating Piston properties
26 Updated to r2857 (56 changes)
27 $ svn commit —message "Updated Mephisto to r2857 (56 changes)"

If you use Piston to manage sub-directories (such as vendor/rails), everything will still work:


1 $ svn update
2 At revision 2.
3
4 $ piston import http://dev.rubyonrails.org/svn/rails/trunk vendor/rails/
5 Exported r6957 from ‘http://dev.rubyonrails.org/svn/rails/trunk’ to ‘vendor/rails’
6
7 $ svn commit —message "Imported Rails into vendor/rails at r6957"
8
9 $ svn update
10
11 $ piston update
12 Processing ‘.’…
13 Fetching remote repository’s latest revision and UUID
14 unchanged from revision 2857
15 Processing ‘vendor/rails’…
16 unchanged from revision 6957

Well, I’m sorry. The whole Piston package was broken for the last two weeks. In fact, it was a problem with internal reorganizations and broken requires.

This also illustrated the fact that I needed some kind of automated testing… Which I advocate, but didn’t do anything about until now for Piston. Well, no more ! Piston now has a suite of specifications, coded using RSpec.

I briefly touched on this issue a week ago in Piston import bugs and behavioural specifications. Back then, I had three specifications. I now have twenty:


1 $ spec —format specdoc specs
2
3 update when no local changes
4 – retrieves the latest fulltext from the remote repository
5 – records remote revision that was merged
6
7 update when a local change
8 – should merge local changes with remote ones
9
10 convert with no svn:externals
11 – does not touch the working copy
12
13 convert with one svn:externals
14 – removes existing folder to replace with piston export
15 – remembers the revision we converted from
16
17 convert with hard-coded revision in svn:externals
18 – retrieves the specified revision text
19 – locks the pistoned directory to that revision
20
21 convert with non HEAD externals
22 – retrieves the same revision we had in our WC
23 – remembers the revision that was present, not HEAD
24 – does not lock the pistoned directory
25
26 import with a valid repository URL
27 – gets the fulltext of all files
28 – remembers the root of the import
29 – remembers the upstream repository’s UUID
30 – remembers the revision we imported from
31 – remembers the revision this WC was at when we imported
32
33 switching to a branch in the same repository (without local mods)
34 – gets the fulltext of the branch
35 – changes the root of the pistoned dir to the new import location
36 – keeps the upstream repository’s UUID unchanged
37 – remembers the upstream revision we pistoned from
38
39 Finished in 103.968133 seconds
40
41 20 specifications, 0 failures

At the moment, the four most important commands are specified: convert, import, update and switch. And even then, only the golden path. Which means I am missing tons of specifications. But at least, I can now guarantee that Piston works, which is a huge step forward from the 1.3.2 relelase.

As I continue to further specify Piston, things can only get better.

Also, a very big thank you to all the people who told me they had problems with Piston. Bug reports, although never enjoyable, are an absolute necessity.

Well, I originally only set out to build a quick script to help me, then I decided to release it. Now, a few versions later, I find myself without the safety of a suite of behavioural specifications.

I have received three separate reports of piston import being broken. One was even reported in the comments here.

To that end, I have started adding behavioural specifications to Piston. The specs run very slowly, due to the nature of the specifications I wrote:

  1. svnadmin create remote-repository
  2. svn checkout
  3. make some changes
  4. svn commit
  5. svnadmin create local-repository
  6. svn checkout
  7. use the piston commands
  8. working_copy.should match_expectations

Here is one specification:

specs/convert_spec.rb

1 context "convert with no svn:externals" do
2 context_setup do
3 remote_repos</span> = <span class="co">Repository</span>.new <span class="no"> 4</span> <span class="iv">rwc = WorkingCopy.new(remote_repos</span>) <span class="no"> <strong>5</strong></span> <span class="iv">rwc.checkout
6 rwc</span>.mkdir(<span class="s"><span class="dl">&quot;</span><span class="k">/trunk</span><span class="dl">&quot;</span></span>) <span class="no"> 7</span> <span class="no"> 8</span> <span class="iv">rwc.add("/trunk/README", "this is line 1")
9 rwc</span>.commit <span class="no"><strong>10</strong></span> <span class="no">11</span> <span class="iv">rwc.add("/trunk/main.c", "int main() { return 0; }")
12 rwc</span>.commit <span class="no">13</span> <span class="no">14</span> <span class="iv">local_repos = Repository.new
15 lwc</span> = <span class="co">WorkingCopy</span>.new(<span class="iv">local_repos)
16 lwc</span>.checkout <span class="no">17</span> <span class="no">18</span> <span class="iv">lwc.mkdir("/vendor")
19 lwc</span>.commit <span class="no"><strong>20</strong></span> <span class="iv">lwc.update
21
22 convert(lwc</span>.path + <span class="s"><span class="dl">&quot;</span><span class="k">/vendor</span><span class="dl">&quot;</span></span>) <span class="no">23</span> <span class="r">end</span> <span class="no">24</span> <span class="no"><strong>25</strong></span> setup <span class="r">do</span> <span class="no">26</span> convert(<span class="iv">lwc.path + "/vendor")
27 end
28
29 teardown do
30 lwc</span>.revert(<span class="s"><span class="dl">&quot;</span><span class="k">--recursive</span><span class="dl">&quot;</span></span>) <span class="no">31</span> <span class="iv">lwc.status.split.each do |path|
32 FileUtils.rm_rf(path) if path =~ /^\?/
33 end
34 end
35
36 context_teardown do
37 lwc</span>.destroy <span class="no">38</span> <span class="iv">local_repos.destroy
39 rwc</span>.destroy <span class="no"><strong>40</strong></span> <span class="iv">remote_repos.destroy
41 end
42
43 specify "does not touch the working copy" do
44 lwc</span>.status.should == <span class="s"><span class="dl">&quot;</span><span class="dl">&quot;</span></span> <span class="no"><strong>45</strong></span> <span class="r">end</span> <span class="no">46</span> <span class="no">47</span> <span class="r">def</span> <span class="fu">convert</span>(non_options=[], options=[]) <span class="no">48</span> <span class="iv">command = Piston::Commands::Convert.new([non_options].flatten, options)
49 command</span>.logging_stream = <span class="iv">stream = StringIO.new
50 @command.run
51 end
52 end

In RSpec documentation, the developers of RSpec say about #context_setup and #context_teardown:

The use of these is generally discouraged, because it introduces dependencies between the specs. Still, it might prove useful for very expensive operations if you know what you are doing.

Well, I did find a use for it. And thank god for that. The specs run in 20 seconds, and I only have three specifications at this time:


1 $ spec —format=specdoc
2
3 import with a valid repository URL
4 – copies the files into a named directory
5
6 convert with no svn:externals
7 – does not touch the working copy
8
9 convert with one svn:externals
10 – removes existing folder to replace with piston export
11
12 Finished in 18.186754 seconds
13
14 3 specifications, 0 failures

This journey has been interesting. And interestingly enough, the code I am developing to help me in my specs will help me in the original implementation. I now have real WorkingCopy and Repository objects.

One interesting thing: svnadmin create requires a good entropy source, or else it will block, waiting for more entropy to be generated. I had to recompile Subversion and set --with-devrandom=/dev/urandom. In fact, this is a flag that is passed to APR.

I will release a new Piston version today or tomorrow with at least the golden paths tested.

Piston 1.3.0 introduced the switch subcommand, but it did not do one important thing: it did not remember what new repository root we are reading from. Version 1.3.1 corrects that.

The CHANGELOG says:

2007-03-09 1.3.1

  • piston switch would fail if the branch from which we are reading had been
    deleted.
  • piston switch had a major bug. It did not update the piston:root property
    to remember the new repository root. Reported and fixed by Graeme
    Mathieson.
  • piston switch errors out early if not provided with the right arguments.
    Thanks to Graeme Mathieson for the info and patch.
  • New internal command parser. No visible external changes.

Piston is a utility that eases vendor branch management. This is similar to svn:externals, except you have a local copy of the files, which you can modify at will. As long as the changes are mergeable, you should have no problems.

Download using the usual command:


1 $ sudo gem install —include-dependencies piston

The latest release of Piston provides you with the ability to switch the upstream repository locations without losing any history.

For example:


1 $ piston switch http://dev.rubyonrails.org/svn/rails/tags/rel_1-2-1 vendor\rails
2 Processing ‘vendor\rails’…
3 Fetching remote repository’s latest revision and UUID
4 Restoring remote repository to known state at r6010
5 Updating remote repository to http://dev.rubyonrails.org/svn/rails/tags/rel_1-2-1@5990
6 Processing adds/deletes
7 Removing temporary files / folders
8 Updating Piston properties
9 Updated to r5990 (663 changes)

Piston 1.3.0 also shows the revision number of locked directories.

Installation

As usual, nothing could be simpler:


1 $ sudo gem install —include-dependencies piston

Well, I never expected that. Piston is listed on the Ruby Advent Calendar.

Thanks go to Chris Wanstrath and Peter Cooper.

A minor correction to Piston:

  • Import subcommand would fail with a “svn: Explicit target required (‘vendor/rails’ interpreted as prop value)” error. This was a minor error in the import code. Reported by Daniel N.
  • The import subcommand could import another revision than what was intended, if HEAD was updated while the import is in progress.

Install using:


1 $ gem install piston

Piston 1.2.0 is here. From the ChangeLog:

  • New status subcommand. Shows M if locally or remotely modified. Applies to one, many, all folders. This subcommand requires the use of a Subversion 1.2.0 client. Thanks to Chris Wanstrath for the inspiration. His Rake
    tasks are available at http://errtheblog.com/post/38.
  • Minor patch by Miguel Ibero Carreras to make Subversion always use the C locale, instead of the current one. This allows Piston to be used with internationalized versions of Subversion. David Bittencourt later reported the same problem. Thanks!
  • Better handle how update finds it’s latest local revision to prevent conflicts. If you had never locally changed your vendor repositories, this fix will change nothing for you. This helps prevent local conflicts if you had ever applied a local patch.

Installation


1 $ gem install piston
2 Successfully installed piston, version 1.2.0

There is no need to install Piston on your server. A local installation on your development machine is all that you need to administer the remote code.

Caveat

It is imperative that you locally update your piston:local-revision property manually, just this once. Prior to 1.2.0, Piston would almost never update the piston:local-revision property, meaning it would always think there were local changes. With the new update implementation, Piston would try to merge changes that had already occured, causing many conflicts. You need to reset the value of piston:local-revision to the revision where you last did a piston update or import.

An example is in order.


1 $ svn propget piston:local-revision vendor\rails
2 29
3
4 $ svn log r head:1 vendor/rails
5 -
———————————————————————————————————
6 r31 | francois | 2006-11-17 10:05:54 0500 (ven., 17 nov. 2006) | 1 line
7
8 Updated to Rails r4500
9 -
———————————————————————————————————
10 r30 | francois | 2006-11-17 10:03:48 0500 (ven., 17 nov. 2006) | 1 line
11
12 Import Rails r4393
13 -
———————————————————————————————————
14
15 $ svn propset piston:local-revision 31 vendor\rails
16 property ‘piston:local-revision’ set on ‘vendor\rails’
17
18 $ svn commit —quiet —message "Updated piston:local-revision to 31, per ChangeLog instructions"

Visit Piston’s website.

Search

Your Host

A picture of me

I am François Beausoleil, a Ruby on Rails and Scala developer. During the day, I work on Seevibes, a platform to measure social interactions related to TV shows. At night, I am interested many things. Read my biography.

Top Tags

Books I read and recommend

Links

Projects I work on

Projects I worked on