Fear of Fish

It’s been a while since I posted anything, and I intend to fix that with smaller posts that describe things I’m doing. So let’s start off this new regime with a little post about a tiny script I wrote the other day for my WoW guild. We are now using a twitter account to post scheduled raids and other announcements, and I knocked together a little script that parses our guilds (Enigma on Hellfire) guildomatic site for raids we’ve rostered for later that day, and it posts them to twitter.

I needed to parse some html and for that I use the venerable HPricot gem, and to communicate with Twitter I use the Twitter gem. Here’s the code:

  require 'rubygems'
  require 'twitter'
  require 'hpricot'  
  require 'open-uri'  
  require 'parsedate'

  TWITTER_USERNAME = 'replaceme'
  TWITTER_PASSWORD = 'replaceme'
  # http://enigma-hellfire.guildomatic.com/ is ours
  GUILD_SITE = 'replaceme'

  def get_raids(doc)  
    message = "[RAID REMINDER] "
    curr_date = Time.new
    raids_today = false
    (doc/"tr.today").each do |post|
      raids_today = true
      (post/"td.clickable/a").each do |name|
        unless name.inner_html.include?( 'Roster' ) or name.inner_html.include?( 'roster' )
          message << name.inner_html 
          (post/"td.inviteAtTime").each do |start_time|
            message << " (" << start_time.inner_html << ") | "
          end
        end
      end
    end
    raids_today ? message[0..-3] : nil
  end  
  
  doc = Hpricot(open(GUILD_SITE))  

  httpauth = Twitter::HTTPAuth.new(TWITTER_USERNAME, TWITTER_PASSWORD)
  base = Twitter::Base.new(httpauth)
  message = get_raids(doc)

  if message
    base.update(message)
  else
    base.update("[RAID REMINDER] There are no raids today, take a break, slacker!")
  end

Guildomatic has the raids for today posted in a table cell with a class of ‘today’, which made this much easier. You get the idea from the code.

I’m also cheered up immensely on what 10 minutes of key bashing can output. I run this on my server (hosted by Linode) in a cron job at 12pm every day.

I wrote this article for the first edition of “The Rubyist”, and having left it a good set of months, I’m posting it here for your perusal.

The internet is full of tutorials and blogs on Rails, but lacking in the Ruby Gem building department, and as part of my work for Engine Yard over the last year I’ve been building internal tools packaged as Ruby Gem’s. I’d like to share my method’s with you now, and I hope it helps you building your own Gem’s.

We’re going to build a Gem that translates from American to British, it’s going to be simple, but it will demonstrate how to think about what you’re making and how to go about it. We’ll be testing with RSpec, for more details on the syntax of RSpec, please see their site.

If you’d like to skip ahead and see the entire application, I’ve put it on my github account.

Skeleton Structure

I use a gem called Mr Bones to generate the structure for my Gem’s, you can get it yourself using the gem command. On windows, omit the sudo command.

  $ sudo gem install bones

I use bones to generate a skeleton structure for our gem. It creates Rake files that do lots of funky bits and pieces for us, as well as a good folder structure and a ready to roll ‘Britify’ module under the lib folder. Let’s create that now.

  $ bones create britify

Mr Bones has now built us a structure, and all you have to do is edit the files Mr Bones told you to. The next step is to start building up the specs for our Gem, so let’s get cracking.

Start the Build Cycle

There is a specs/britify_spec.rb file already in place, but I prefer to structure my spec folder so that it automatically works with the autotest application. We won’t be using it in this tutorial, but bear in mind that I usually have it running so that I don’t need to re-run my tests constantly. I suggest you look into it on the ZenTest web site.

If we think about how we’re going to deal with the translations a little bit, it makes sense that we’ll need a Translation class to do the actual work. Now you’ll see exactly how BDD really works, as we build our first spec. Go ahead and create the following file, I’ve annotated it with comments to describe each piece. You’ll need to duplicate the exact folder structure for any file I describe in a code block, like below, use the first comment which tells you where to create it.

  # spec/britify/translate/translate_spec.rb
  # $Id$
  
  # Require the spec helper relative to this file
  require File.join(File.dirname(__FILE__), %w[ .. .. spec_helper])
  
  # No need to type Britify:: before each call
  include Britify
  
  describe Translate do
    # All of our specs for Translate will go in here
    
  end

  # EOF

It’s pretty empty right now, but the basics are in place. In fact, if we run this spec it will fail which is exactly what we’d expect. Let’s get something in it first though.

A Translate class, in my opinion, would be instantiated without the need for any arguments. This seems like a good place to start for our specs. In between the describe block above, let’s elaborate on that thought.

  # spec/britify/translate/translate_spec.rb
  it "should be instantiated without any arguments" do
    lambda { Translate.new(  ) }.should_not raise_error
    lambda { Translate.new( "moo" ) }.should raise_error(ArgumentError)
  end

This seems reasonable enough. Running rake on the command line (to run our tests) tells me that I have an uninitialized constant Translate. Well of course I have that error, I haven’t created that class yet! I use this as my sanity check, now it’s time to make that spec pass.

  # lib/britify/translate/translate.rb
  module Britify
    class Translate
    end
  end

Run your specs and you’ll get a successful pass. Brilliant. Further into the rabbit hole we go! I would expect our class to have a method that accepts an American sentence or word and returns the British version, I’d probably call it translate, but that’s too big a leap for my liking, I’ll tone it down to go in little steps.

  # spec/britify/translate/translate_spec.rb
  it "should accept a string on a translate method, and return a string" do
    t = Translate.new
    t.translate("wonky").should be_instance_of(String)
  end

It seems like a strange test, because we’re merely saying that if I pass in a string that i should get one returned. We’re not saying that it should be a correct translation, but that’s the whole point. We want to go in small steps so the specs cover as much of our thinking as possible and basically describe our application logic. The pattern from here on is pretty similar, so apart from the explanation of my thought pattern I’ll whizz you through each file and change that we make as we go.

  # lib/britify/translate/translate.rb
  def translate( string )
    "wonky?"
  end

Yup, tests are passing again. Now, before I can implement a translation method I’ll be wanting somewhere to store my translations for looking up. I like the idea of YAML for this simple app, I’ll store it in data/translations.yml, and I’m thinking a simple hash of key value pairs like "American" => "British" will suffice for now.

It makes sense that we’ll want the Translate class to load this in straight away so they’re immediately available for translations.

  # spec/britify/translate/translate_spec.rb
  it "should load the translations in from the data file on instantiation" do
    IO.should_receive(:read).and_return(
      "--- \nshut your gob: shut your mouth\nsnog: make out"
    )
    t = Translate.new
    t.translations.should be_a_kind_of(Hash)
    t.translations.keys.size.should == 2
  end

Notice the should_receive method there? This basically overrides the real functionality and fakes a response. The usual reason for this is that we don’t want to test the functionality of others classes and libraries, only our own. However, the real reason here is that as our translations grow the test will fail. So I fake a response in this spec and I’ll test the functionality more in other tests to ensure as we grow everything stays sane.

We need to make two changes to our Translate class for this to pass, the first is we need to require the YAML library at the top of the Translate class (above everything).

  require 'yaml'

Secondly we need to implement the YAML loading functionality.

  # lib/britify/translate/translate.rb
  def initialize
    translations_file = File.join(File.dirname(__FILE__), %w[ .. .. .. data translations.yml ])
    @translations = YAML.load( IO.read(translations_file) )
  end

We also need to expose the instance variable as a reader.

  # lib/britify/translate/translate.rb
  
  attr_reader :translations

Okay, so far so good. Our specs are ready for our simple YAML store, and our tests are passing. How about a real translations file.

  # data/translations.yml
  --- 
  # American: British
  shut your mouth: shut your gob
  make out: snog
  not straight: wonky
  rubber boots: wellies

Now’s the time to start getting some real translation tests in place. I’ll adjust the spec for accepting a string to need a real translation.

  # spec/britify/translate/translate_spec.rb
  it "should accept a string on a translate method, and return a string" do
    t = Translate.new
    t.translate("not straight").should be_instance_of(String)
    t.translate("not straight").should == "wonky"
  end

I’ve also noticed I’m repeating myself instantiating a Translate class, let’s dry that up. Place the following immediately after the describe Translate do line in our spec.

  # spec/britify/translate/translate_spec.rb
  setup do
    @t = Translate.new
  end

Now change any reference of t.translate to @t.translate and get rid of the Translate.new calls too (except for the one you just added in the setup block).

Let’s get our specs passing again.

  # lib/britify/translate/translate.rb
  def translate( string )
    @translations[string.downcase]
  end

Summary

At this point we have a working (in a comic way) translation library, it needs a lot of work to make it stable though. Using your new found BDD cycle skills you could easily get in some more functionality and sanitize the input. Here’s some examples of what needs doing:

1. Sanitize the input from non-alphabetic characters
2. Create a command line interface for it
3. Add more translations!

If you’d like to play around with the idea some more, a more rounded version (with the completed tasks above) is available on my github account:

Please do fork it and make changes, I’m open to patches. I hope you’ve enjoyed this view into my development mind and if you have any questions please email me.

A while ago I gave away my Ruby on Rails handout for beginners. It was a PDF that contained the real basics for getting started or at least understanding the fundamentals of what Ruby on Rails is all about. However, my blog changed, the article url moved and ultimately it disappeared.

Well here it is again, there are plenty of books like this out there, but hopefully there’s something in it that will help you start.

In the works is the Ruby on Rails guide to testing, featuring Shoulda and Machinist.

For now, here’s the Ruby on Rails Beginner Handout again.

One of my last projects required me to spend a ridiculous amount of time categorising images into groups of colour. What a pain in the gluteus maximus. So bollocks to that, let’s see if we can’t automate it a little to cut down on the flak.

Prepare

So, the tools for today’s script are:

1. Imagemagick (Mac OS X: port install imagemagick)
2. Color-Tools (gem install color-tools)
3. Optional: Ruby on Rails (I’m doing this in a Rails app)
4. Optional: ActiveRecord
5. Optional: Paperclip (For file uploads)
6. Optional: Marmite on Toast (Om nom nom)

Once you have these tools, then let’s get a rails application together and a model to handle our images.

23:57 abloke@hercules:~/a/demo % rails classify
<snip>
      ...
      create  log/test.log
23:57 abloke@hercules:~/a/demo % ./script/plugin install git://github.com/thoughtbot/paperclip.git
<snip>
From git://github.com/thoughtbot/paperclip
 * branch            HEAD       -> FETCH_HEAD
23:57 abloke@hercules:~/a/demo % ./script/generate model Fabric

Adjust the migration to add the paperclip fields (what the heck, add a design field too).

# app/models/fabric.rb
class CreateFabrics < ActiveRecord::Migration
  def self.up
    create_table :fabrics do |t|
      t.string   :design
      t.string   :nearest_color
      t.string   :file_file_name, :string
      t.string   :file_content_type, :string
      t.integer  :file_file_size
      t.datetime :file_updated_at

      t.timestamps
    end
  end

  def self.down
    drop_table :fabrics
  end
end

Add the Fabric model with our Paperclip method call has_attached_file.

  class Fabric < ActiveRecord::Base
    has_attached_file :file, 
                      :styles => { :large => "340x340#", :medium => "135x135#", :thumb => "70x70#" }
  end

Now, let’s not jump right in to using a form in our Rails application, nobody likes to be premature. We’re going to play around on the console, and then you can decide to implement this in a rake task, a script, or a form. We’re going to need to require our color-tools.

# config/initializers/color-tools.rb
require 'color/palette/monocontrast'

Now let’s put a few helpers on our model so that we know everything’s working okay.

class Fabric < ActiveRecord::Base
  has_attached_file :file, 
                    :styles => { :large => "340x340#", :medium => "135x135#", :thumb => "70x70#" }

  attr_reader :red, :green, :blue
  
  def to_rgb
    command = "convert #{file.to_file.path} -scale 1x1\! -format '%[pixel:u]' info:-"
    color = %x[#{command}]
    
    if color && $?.exitstatus != 0
      raise StandardError, "There was an error determining the color!"
    end
    @red, @green, @blue = color[/rgb\((.*)\)/, 1].split(",").collect(&:to_i)
  end
end

This could be a lot cleverer? Silence minion! We’re just hacking away. Let’s try this out on our trusty console. Grab a lollipop (I have red or green or blue for your picking) and tally ho!

00:20 abloke@hercules:~/a/demo/classify % rake db:migrate
(in /Users/abloke/a/demo/classify)
==  CreateFabrics: migrating ==================================================
-- create_table(:fabrics)
   -> 0.0100s
==  CreateFabrics: migrated (0.0102s) =========================================

00:20 abloke@hercules:~/a/demo/classify % ./script/console --sandbox
Loading development environment in sandbox (Rails 2.2.2)
Any modifications you make will be rolled back on exit
>> f = Fabric.new :design => 'red'
=> #<Fabric id: nil, design: "red", file_file_name: nil, string: nil, file_content_type: nil, file_file_size: nil, file_updated_at: nil, created_at: nil, updated_at: nil>
>> f.file = File.open('test/fixtures/images/red.jpg')
=> #<File:test/fixtures/images/red.jpg>
>> f.to_rgb
=> [247, 2, 1]

Pow! Now we have RGB values for any files we upload, that’s pretty schweet. Let me explain.

command = "convert #{file.to_file.path} -scale 1x1\! -format '%[pixel:u]' info:-"
color = %x[#{command}]

Here we build up a command that we’re going to fire off on the command line. The command is ‘convert’, which is an ImageMagick command that has lots of juicy options. One of them is --scale 1x1\! which will reduce an image (!) to whatever dimensions you give it. We give an FX Expression of --format '%[pixel:u]', this lets ImageMagick know we want the first image out of our sequence (yes we only have one, but it’s necessary). Finally, we pass info:-, which changes the output from a file to information only.

So really it’s self explanatory. Take our image, squash it to 1 pixel (giving us the predominant colour), then output RGB information on it. We take that information and parse it. So what now, buttercup? Well, surely we don’t want to categorise by just red green or blue, we need something else.

class Fabric < ActiveRecord::Base
  COLORS = {
    :blacks   => { :readable => 'Blacks & Aubergines',  :rgb => [ 30, 0, 0 ] },
    :browns   => { :readable => 'Beiges & Browns',      :rgb => [ 150, 75, 0 ] }, 
    :greens   => { :readable => 'Greens',               :rgb => [ 18, 60, 53 ] }, 
    :neutrals => { :readable => 'Neutrals',             :rgb => [ 210, 188, 151 ] }, 
    :reds     => { :readable => 'Reds & Pinks',         :rgb => [ 133, 51, 53 ] },
    :blues    => { :readable => 'Blues',                :rgb => [ 131, 155, 181 ] }
  }
  
  def self.readable_color_for( color )
    COLORS[color.to_sym][:readable]
  rescue NoMethodError
    raise Fabric::UnknownColorException
  end
end

Here’s a simple set of colours I want to group my images in. The RGB needs tweaking, but it will do for our examples. I’ve made a hash of the RGB value and the readable version, I dislike this implementation immensely, but this was only hacked together for an initial import so there’s no need to be anal. At the bottom I tagged on a quick helper for iterating over in my views.

Okay, now on to categorizing the colours by group. To start off with, I’ll override Paperclip’s file= method, because I want access an uploaded file immediately.

def file=( file_object )
  categorize(file_object.to_tempfile.path) unless file_object.is_a? String
  attachment_for(:file).assign(file_object)
end

def to_rgb( filename=nil )
  filename ||= file.to_file.nil? ? nil : file.to_file.path
  raise("There is no image to determine the RGB values on") if filename.nil?
  command = "convert #{filename} -scale 1x1\! -format '%[pixel:u]' info:-"
  color = %x[#{command}]
  
  if color && $?.exitstatus != 0 && @whiny_thumbnails
    raise StandardError, "There was an error determining the color!"
  end
  @red, @green, @blue = color[/rgb\((.*)\)/, 1].split(",").collect(&:to_i)
end

def to_hex
  to_rgb if @red.nil? or @green.nil? or @blue.nil?
  "#%2x%2x%2x" % [ @red, @green, @blue ]
end

Once I’ve categorised the file I want Paperclip to continue on with it’s business, so I use attachment_for. Notice I’ve shown the to_rgb method again, purely because I changed it. When file= is called we don’t have a file object yet, it’s not been moved from the Tempfile that uploading creates, so we pass over the filename directly. I’ve also shown my rudimentary to_hex method (with a recursive error problem that could potentially lock our app up, note to user), which was useful for colouring in text field backgrounds with visual indicators of the colour group, for debugging.

Next, the actual categorising…grip on to something, this is both ugly and confusing!

def categorize( filename=nil )
  filename ||= file.to_file.nil? ? nil : file.to_file.path
  # Create a new contrast object
  mono = Color::Palette::MonoContrast.new(Color::RGB.new(0,0,0))
  # Grab the RGB values of our current image
  my_color = self.to_rgb(filename)
  compare_to = Color::RGB.new(my_color[0], my_color[1], my_color[2])
  
  # Which category does it have the least contrast against?
  self.nearest_color = COLORS.min do |a, b| 
    color1 = Color::RGB.new(a[1][:rgb][0],a[1][:rgb][1],a[1][:rgb][2])
    color2 = Color::RGB.new(b[1][:rgb][0],b[1][:rgb][1],b[1][:rgb][2])
    mono.color_diff(color1, compare_to) <=> mono.color_diff(color2, compare_to)
  end[0].to_s
end

Oh my gosh. I am ashamed of this hackery, but it did the trick in a short amount of time. The key to what we’re doing here is using the MonoContrast class to tell us the difference in contrast between two colours. Which will work for the most part, but I wouldn’t enter it into any competitions!

In essence its really simple, just badly written. The first part is commented, as for the second part, we merely loop over each colour group creating an RGB object for each. Once we have that we can get a value for the amount of contrast difference. By doing this in a min loop on the Hash, we get the winner.

Summary

Clearly this isn’t the most accurate colour categorising in the world. In some cases it’s plain stupid. However, with the limited amount of knowledge I have on ImageMagick, and technical info on colours, this worked for my simple little task. It took a total of 30 minutes (ignoring the 30 minutes I tweaked it after I’d used it for the import), and if it helps anyone then I’m happy to have helped. It was certainly an interesting problem and no doubt I’ll revisit it later as there are numerous applications. I mean, what if a user would like to upload a photo of their kitchen and be shown matching items that would go with it? Interesting possibility.

Farewell, my blog reading chums.

It is with great pleasure and humility that I present to you, my new writings. I have decided to focus more on my real name and less on my dreamt up internet persona. Greetings everyone, for I am Jamie van Dyke a.k.a. fearoffish.

I have a new beginning. A new venture. Going back, shall we say, to when I taught so many through my snippets of education. I long to again provide that knowledge to you, the reader. Not through the training courses that I run, but through this blog.

I do hope you enjoy where we go from here, and urge you to check back regularly as I shall be implementing comments, links to my dear friends, and of course, articles on my latest musings.