[BioRuby] rq: Zero configuration job scheduler for computer clusters and multi-core

Spotted this 'gem' in the bioruby mailling list.. so sharing it here.

---------- Forwarded message ----------
From: Pjotr Prins <>
Date: Sun, Jul 24, 2011 at 7:09 PM
Subject: [BioRuby] rq: Zero configuration job scheduler for computer clusters and multi-core


Use those cores!

I just created a functional gem for rq, a job scheduler created by
Ara Howard. It allows running jobs in parallel, without any
configuration (just a shared directory between processes/machines).

QUICK START

 install rq using rubygems, after installing sqlite 2.x (dev version,
 on Debian apt-get install libsqlite0-dev)

   gem1.8 install rq-ruby1.8

 the binary is in /var/lib/gems/1.8/bin/, so add that to the path,
 or create a symbolic link

   ln -sf `gem1.8 contents rq-ruby1.8|grep bin/rq$` /usr/local/bin/rq

 now rq should work

   rq --help

 run the integration test

   /var/lib/gems/1.8/bin/test_rq.rb

 set up a directory for your queue - this can be a local, or an NFS/sshfs
 mounted drive:

   rq dir create

 on every node create a queue runner, specifying the number of cores (here 8)

   rq dir feed --daemon --log=rq.log --max_feed=8

 submit two jobs - shell style

   rq dir submit 'sleep 10'
   rq dir submit 'sleep 9'

 check status

   rq dir status

 shows

       ---
       jobs:
         pending: 0
         holding: 0
         running: 2
         finished: 0
         dead: 0
         total: 2
       temporal:
         running:
           min: {2: 00h00m03.49s}
           max: {1: 00h00m03.60s}
       performance:
         avg_time_per_job: 00h00m00.00s
         n_jobs_in_last_hrs:
           1: 0
           12: 0
           24: 0
       exit_status:
         successes: 0
         failures: 0
         ok: 0

 Now, that was easy!!

rq will be a standard feature of the BioLinux VMs.

I also ported the code to Ruby1.9 - it is sitting in the ruby1.9 branch on github:

 https://github.com/pjotrp/rq/tree/ruby1.9

it would be good if others were to test that in some production setup,
before I push that to rubygems.

Pj.
_______________________________________________
BioRuby Project - http://www.bioruby.org/
BioRuby mailing list
http://lists.open-bio.org/mailman/listinfo/bioruby