Well it turns out, I am kinda tired of dealing with broken threads in Ruby and also threadsaftey issues in gems which I don’t have control over.

So I’ve been playing around with EventMachine+Fibers and came across the em_postgresql gem by Mike Perham (incidentally, an old coworker of mine). Sweet! Mike has done a lot of the initial leg work for making the EM+Fibers paradigm usable in real world Ruby/Rails applications, thanks Mike!

We’ll be using his em_postgresql gem as a case study to show how some of the threading synchronization complexities are still present when using EventMachine+Fibers. Let’s get started.

So Rails uses a database connection pool that assumes you’re running in a threaded environment. Part of the em_postgresql gem is to hack ActiveRecord::ConnectionAdapters::ConnectionPool to use fibers instead of threads (to be “fiber aware” as is sometimes said). Well, unfortunately it doesn’t work, as this code snippet demonstrates:

gem "postgres-pr"
gem "em_postgresql"
require "eventmachine"
require "fiber"
require "active_record"
require "benchmark"

ActiveRecord::Base.logger = Logger.new(STDOUT)
ActiveRecord::Base.establish_connection :adapter  => "em_postgresql",
                                        :port     => 5432,
                                        :pool     => 2,
                                        :username => "cjbottaro",
                                        :host     => "localhost",
                                        :database => "test",
                                        :timeout  => 3

EM.run do
  Fiber.new do
    fibers = []
    time = Benchmark.realtime do
      fibers << Fiber.new{ ActiveRecord::Base.connection.execute "select pg_sleep(1)" }.tap{ |fiber| fiber.resume }
      fibers << Fiber.new{ ActiveRecord::Base.connection.execute "select pg_sleep(1)" }.tap{ |fiber| fiber.resume }
      fibers << Fiber.new{ ActiveRecord::Base.connection.execute "select pg_sleep(1)" }.tap{ |fiber| fiber.resume }
      fibers.each do |fiber|
        while fiber.alive?
          current_fiber = Fiber.current
          EM.next_tick{ current_fiber.resume }
          Fiber.yield
        end
      end
    end
    puts time
    EM.stop
  end.resume
end

We specify our our pool size to be 2 and we spawn 3 fibers. We expect that only two connections to the database are made. We also expect that the first two SQL statements run in parallel, then the third runs after they are done, giving us a total running time of ~2 seconds.

This is what we get instead:

$ ruby em_postgres.rb 
Connecting to localhost:5432
Connecting to localhost:5432
Connecting to localhost:5432
  SQL (0.6ms)   SET client_min_messages TO 'panic'
  SQL (1.0ms)   SET client_min_messages TO 'panic'
  SQL (1.1ms)   SET standard_conforming_strings = on
  SQL (0.4ms)   SET client_min_messages TO 'notice'
  SQL (1.3ms)   SET standard_conforming_strings = on
  SQL (0.8ms)   SET client_min_messages TO 'panic'
  SQL (1.0ms)   SET client_min_messages TO 'notice'
  SQL (1.0ms)   SET standard_conforming_strings = on
  SQL (0.5ms)   SET client_min_messages TO 'notice'
  SQL (1001.1ms)   select pg_sleep(1)
  SQL (1001.0ms)   select pg_sleep(1)
  SQL (1000.7ms)   select pg_sleep(1)
1.0179660320281982

Three connections are made and all 3 statements run in parallel giving us a running time of ~1 second. What went wrong?

First let’s look at ConnectionPool#checkout.

      def checkout
        # Checkout an available connection
        @connection_mutex.synchronize do
          loop do
            conn = if @checked_out.size < @connections.size
                     checkout_existing_connection
                   elsif @connections.size < @size
                     checkout_new_connection
                   end
            return conn if conn
            # No connections available; wait for one
            if @queue.wait(@timeout)
              next
            else
              # try looting dead threads
              clear_stale_cached_connections!
              if @size == @checked_out.size
                raise ConnectionTimeoutError, "could not obtain a database connection#{" within #{@timeout} seconds" if @timeout}.  The max pool size is currently #{@size}; consider increasing it."
              end
            end
          end
        end
      end

em_postgresql hacks @connection_mutex.synchronize to do nothing. I think the rational is that we don’t have to worry about synchronization problems (like when using threads) because we won’t be preempted unless we explicitly say to in the code (by calling Fiber.yield).

The problem is that we are unknowingly calling methods that deep down eventually call Fiber.yield.

Considering our example case, when we call checkout, the condition elsif @connections.size < @size is met and we call checkout_new_connection which is defined as follows:

      def checkout_new_connection
        c = new_connection
        @connections << c
        checkout_and_verify(c)
      end

Fiber.yield is called not once by twice in that unsuspecting bit of code. Remember that the whole point of em_postgresql is to allow network IO to occur in parallel. Creating a new connection to the database falls under that category, so new_connection is calling Fiber.yield… and the key thing to notice is that the fiber is yielding before @connections is modified. Thus our next fiber will call checkout and hit that same condition as before elsif @connections.size < @size and end up in new_connection where the same thing will happen again with the 3rd fiber. And thus our connection pool’s purpose is defeated.

The second time Fiber.yield is called in that code is in checkout_and_verify(c) which eventually calls #verify! on the connection. em_postgresql defines #verify! to issue a SELECT 1 statement to the database, which is network IO, thus Fiber.yield is called. em_postgres caught the issue with this and overwrote checkout_and_verify(c) as follows:

      def checkout_and_verify(c)
        @checked_out << c
        c.run_callbacks :checkout
        c.verify!
        c
      end
    end

It differs from the original in that @checked_out is modified before calling c.verify! (which yields the fiber). Thus any fibers that enter checkout will see the new size of @checked_out when evaluating the if statements.

This same strategy needs to be applied to checkout_new_connection and @connections. The problem lies in that @connections is a list of connections and the act of simply creating a connection yields the fiber, so it’s impossible to modify @connections before yielding the fiber.

I’ll leave it as an exercise for the reader to come up with a working solution (it’s not too hard).

In conclusion, even without random preempting, as is the case with cooperative concurrency, it is still somewhat difficult to manage synchronization issues… at least I think so.

The problem happens when a thread locks a Mutex (or Monitor) inside a timeout block and the timeout expires before the thread can unlock it.

require "thread"
require "timeout"

N = 10
M = 10
T = 0.1

lock = Mutex.new
func = Proc.new do
  begin
    Timeout.timeout(T){ lock.synchronize{ sleep } }
  rescue Timeout::Error
    nil
  end
end

threads = N.times.collect{ Thread.new{ M.times{ func.call } } }
threads.each{ |thread| thread.join }

puts "no deadlocks, yay!"

The bug doesn’t happen very frequently; you may even have to run that snippet a few times before you see it happen, but my threaded asynchronous task processing daemon has high enough throughput in our production environment to where we were seeing the problem very frequently.

It wouldn’t be so bad if the mutex could get out of its bad state; what’s a failed task here and there out of thousands? Guess it depends on the tasks… ;) Unfortunately, once the mutex is in the bad state, all subsequent tasks run on that thread will fail.

There is hope though! This disgusting little hack completely alleviates the problem…

require "thread"
class Mutex
  def lock_with_hack
    lock_without_hack
  rescue ThreadError => e
    if e.message != "deadlock; recursive locking"
      raise
    else
      unlock
      lock_without_hack
    end
  end
  alias_method :lock_without_hack, :lock
  alias_method :lock, :lock_with_hack
end

Now before you go off and include that snippet in your code, be aware that it effectively turns Mutex into Monitor (i.e. allows recursive locking). In our use case (and I think most), that is fine though.

Here is a little Ruby script you can run on the command line that demonstrates the bug and also shows how the hack fixes it. Run like so:

ruby deadlock.rb                   # demonstrates bug
ruby deadlock.rb --with-mutex-hack # demonstrates fix

I’ve reported the bug on Ruby’s issue tracker where it seems to be getting some attention.

At first I was going to make a SynchronizedMemCacheStore in the same spirit as SynchronizedMemoryStore in ActiveSupport, but we debated over whether a single lock would become a bottleneck. Our threaded asynchronous task processor isn’t running nearly at capacity, but maybe it will someday in the future.

So the fix was to override Rails.cache to use thread local variables.

# inside Rails::Initializer.run in environment.rb
config.after_initialize{ require "threadsafe_rails_cache" }

# threadsafe_rails_cache.rb
module Rails
  def self.cache
    Thread.current["Rails.cache"] ||= begin
      ActiveSupport::Cache.lookup_store(configuration.cache_store).tap do |cache|
        cache.logger = Rails.logger
      end
    end
  end
end

Some ramifications of this…

• RAILS_CACHE is still defined internally by Rails. Obviously you shouldn’t use it and use Rails.cache instead. I’m not sure if Rails uses RAILS_CACHE internally though.
• You’re going to have a lot more connections open to Memcached.

Here’s a nifty graph to show that the fix worked. Red indicates failed tasks, and green indicates successes. The fix was deployed around 12:00.

Versions

• Ruby 1.9.2
• Rails 2.3.10
• memcache-client 1.8.5

I have a Linux NAS, but all my other computers are Macs. NFS is simple, but buggy. There are all sorts of problems if you disconnect from the network without unmounting first. Also, I ran into issues where Finder would just lockup upon opening an NFS mounted directory.

Time to make the switch to AFP.

Installing netatalk

OS X doesn’t like sending clear text passwords, so you have to install libgcrypt-dev so netatalk builds the uams_dhx2.so module.

sudo aptitude install build-essential libdb-dev libgcrypt-dev
wget http://prdownloads.sourceforge.net/netatalk/netatalk-2.1.4.tar.bz2?download -O netatalk-2.1.4.tar.bz2
tar xjf netatalk-2.1.4.tar.bz2
cd netatalk-2.1.4
./configure --prefix=/usr/local/netatalk-2.1.4
make && sudo make install

Because we installed by source, we gotta do some symlinking and creating of config files, etc. First off, I like to make a link in /usr/local.

sudo ln -s /usr/local/netatalk-2.1.4 /usr/local/netatalk

Now we’re going to link the config stuff to /etc.

sudo ln -s /usr/local/netatalk/etc/netatalk /etc/netatalk

Now install this initd script as /etc/init.d/netatalk and edit SBIN_PATH and BIN_PATH according to where you installed netatalk.

BIN_PATH=/usr/local/netatalk/bin
SBIN_PATH=/usr/local/netatalk/sbin

Configuring netatalk

Pretty easy. You just gotta configure two things: afpd and what volumes you want to share.

My /etc/netatalk/afpd.conf is just one line.

- -tcp -noddp -uamlist uams_dhx2.so,uams_guest.so -nosavepassword

It says “my default server uses tcp, not ddp and allows users to authentica via dhx2 or as a guest without saving their password.” At least, that’s what I think it says. The afpd.conf file is well document and it also has a man page.

My /etc/netatalk/AppleVolumes.default is similarly simple.

:DEFAULT: options:upriv,usedots
/mnt/quantum "Quantum" allowed_hosts:192.168.0.0/16

The :DEFAULT: section applies to all volumes. upriv means use Unix privileges and usedots means hide files that start with a dot. I’m only exporting one volume named Quantum and the arguments are pretty self explanatory. Again, the AppleVolumes.default is well documented and there a man page for it as well.

Now you can start it up.

sudo /etc/init.d/netatalk start

And connect to it via Finder -> Go -> Connect to Server…

For the “Server Address:” I fill out afp://hostname/Quantum where “hostname” is my Linux machine’s hostname and “Quantum” is the name of my volume.

The authentication credentials are your Linux username and password. Your Linux user will need permissions on the exported volume (obviously).

Troubleshooting

Can’t connect? See what’s going on by changing your afpd.conf to be something like.

- -tcp -noddp -uamlist uams_dhx2.so,uams_guest.so -nosavepassword -setuplog "default log_debug /tmp/afpd.log"

Now you can tail -f /tmp/afpd.log to see what’s going on when you try to connect. Note, there are other log levels than “log_debug” if you need more info. See the man page.

Configuring Avahi

So by now you’ve noticed that your Linux AFP server doesn’t show up in the Finder’s sidebar like other real Macs do. Not a problem, we can fix that by installing and configuring Avahi to advertise it.

sudo aptitude install avahi-daemon libnss-mdns

Now edit /etc/nsswitch.conf and make sure the “hosts” line has “mdns” in it.

hosts:          files mdns4_minimal [NOTFOUND=return] dns mdns4 mdns

Now create a new file /etc/avahi/services/afpd.service and fill it out.

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
  <name replace-wildcards="yes">%h</name>
  <service>
    <type>_afpovertcp._tcp</type>
    <port>548</port>
  </service>
  <service>
    <type>_device-info._tcp</type>
    <port>0</port>
    <txt-record>model=MacPro</txt-record>
  </service>
</service-group>

You can change “MacPro” to one of PowerBook, PowerMac, Macmini, iMac, MacBook, MacBookPro, MacBookAir, Xserve, AppleTV1,1, AirPort in order to change the icon that shows up in Finder.

Now restart Avahi and your Linux AFP server should show up in the Finder’s sidebar!

sudo /etc/init.d/avahi-daemon restart

1. Monit does not honor limits.conf

We have a Solr process with 3 cores. The number of open file descriptors can get pretty big between optimizes, exceeding the default limit of 1024 in Ubuntu. No problem, I’ll just edit /etc/security/limits.conf and reboot! No dice. Looking at /proc/<solr_pid>/limits shows that the changes didn’t take.

At least the solution is easy. Change the Monit start command to execute a bash script instead of starting the Solr process directly. In the bash script, just call ulimit -n 2048 before starting the Solr process.

2. Monit kills your environment

This is actually well known and documented. Again the solution is to have Monit call a bash script and just set the environment there before starting your process.

But if it’s just one or two environments variable you need to set, you can easily do it in the Monit start command using /usr/bin/env .

check process some_process
  start = "/usr/bin/env RAILS_ENV=production DEBUG=false /path/to/some_executable --arg1 foo -arg2 bar
  ...

PHP prior to 5.3.3 doesn’t include PHP-FPM. You have to download the PHP sources, unpack them, then inject the PHP-FPM sources into them via Subversion.

cd $PHP_SRC
svn co http://svn.php.net/repository/php/php-src/trunk/sapi/fpm sapi/fpm
./buildconf --force
./configure --enable-fpm --with-mysql --with-zlib --with-curl --with-openssl --prefix=/usr/local/php-5.3.2 --with-libevent-dir=/usr/local/libevent-1.4.14b-stable/ --with-gd
make && sudo make install

Take a look at the configure line. We can see that we need the dev packages for MySQL, zlib, curl, openssl, and gd. libevent was installed by source.

Now the php-fpm binary should be in /usr/local/php-5.3.2/sbin. It needs two configuration files: php-fpm.conf and php.ini. You can put them anywhere, but I chose to put them in /etc/php.

Ok, we’re ready to start PHP-FPM…

/usr/local/php-5.3.2/sbin/php-fpm -c /etc/php/php.ini -y /etc/php/php-fpm.conf

Assuming you’ve built and installed Nginx, just add a server like this to your configuration file:

server {
  listen              80;
  server_name         www.stochasticbytes.com;
  root                /home/webapps/blog.stochasticbytes.com;
  location / {
    index index.php;
    
    # if file exists return it right away
     if (-f $request_filename) {
       break;
     }
    
     # otherwise rewrite
     if (!-e $request_filename) {
       rewrite ^(.+)$ /index.php$1 last;
       break;
     }
  }
  
  # This matches urls like:
  # /index.php/categories/blog
  location ~ ^/index\.php {
    fastcgi_pass 127.0.0.1:9000;
    fastcgi_param SCRIPT_FILENAME
                  $document_root$fastcgi_script_name;
    fastcgi_param PATH_INFO
                  $fastcgi_script_name;
    include fastcgi_params;
  }
  
  # this matches urls like:
  # /wp-content/plugins/blah/bleck.php
  location ~ \.php$ {
    fastcgi_pass 127.0.0.1:9000;
    fastcgi_param SCRIPT_FILENAME
                  $document_root$fastcgi_script_name;
    fastcgi_param PATH_INFO
                  $fastcgi_script_name;
    include fastcgi_params;
  }
}

Start Nginx and you’re off to races… assuming you have a WordPress installation at that root directive.

Logical Sector Size of USB Stick

Some USB sticks can’t be used for booting (or are difficult to). Gnu’s Parted doesn’t like devices with logical sector sizes of anything but 512. Also, partman (Ubuntu installer’s disk partitioner) will completely hang during installation while scanning for devices if you have a USB storage device plugged in with logical sector size != 512.

UPDATE: My stick has 512 logical sector size, but still hung because it came with a FAT32 partition. Removing the partition fixed the problem.

So I bought a 4 GB Kingston DataTraveler 101. It has a logical sector size of 512 and will show up as a normal hard drive during the Ubuntu installation process.

Getting Started

I like to torrent the Ubuntu installation CD. Bittorrent is pretty damn fast.

Boot your MacBook using the Ubuntu installation CD (hold down ‘c’ while it is booting). Choose “Install Ubuntu” and follow the instructions like normal.

Partitioning

When you get to the disk partitioning part, make sure you partition your USB stick and NOT your MacBook’s hard drive. My USB stick was /dev/sdc and my MacBook’s hard drive was /dev/sda.

I kept it simple and made a single root partition with an ext4 filesystem. Making a swap partition is optional; the idea being that if you omit it, you won’t put as much wear on your USB stick. Discussion on running Linux without swap space can be found here.

Some people think that using a journaling filesystem will prematurely wear a USB stick, but this guy doesn’t think so.

Installing Grub

Towards the end of the installation, it will ask you where to install Grub. Again, be sure to specify your USB stick device and NOT your MacBook’s hard drive. I specified /dev/sdc. Notice that’s the entire device (not just a partition), thus installing Grub on the MBR.

All Done?

Eventually, the installation process will ask you to eject the CD and reboot. So we’re done, right? Oh no no no, it’s just starting to get interesting… =/

Hit ctrl-alt-f2 and drop into a shell…

Fixing Grub

UPDATE: My Dell booted fine without following this step.

When the Ubuntu installer ran, the USB stick partitions get assigned some device ids like hd(1,1). When Grub installed, those are the device ids it wrote to /boot/grub/grub.cfg. When you boot from the USB stick, it will get assigned ids like hd(0,1) and thus Grub will fail because it can’t find the devices specified in /boot/grub/grub.cfg. I’m not quite sure if that’s exactly what happens, but it’s something like that.

Mount your USB stick partitions and edit /boot/grub/grub.cfg. Replace all instances of hd(1 (or whatever number your installation happened to use) with hd(0.

You’re not supposed to edit grub.cfg because it’s the output of a bunch of template files and bash scripts and thus will be overwritten each time you install a new kernel or do anything that calls grub-install (or grub-setup or update-grub or whatever), but honestly I don’t know how to edit the templates properly, so I just have to remember to edit grub.cfg each time it gets overwritten.

Inspiration for this step came from here.

Using tmpfs

Flash memory has a limited number of writes so it’s a good idea to try to minimize them. One way to do that is use tmpfs for directories that are frequently written to.

Append the following to your /etc/fstab file:

tmpfs /tmp       tmpfs defaults,size=64M 0 0
tmpfs /var/crash tmpfs defaults,size=64M 0 0
tmpfs /var/lock  tmpfs defaults,size=64M 0 0
tmpfs /var/log   tmpfs defaults,size=64M 0 0
tmpfs /var/run   tmpfs defaults,size=64M 0 0
tmpfs /var/tmp   tmpfs defaults,size=64M 0 0

One nifty thing about tmpfs is that it won’t actually use memory unless you put files on it, so it’s ok to specify the size option to be something much larger than what you’ll actually use at that mount point.

Fixing Apt/Aptitude

Most programs that write to /var/log are fairly robust: they will create files/directories if they don’t exist (perfect since we’re using tmpfs). Apt is not. It will crash if the directory /var/log/apt doesn’t exist.

I was lazy and just added

mkdir /var/log/apt

to /etc/rc.local instead of going through the rigamarole of using update-rc.d.

Done! … ?

Alright, we can finally finish the Ubuntu installation process, eject the CD, pull out the USB stick and boot another computer with it (my Dell Inspiron). So let’s try that…

Where’s my internet?

For fuck sake. Internet doesn’t work. As far as I can tell, this is what happened…

When we installed Ubuntu on the USB stick, it thought it was installing on my MacBook so it detected all of it’s hardware and wrote network configurations for eth0 where ever the hell it writes those things.

Now after I boot my Dell with the USB stick, I run dmesg | grep eth and see messages like…

udev: renamed network interface eth0 to eth1
udev: renamed network interface eth0 to eth2

Now I don’t really know what this means, but I suppose it has something to do with the hardware that Linux detects at boot conflicts with the network configuration written at installation.

At least the fix is easy, just edit /etc/network/interfaces and append the lines:

auto eth1
iface eth1 inet dhcp

And reboot.

That fixes my ethernet. I’m not sure if subbing eth2 will enable my wireless because I’m not really interested in using wireless.

Done

Ok, I think that’s it. I’ll update this post if I run into more issues.

require 'rubygems'
require 'activesupport'
dt = DateTime.now
puts dt.to_f
# => 1240455468.91318
t = YAML.load(dt.to_yaml)
puts t.class.name
# => Time
puts t.to_f
# => 1240455468.0

Also, if you convert a DateTime to UTC then to a Time, it drops the fractional seconds. Converting a DateTime directly to a Time does NOT drop the fractional seconds, nor does converting a Time to UTC. Weird.

require 'rubygems'
require 'activesupport'
dt = DateTime.now
puts dt.utc.to_f
# => 1240456717.12572
puts dt.to_time.to_f
# => 1240456717.12572
puts dt.utc.to_time.to_f
# => 1240455680.0
puts dt.to_time.utc.to_f
# => 1240456717.12572

As you can see, I’m using ActiveSupport. I don’t really know how much (or if any) that is affecting this weird behavior.

Heh, I don’t really know if “fractional seconds” is proper terminology either.

1. there is a has_many association being eager loaded
2. the query is being limited
3. there is order specified by a named scope

I fixed the bug by making modifications to construct_finder_sql_for_association_limiting. You can apply the fix by including the following code into your Rails project:

module ActiveRecord
  module Associations
    module ClassMethods

      def construct_finder_sql_for_association_limiting(options, join_dependency)
        scope       = scope(:find) || {}
        order       = [options[:order], scope[:order]].compact.join(', ')

        # Only join tables referenced in order or conditions since this is particularly slow on the pre-query.
        tables_from_conditions = conditions_tables(options)
        tables_from_order      = order_tables(options)
        all_tables             = tables_from_conditions + tables_from_order
        distinct_join_associations = all_tables.uniq.map{|table|
          join_dependency.joins_for_table_name(table)
        }.flatten.compact.uniq

        is_distinct = !options[:joins].blank? || include_eager_conditions?(options, tables_from_conditions) || include_eager_order?(options, tables_from_order)
        sql = "SELECT "
        if is_distinct
          sql << connection.distinct("#{connection.quote_table_name table_name}.#{primary_key}", order)
        else
          sql << primary_key
        end
        sql << " FROM #{connection.quote_table_name table_name} "

        if is_distinct
          sql << distinct_join_associations.collect(&:association_join).join
          add_joins!(sql, options, scope)
        end

        add_conditions!(sql, options[:conditions], scope)
        add_group!(sql, options[:group], scope)

        if order and is_distinct
          connection.add_order_by_for_association_limiting!(sql, :order => order)
        else
          add_order!(sql, options[:order], scope)
        end

        add_limit!(sql, options, scope)

        return sanitize_sql(sql)
      end
      
    end
  end
end

If you are curious about what exactly was changed, see this diff pastie.

Please excuse my extremely contrived example.

p = Post.find(...)
if p.contains_porn?
  class << p
    before_save :filter_content
  end
end
p.save # should trigger filter_content

Well it doesn’t work.

After digging through the ActiveSupport source code, I saw that it only looks for callback chains in the object’s class, not metaclass. That’s not too hard to fix…

module ActiveSupport
  module Callbacks
    def run_callbacks(kind, options = {}, &block)
      callback_chain_method = "#{kind}_callback_chain"
      # Meta class inherits Class so we don't have to merge it in 1.9
      if RUBY_VERSION >= '1.9'
        metaclass.send(callback_chain_method).run(self, options, &block)
      else
        callbacks = self.class.send(callback_chain_method) | metaclass.send(callback_chain_method)
        callbacks.run(self, options, &block)
      end
    end
  end
end

Credit to Josh Peek for the Ruby 1.9 fix.

Ok, so that got it working… with one major caveat: you cannot serialize objects that use callbacks. Ruby cannot serialize objects that have metaclasses and the run_callbacks method creates a metaclass whether you use it or not.

That’s fine with me, I don’t really like serializing objects anyways, but apparently it’s a major problem for other people… see here and here.

Josh Peek suggested a solution where we store the callback chains on the objects themselves, so object specific callbacks would be done like this…

p = Post.find(...)
p.before_save :filter_content if p.contains_porn?
p.save # should trigger filter_content

That seems like a very simple, clean and rational solution. I’ll look into coding it up later.