Last year at TaskRabbit, we decided to go headlong into this Service Oriented Architecture thing. We ended up with several Rails and other Ruby apps that loosely depended on each other to work. While this was mostly good from a separation of concerns and organizational perspective, we found effective automated testing to be quite difficult.
Specifically, we have one core platform application whose API is the most used. We also allow other apps to have read-only access to its database. Several of the apps are more or less a client to that server. So while the satellite apps have their own test suite, the integration tests can only be correct if they are exercising the core API.
To handle this use case, we created a gem called Offshore. A normal test suite has factories and/or fixture data and it uses database transactions to reset the data for every test. Offshore brings this the SOA world by providing the ability to use another application’s factories from your test suite as well as handling the rollback between tests. Through the actual exercising of the other application as well as a simplified usage pattern, we found that we trusted our integration tests much more that alternative approaches.
What we have are several apps based on use case, all using a core platform. There are web apps for each of the participants in our marketplace: workers, consumers, and businesses. We also have iPhone and Android apps along those same lines.
The apps communicate in a few ways. All of them use the platform’s API synchronously. The web apps use Resque Bus to subscribe and publish asynchronously. We also allow the web apps to read the platform database, but not write.
The APIs and data involved are basically about tasks and users in the system and their various state transitions. For example, our business app signs up a user and post a tasks using the API. The user will then be on the their dashboard seeing a list of tasks in various states, conversations with the workers, payment details, etc.
The other thing we want to test are flows that got the user into those states. For example, can the user post a task, receive some bids, chat back and forth, select a worker, have the worker mark it complete, pay for it, and rate the worker without any problems?
First, there was still the question of what server to use. Should it be the local one or staging one or something else? Second, you kind of had to get it right the first time if there was any state involved in the test. If test signed up a user and then posted a task, for example, and it failed just a bit the first time, then it would fail harder the second time because that database was not reset (and the user existed when it went to sign up). Third, I was concerned that this easily get out of date with our evolving platform. Finally, if anything was read directly from the database, this would not work at all (as the data wouldn’t actually be there).
When using Offshore, we actually run the platform server and the app uses that instead of stubbing. Both have the
offshore gem installed.
Before a test, the app tells the platform that it’s about to start a test. It is then given a token that lets it make other requests to it during the test. It will create objects using factories, make regular API calls, and read/write to the platform database as required. It can do it’s own checks throughout to see if the test should pass. When it’s done, it let’s the platform know.
describe "Integration Test", type: :request do it "should work as expected" do Offshore.test.start(example) user = users(:billy) worker = users(:robby) task = FactoryOffshore.create(:task_posted, :user_id => user.id) offer = FactoryOffshore.create(:offer, :worker_id => worker.id) task.state.should == "opened" visit "/tasks" click_on "Hire!" task.reload.state.should == "assigned" task.worker_id.should == worker.id Offshore.test.stop end end
The point is that this is more or less the same test that you’d write if you were using factories and capybara in a regular test suite.
The server app is the one with the factories and the database that your app needs to work. For Rails, add this to your Gemfile:
group :test do gem 'offshore' end
We also add the rake tasks to your Rakefile
begin require 'offshore/tasks' if defined?(Offshore) rescue end
You might need something like this to your test.rb application config:
Offshore.redis = "localhost:6379" Offshore.enable! if ENV['OFFSHORE'] == 'true'
Then run something like this on the command line
$ rake offshore:startup $ OFFSHORE=true rails s thin -e test -p 6001
In you want it anything but a blank database, you must create a rake task called offshore:seed that creates the test database referenced in the database.yml file in the test environment. Something like this would work:
namespace :offshore do task :preload do ENV['RAILS_ENV'] = "test" end task :setup => :environment desc "seeds the db for offshore gem" task :seed do Rake::Task['db:migrate'].invoke Rake::Task['db:test:prepare'].invoke Rake::Task['db:seed'].invoke end end
:setup steps will be invoked in that order before your
:seed call. They are actually unnecessary here, but shown in case you have something more complex to do.
The client app is the one running the tests.
The same thing in the Gemfile:
group :test do gem 'offshore' end
The Rspec config looks likes this:
config.before(:suite) do Offshore.suite.start(:host => "localhost", :port => 4111) end config.before(:each) do Offshore.test.start(example) end config.after(:each) do Offshore.test.stop end config.after(:suite) do Offshore.suite.stop end
You could also do this based on tags if you didn’t need this behavior in all your tests.
How It Works
rake offshore:startup calls your seed rake worker and then takes a snapshot of the database. We use fixture data and by making a “template” of the database, Offshore is able to copy it into place when needed to create the illusion of “transactional” behavior.
So now we run the server and have it enabled. Requiring the gem required a Railtie that added the
Offshore::Middleware which will respond to the requests that it serves (
Offshore.enable! will tell it to handle the requests. Note you can just add
Offshore::Middleware if you want to use this with Sinatra or other Rack apps.
When it receives the
suite_start command, it sets everything up to run and records who is running. The main things to set up are the database and Redis lock. Offshore uses Redis to make sure only one test is using it’s database at a time, further simulating the notion of transactions.
test_start command acquires the lock and copies the template to be the
real database. If the lock is not acquired, the server will return an error code to the client to say wait. The client will poll until it’s available.
The client can now call
factory_create or real APIs as much as it wants. The changes are made in the real database.
test_stop command releases the lock.
We can go through many tests, calling
start_test each time to reset the database so we get a fresh copy.
At the end, the
suite_stop method notes that this client is no longer running.
Developers can run this locally fairly easily using the instructions above. We have also deployed this to a server that auto-refreshes based on our
master branch. This allows our continuous integration service called Tddium to use Offshore as well. Multiple branches can be building at the same time and it works out because of the locks.
We’ve gained a lot more confidence in the overall performance of our environment by exercising both the server and client app in parts of our test suites. Offshore makes this possible by enabling factories and database “transactions” across apps and threads. There’s plenty more things to make this even better to improve performance, but we thought it was an interesting pattern. Let us know if you find it helpful.
Note that while this is the best approach we could come up with for multiple apps, in our newer project, we chose to have a single app with multiple engines. This was in part to make tricks like that Offshore does unnecessary.