Zero to Smoke Test with Sinatra

The other day I put an app in production with (gasp) no automated tests. I’m careful to say no automated tests, because of course I had been testing it manually throughout the ~12 hours it took me to write the initial version of the app. In lieu of tests, I made sure to log copious amounts of information, which I then collected from my deployed app into Papertrail.

For a single day’s work this was acceptable. But I knew that once I had a version in production, pushing out even the smallest, seemingly benign change would be an extremely dicey proposition without some automated regression tests in place.

I had also deliberately written the app in a deliberately, uh, “un-factored” fashion. Building the app was an exploratory process, and I wanted my design to evolve out of use rather than impose design assumptions up-front.

An element in choosing to write the app this was way was the knowledge that I have the tools available to me to refactor the design incrementally. But in order to do that, I needed tests to assure me that I hadn’t broken anything. Those tests needed to be extremely high-level, so that they could continue running unchanged even as I heavily refactored the app’s internals.

So I set out to write an initial smoke test, one that would put the app through its paces in a “happy path” scenario. Because this was a Sinatra app, I needed to do a fair amount of setup to get this smoke test running. I’d done this before in other Sinatra apps, so a lot of it was a matter of cribbing off of those apps.

Since some of these steps were non-obvious, and because getting over the “first test” hump can be a pretty daunting prospect, I thought I’d document the process.

RSpec

First off, I needed a testing framework, and as a matter of habit I use RSpec. So I added a :testing group to my Gemfile, and added a dependency on the RSpec gem.

Then I ran rspec –init to populate my project with starter spec/spec_helper.rb and .rspec files.

I left these mostly unchanged. My project has a top-level environment.rb file which sets up basic stuff like Bundler and Dotenv. Every test would need this environment set up consistently, so I added a line to the top of spec/spec_helper.rb:

I also made one change to .rspec. By default, RSpec configured this file to enable Ruby’s warnings mode. Sadly, my project includes some gems which are not warningfree. So I removed this line from the file:

The test

It was time to write a test. I like to keep high-level tests in a spec/features directory, so I created it. Inside it, I created the file spec/features/smoke_spec.rb.

I wrote the file incrementally, but I’m going to paste in its final contents here, somewhat elided.

I’ll be going over this code one piece at a time over the course of this post. A few notes to start out with:

First, I’m using the new monkeypatch-free syntax from RSpec 3.0 to declare a test:

Second, in that last line you can see some RSpec metadata: feature:   true . There’s nothing special about the name feature , it’s just a tag that I picked. I’ll be using this later to target some RSpec configuration to only apply to example groups with this tag.

Feature spec helper

You might have noticed that the test does not reference the spec_helper.rb file directly. Instead, it has this:

One of the practices advocated by the RSpec team is to limit the number of dependencies that have to be loaded before every test. One way they advise for doing this is to keep spec/spec_helper.rb file minimal, only requiring libraries in it which are required by every test. If some tests need more than the baseline level of support, they can require a specialized spec helper instead.

I opted to follow this advice, and have my feature specs require a specialized spec/feature_spec_helper.rb. Initially, this file just referenced the main spec helper:

I made various additions to this file as I implemented the test.

Environment variables

The test starts out by fetching some volatile and/or sensitive data from the system environment. I keep configuration info like this in environment variables to make it easy to change; for ease of deploy to Heroku; and to make it easier to set up remote Continuous Integration without committing sensitive information to my Github repo.

I use ENV.fetch to grab all of the variable values my test needs. The test will fail if any of these variables are missing, so I want to know right away if there’s an unset variable. ENV.fetch will raise a KeyError if it doesn’t find the variable, letting me know exactly what var I forgot to add to my .env file or to my CI configuration.

The last variable is expected to be an integer. I don’t just want to ensure that the variable is set; I want to be assured that it is a valid integer as well. So rather than using the lenient #to_i > method, I use the Integer() conversion function. This function will raise an exception if the value it is given can’t be sensibly interpreted as an integer.

Database

Since this is my very first test, I am making no assumptions. Thus, the next stanza of my test does a sanity check that the database tables I’m interested in start out empty. This is important, because if I later check for the existence of some record, that check is meaningless unless I also know that the record didn’t exist at the beginning of the test.

These lines also force me to set up database integration for my tests.

I’m using Sequel as the database layer for this project. In order to make the database connection available to my tests, I add some code to the spec/feature_spec_helper.rb file.

The first line requires a file named db.rb. This file lives at lib/db.rb in my project, and is responsible for setting up a global DB constant which refers to a Sequel database connection.

Next I define a module for database-related test helper methods. It defines a db attribute.

And then I update the RSpec config to include this module into any example tagged with feature: true . I also add a before hook to initialize the db attribute with the value of the global DB constant.

I could avoid all this by referencing the DB constant directly in my tests. But I’m not sure I’m going to keep using a global constant, and I like to be able to override the default DB for individual tests.

Test setup

Next up in my test, I do some setup. The software that is being tested is supposed to add a user to a Github team when they purchase a product. In order to meaningfully test this behavior, I have to first be sure that the test user isn’t in the team to begin with. So I use Octokit to get the scenario started in the right state. Since I’m not certain if an exception will be raised on failure, I verify that the test user is not a team member after making the change.

This setup illustrates a general rule in tests: it’s better to enforce a particular state of affairs before the test than to try to return things back to a “clean slate” state after the test. Tests can often be interrupted in the middle, especially when you’re first writing them. A robust test makes sure everything is where it is expected to be before getting started.

Rack::Test

Now that my setup is done, I need to kick off the test proper. The first step is to simulate a product purchase. To do this, I need to hit a webhook action with some simulated IPN data.

In order to simulate a POST to my Sinatra app, I need to set up Rack::Test. First, I add it to my Gemfile.

Then I bundle install.

Next I require rack/test in the spec/feature_spec_helper.rb. In order to use Rack::Test , I need my Sinatra app to be accessible, so I require that as well. It’s found in a top-level project file called app.rb.

I create a helpers module for Rack::Test-enabled examples.

The app attribute is used by Rack::Test to find the Rack app to be tested.

I add some RSpec config to include and configure this new module in feature tests.

Now I can simulate an authorized IPN POST using Rack::Test helper methods.

OmniAuth

Shortly this test is going to be simulating a user clicking a link in an email. But before that can happen, I need to do a little more setup. The link is going to redirect them to a Github OAuth authentication page. My test can’t automate actions performed on sites outside of the Sinatra app being tested, so I need a way to fake out this authentication step.

Thankfully, I’m using OmniAuth, and OmniAuth has built-in support for faking authentication. First off, I go over to my spec/feature_spec_helper.rb and add this line:

This should be pretty self-explanatory.

Next, in my smoke test I fake up an authentication hash. Normally OmniAuth calls the app back with auth data sourced from the authentication provider. But in this test, it’s just going to call the app back with the auth data that I give it.

I tell OmniAuth to call back with this fake data the next time the app triggers authentication.

EmailSpec and Capybara

I’m almost ready to have the app simulate a user opening their email and answering an invitation. But again, I can’t test systems outside my own, which means I need a way capture emails that the system sends. To do this, I’ll use EmailSpec. EmailSpec in turn depends on Capybara to simulate a human clicking on email links, so I’ll need to add that to the project as well.

I’ll start with Capybara. I add it as a project dependency, adding EmailSpec at the same time while I’m at it.

Then I update the spec/feature_spec_helper.rb file to set up Capybara/RSpec integration. I require the library, tell Capybara what app to use, and include some helper modules into feature example groups.

Now for EmailSpec. I require the library, and include some helper modules into feature example groups. Then I add a before hook to feature specs, resetting the delivery bin. If I neglected to do this, emails from one test might “bleed” over into others.

Notice that I require the pony library before email_spec. EmailSpec monkeypatches the two most common Ruby mail-sending libraries, ActionMailer and Pony, to send their email into a fake test delivery bin instead of to a mail server. It automatically detects which library to patch, based on what is available. I use Pony in my app, so I make sure that the pony library is loaded before email_spec.

I’m finally able to add some lines to the test, simulating a user opening their email and clicking on a link they find there.

Assert

In theory, the user clicking on the link in the welcome email should trigger a series of actions: they should authenticate with Github, which will be simulated as we saw earlier. Then the app will add them to a Github team. Finally, it will send them an email welcoming them to the team.

The rest of the test asserts that this is, in fact, what happens.

First, that the user has been added to a Github team:

And second, that they have a welcome email in their inbox:

DatabaseCleaner

All this works… exactly once. The second time I run the test, it fails at the beginning, where I checked that the database starts out empty.

In order to make sure the database starts out clean every time, I’m going to add DatabaseCleaner to the project. I’ve written about DatabaseCleaner before, so I’m not going to go into detail about this part. But for the record, here’s the configuration.

First, I need to add it to the Gemfile (and then bundle install).

And then I add DatabaseCleaner setup to spec/feature_spec_helper.rb.

Notice that I am specific about which database connection to use:

Files

Here is full addition made to Gemfile :

And here is the completed spec/feature_spec_helper.rb :

As the app grows I will probably begin to split this setup out into multiple files.

Conclusion

And there you have it, a first smoke test for a Sinatra app. This was a lot of setup, but it also accomplishes a lot. It simulates an external API call and a user at a browser; it fakes out both OAuth authentication and sending emails; and it interacts with a real external API.

Since I wrote this test I’ve added VCR to speed things up and keep me from hitting the API too often from, tests. I’ve also set up CircleCI to only push my app to Heroku if this smoke test passes. I can now make changes—little ones, or sweeping design renovations—with confidence, knowing that I haven’t broken the essential functionality of my application.

EDIT: As someone pointed out in the comments, this test is awful by a number of measures of test quality. The name talks about “buying a book”, but there’s no point in the test where a book appears to be purchased. Assertions are mixed in with actions. The test is long. If I had to read this test without any familiarity with the codebase, it would take a while before I fully understood what it was testing.

And yet it is still 1000x better than the tests which preceeded it, which is to say no tests at all. Before, I couldn’t change anything in this app, for fear of screwing up a working steady state. Now I can make changes a fair degree of confidence.

I have a bad habit of agonizing over my tests. I want them to be beautiful and well organized. And at least in some contexts there are some good reasons for wanting these qualities, for reasons I’ll go into in another post. But sometimes the most important thing is to simply get some kind of repeatable, automated test in place. Toward that goal, this test succeeds with flying colors.