Menu Sidebar

On the limits of TDD, and the limits of studies of TDD

Greg Wilson writes:

This painstaking study is the latest in a long line to find that test-driven development (TDD) has little or no impact on development time or code quality. Here, the authors repeated an earlier study with a couple of new wrinkles, and then blinded their data before giving it to someone else to analyze to remove any possibility of bias. The result: no significant difference between TDD and iterative test-last (ITL) development.

I think it’s really important to pay attention to studies like this. Which is why I’m so glad that Greg is out there drawing attention to empirical science being done on software engineering.

It’s also important to keep in mind that science is always limited by the questions being asked. In this case, my eye was drawn to the experimental design:

The baseline experiment utilized a single experimental object: the Bowling Scorekeeper (BSK) kata. The task required to participants to implement an API for calculating the score of a player in a bowling game. The development of a GUI was not required. The task was divided into 13 user stories of incremental difficulty, each building on the results of the previous one. An example, in terms of input and expected output, accompanied the description of each user story. An Eclipse project, containing a stub of the expected API signature (51 Java SLOC); also an example JUnit test (9 Java SLOC) was provided together with the task description.

(Emphasis mine.)

A commenter on Greg’s blog already noted that this is an exceptionally tiny example coding problem, and questioned whether results on such a small, easy-to-conceptualize program scale meaningfully to real-world software projects. I think that’s a valid criticism.

But I’m more interested in just how well-defined the problem is.

For me, perhaps the greatest value in practicing Test-Driven Development has always been getting over the blank-page brain-freeze towards the beginning of writing a software component. And how, at the same time, TDD forces me to tightly define the problem before addressing it. My TDD process has always been dominated by these questions:

  1. Am I done yet?
  2. …well, do the tests pass?
  3. …and if they do, do the tests describe a completed solution?

This discipline has done more than any other I’ve tried to keep me focused, to help me whittle down the problem statement to specific essentials, and to avoid superfluous tangents.

In the experimental design quoted above, most of that mental work has already been done.

I also quipped on Twitter:

…which may seem flippant, but I’m actually kinda serious about it. Research, understandably, tends to focus on questions that are easier to ask. But a lot of the most important questions (in my opinion) that need to be asked about software today have to do with difficult-to-measure externalities. Technical debt is one such tough-to-measure externality. But even more difficult, and more vital, to ask are questions like: how much of our developers’ happiness, wellbeing, and calm are we burning to achieve these easily-measured productivity/quality results? What state are we leaving developers in for their next project?

I’m glad research like the study cited above is happening. We need to be mindful of it. But we also need to be aware of the questions that aren’t being asked.

Minimal CSS frameworks for coding demos

On RubyTapas, I occasionally need to demonstrate web application code. I often use little Sinatra apps for this purpose, because it allows me to illustrate a web app programming concept with all the usual features like routing, sessions, and templates without switching between lots of different files.

When I do this, I also like the demo web pages to look modern and attractive. For some reason, default browser HTML styling still looks terrible, so this means turning to a CSS framework.

When I’m preparing a 1-file demo web application, I don’t want to think about asset packaging or selecting modules from an industrial-strength framework like Bootstrap or Foundation. I don’t want to think about whether I also need some JavaScript libraries to make the framework function correctly. I want something that’s just as minimal as Sinatra. Ideally, I should source it as a single CSS file.

Recently, as I was prepping another demo app, I reviewed several “micro CSS frameworks” for this purpose. Then I asked around for suggestions, and took a look over the recommendations I received as well.

Here are some notes. Keep in mind, these are in the context of a very specific and not terribly common use case: an aid to demonstrating web programming techniques. This is not an evaluation of these frameworks from the POV of production use.

This is also from the perspective of a developer, not a designer.

Skeleton: One of the oldest frameworks I’m aware of. Styles are a bit on the “delicate” side, not quite what I look for in a demo. More importantly, it’s a little too minimal. As far as I can tell there are no styles for form field errors or notifications.

Cutestrap: Quite nice looking. I started using this for my last demo app, but then realized that it too was a little more minimal than I was looking for. Like Skeleton, it lacked form error indicators or styles for notifications.

Spectre.css: I was initially pretty enthused about this one. It’s “batteries included”, with rich form styling, “toast”-style notifications, menus, and much more. I like this one because I know I could drop it into any demo app and have styles for any user interaction I might conceivably need to show.

What I don’t like about Spectre is that it requires the kind of markup verbosity I associate with Bootrstrap. For instance, a button might be marked up like so:

Paraphrased: “here’s a button which is a button it’s also a primary button and also a large button“.

This kind of repetition is fine in large apps where the HTML is mostly generated, but in an exhibition it slows down typing and distracts from the purpose of the demo.

Bass.css: Billed as a “low-level” CSS toolkit. Looks to be of more interest to designers, with control over a lot of fiddly tweaks I don’t care about.

Kube: This looks pretty feature-complete, although I had to dig into the (copious) docs to realize this. The markup is pretty sane:

I’m going to keep this one on my shortlist. Although, it almost feels too big for my purposes. Unlike most of the frameworks in this line-up, it also includes a JavaScript component. Maybe something to keep in mind for a “real” app.

Pure.css: This one also potentially falls into the “too much” category. In features, that is; it’s quite svelte in terms of download size. And if you don’t need all of it, you can cut down the size even more by selecting modules individually. Thankfully there’s a default single-file download that includes the whole ball of wax.

The website and particularly the layout examples are gorgeous. This CSS framework seems extremely flexible and powerful.

Also, the examples of form styling are delightfully free of framework-imposed boilerplate DIVs. E.g.:

Bulma: Batteries-included, lovely to look at, comes all in one file. Also uses some quite understandable composable CSS styles:

One thing that’s particularly notable about Bulma is that it is the Flexbox-based grid system that seems to make columns and tiles particularly easy to lay out.

The one drawback I’ve seen is that the examples of form styling seem to contain a lot of framework boilerplate:

These are the mini/micro-frameworks I’ve run across. Are there others I should look at?

Planning for the wrong kind of scale

Recently I wrote at length to my newsletter subscribers about some of the factors that went into my choice to move RubyTapas to a WordPress platform. Here’s a small excerpt from that letter:

Programmers love solving problems. The corollary is that we also love making problems.

When we’re faced with something which is obviously an old and oft-solved problem, our usual tactic for making new problems is to proclaim the existing solutions “too complicated”.

I just need to sell subscriptions, and only show content to subscribers. Simple! I don’t need one of these bloated, over-complicated pre-built systems.

Well, and let them upgrade and downgrade to different plans.

With proration, so they don’t get overcharged when they upgrade.

And dunning, so they don’t just get cut off when their credit card expires.

And I need to enable downloads for some users but not for others.

And I need to provide a way for users in countries with strict tax laws to put their VAT IDs on their receipts.

Which means I need proper receipts.

And I need categories. And tags. And series.

And an easy way to make some episodes into freebies.

And the episodes should show a nicely formatted social card when posted to Twitter or Facebook.

And… and… and…

Some people will gamely power through all of these feature adds and gotchas, one by one. And then one day, they’ll find themselves scaling up, and wanting to hire someone to do copy-editing on 400 archive posts. That exist in the form of Markdown files in a Github repo. And then they have to teach this new person to work within the constraints of a home-grown, cobbled-together system. And how to commit and re-deploy the site after each page is done.

I can easily imagine the first knee-jerk reaction a lot of developers will have to that snippet: are you seriously suggesting we should plan ahead for that scenario? That’s, like, the opposite of agile!

I sympathize, I do. I’m a big believer in doing the Simplest Thing That Could Possibly Work. I try to embrace the idea that You Aren’t Gonna Need It.

But no Simplest Thing is chosen in a vacuum. When Kent Beck, Ward Cunningham, and co. were coming up with those catchphrases, they weren’t choosing to work with raw machine code. Or a pile of transistors. They were doing The Simplest Thing in the context of a Smalltalk development environment. A Smalltalk environment is actually a rather complex thing. It operates at a higher level of abstraction than did most of the programming languages in common use at the time the Agile Manifesto was penned. The Pharo image on my desktop is a fork of a Smalltalk image that has been under constant development for over 30 years.

Yes, your next step should be the simplest thing that could possibly work. But you still have to choose your starting point. The foundation you’re building on.

What foundation is going to best support your likely future growth?

And don’t tell me you don’t think about future growth at all. We all do, even if we try to suppress the impulse and focus on the present.

Usually, when developers and developer-centric organizations plan ahead for growth, we plan for just one axis of growth: capacity. Number of users; number and frequency of requests; size of dataset.

We build with at least half an eye towards technical scaling for capacity. Even if we can’t support ROFLscale right now, we at least have an idea of what we’ll do if we ever have the need to.

But there’s another kind of scaling. One that’s a lot more pressing for most businesses: organizational scaling.

What’s going to happen when you hire someone else to administer the system? Are they going to have to edit enumerations in the code? Or ask a developer to do it for them?

What’s going to happen when you hire someone else to write copy? Are they going to have a workflow that blocks on a developer, or on an Operations person?

Speaking of Ops… what’s going to happen when you give that hat to someone else?

What about marketing? There are a lot of little tweaks your future marketing team is going to want to try out. Website copy. Email copy. Pricing. Pricing for specific types of lead.

They’re going to want to collect survey data using in-app forms correlated with user info. Are they going to have to come to you for every survey form? Or for every form integration?

How much code are you going to have to write to eliminate yourself as a bottleneck for those changes?

How about support? I know, you’re handling support yourself right now. But what happens when you turn that one over to someone new? Are they going to be emailing you every 5 minutes because they have zero visibility into the system? How will you have to modify the system to give them the insight and control they need in order to leave you alone?

As conscientious programmers we think a lot about writing code for the next developer. But how much time do you spend thinking about enabling other roles?

Markdown files in a Github repo are “simple” from the point of view of a developer. They may even be “simple” for a non-programming intern you train yourself. But are they “simple” for a marketing expert who needs to tweak copy and styling at the same time?

I’m going to say something challenging: if you’re not thinking about your code in this context, you’re not taking your business seriously.

I can say this, because looking back I can see ways i wasn’t taking my own business seriously by focusing on what was technically “simple”. And by insisting that if something was accessible and easy for developers, it was the right way to go.

Forget capacity concerns. Statistically speaking, you probably do not have an Instagram-scale unicorn on your hands. But if you experience any business success at all, you will need to scale your organization.

And that doesn’t just mean hiring more hackers.

Luckily, you can start thinking this way right now, even if you’re working solo.

I know, because I do. I’m at a point where I delegate some of my work, but I still don’t have any full-time employees. But I do wear different hats: Content creation. Project management. Business Strategy. Sales. Support. Operations. Marketing. And, oh yeah, development.

I know from experience that if I run into a roadblock while wearing the “marketing” hat, and have to put on my “developer” hat to fix it, I’m going to lose at least four hours of my day. That’s just the way it always seems to go. There’s always something that makes the code change not completely trivial. Often it’s an operational something: for instance, the other day when I realized that in order to push a new version of code, I also had to update my Heroku stack because the one I was using had gone obsolete.

There’s always something.

So these days, I’m a lot more sensitive to things that might force me to switch hats. And, conversely, I’m a lot more attracted to solutions which will enable me to keep a hat on uninterrupted. Even if those solutions are “heavier”. Or more opinionated and constraining in some ways. Or more expensive. Or more “boring”.

My advice, if you’re a tiny startup, a solopreneur, a just-established agency: Believe in your business. Believe in your capacity for growth. Believe there will be dozens of people under your roof (even if it’s a virtual roof) in a few years.

You don’t have to plan in detail for this scenario. That would be wasteful. But start believing that’s where you’re going to end up. And start thinking about your technologies and your processes in that context.

Start wearing different hats deliberately, and seeing how long you can stay in-character on before a technical problem forces you to put your hacker hat back on.

The wonderful thing is, this will make you more effective right now, even if it’s just you running the show. You’ll stop getting sidetracked so often. You’ll identify what parts of your processes vary, and what parts stay the same. As you spend more time removing programmer-you from the equation, you’ll discover early opportunities to delegate, either to part-time help or to external services.

So stop planning for capacity you might need someday, and start planning for the kind of scaling that will happen.

Half-off Exceptional Ruby, Confident Ruby, & more

UPDATE: This sale, along with my birthday, is now over. Thanks to everyone who made my birthday even sweeter by buying books!

Hey there! This is just a quick note to let you know that it’s my birthday, and like I do every year I’m celebrating with a big sale. Any of those products you see on the sidebar of this site are 50% off, today only! Products like:

You can see the whole list here.

Just use the code HAPPY0X24 at checkout to get your discount.

But in the grand tradition of infomercials: THAT’S NOT ALL!

You can also use this coupon code to get the best RubyTapas plan—normally $35/month—for $17.50/month. See the RubyTapas blog post for details.

Thanks for reading Virtuous Code, and thanks for celebrating with me!

Older Posts

Virtuous Code

"The three virtues of a programmer: laziness, impatience, and hubris" — Larry Wall


News, notes, and commentary from Avdi, weekly-ish.

Books and Screencasts

RubyTapas Screencasts

RubyTapas Screencasts

Small plates of gourmet Ruby code.

Confident Ruby

Confident Ruby cover

32 Patterns for joyful coding.

The Making of

Confident Ruby cover

Watch me build an app in Sinatra and Rails

Objects on Rails

Objects on Rails

A developer notebook on applying classic Object-Oriented principles to Ruby on Rails projects.

Exceptional Ruby

Exceptional Ruby

The definitive guide to exceptions and failure handling in Ruby.