Posted on by & filed under distributed teams, jobs, programming.

If you’re hiring people to write programs, you should work on a program with them.

This advice is nothing new but is too easy to ignore or botch. Switching my interviews to live coding on a real program is the single biggest hiring improvement I’ve ever made. It was helpful enough that I don’t think any of these concerns should stop you from doing the same:

  • Should there be a written test?
  • If so, how tricky should we make it?
  • How many days do they need to complete it?
  • Should they do it on a whiteboard to prove how good they are with markers?

As an engineering team, we’ve worked very hard to develop a culture that rewards programmers who learn to understand the business and our customers, and who make good choices about the right code (and quality) for the problem.

A lot of advice about coding tests comes from companies very different from ours. Instead of working on problems at the bleeding edge of distributed systems or category theory, we work on websites that delight and inform other programmers (and regular folk) with books and videos. That means that most of what I have read from others about interviewing sounds completely contrary to what we do. Instead of favoring a candidate that can plow through a ton of homework, I want someone with enough experience to ask the right questions and challenge shaky assumptions. Instead of picking someone who can write algorithms on the board, I want someone who can use their tools effectively and has great habits.

All this distaste for traditional interviewing methods meant that I was ripe for a change when I sat in on a CTO roundtable at an unconference a few years ago. The theme that emerged was hiring and I was fortunate enough to hear a few very good folks talk openly and honestly about the techniques they used. One shared some very clear lessons about how to do coding interviews and out of that I’ve built my own approach. Here’s the mechanics behind it.

Start with screensharing and time

We’re a distributed team and use chat and Google Hangouts a lot, so I do my coding interviews on a Hangout even when we could meet physically. This may seem like a mistake, but it means that the candidate can actually use their normal setup so they have no excuses. It should also slightly lower the stress level. I work to try to reduce the pressure of the interview both beforehand and during because I’m not interested in seeing how they operate when they’re afraid.

You should plan on quite a bit of time for coding interviews (I schedule two hours). Investing any less time teaches me too little about the person. Since this is so much of my day, we aggressively screen candidates beforehand so the total number of interviews is reasonable.

A two-hour time block also gives both of you time to ask questions or chat in the middle as a break. I typically try to get into the code as quickly as possible at the beginning (after telling them explicitly that I don’t want the interview to be high-stress). Having worked on a little real code together often means they’re more prepared to ask me interesting questions in the middle.

Watch the meta-signals

Most of what I actually watch for during the interview is not the actual code they write. There are rare occasions when someone clearly doesn’t know the language or framework (and these can still be valuable to teach me how to interview better), but most people I see are quite good. What I watch for instead:

  • the fluidity with which they approach the problem
  • the way that they think about solving problems (make them narrate their thinking)
  • their ability to follow the instructions from the welcome email or use potentially new software like Google Hangouts
  • their ability to do basic tasks easily like clone a git repo
  • the tricks they use in their editor without even noticing
  • the amount they bring up the larger context (tests, QA, deployment, performance, etc)

Before starting a round of interviews, I design a realistic problem for them to work on. This means that I am specific in some areas and woefully vague in others (just like a ticket in the real world). I usually try to explain the problem in the same words a ticket would use, to give them an opportunity to ask me questions. These questions are some of the most interesting data points: this is what they will have to do with the whole team if they join.

Use real code and assume something

The advice from the CTO was to take an actual bug from the last week and work on it with the candidate. I think this is a good idea, but had trouble figuring out how to do it in practice. Instead I have taken parts of the real code we work on and extracted it into a separate repo. This means I can share it publicly and don’t have to worry much about setup or libraries on their end.

A mistake I used to make was assuming too little about what the candidate was comfortable with. I’ve now realized that I shouldn’t be shy about assuming they know quite a bit about Django, databases, or testing (for example). All these things were listed on the job posting but I was nervous about filtering out “good” candidates that didn’t have some experience. This was foolish: we wanted people who already knew how to use the tools we work with, so a simplistic test wasn’t helping anyone and wasn’t testing enough about what they would actually do here.

Using real code and a real programming problem from the last few months has three main benefits: the candidate learns something about what they might work on, you actually know the problem and (one) solution, and you both treat it differently because it is an authentic problem. This means that you should update your programming code and default problem every few months, but that is not too much of a burden.

Try to help them take away some benefit

Another critical piece of advice: make sure every single candidate benefits from the coding interview. Even when you’re interviewing someone better than you (and this happens to me quite a bit), there’s still some habit or shortcut or perspective you can give to them to help them be a better developer. They will take this away even if they never take the job and it will reflect better on you both as a person and a company. Focus on moments in the interview when you can actually collaborate rather than making them work for a long time with complete silence from the interviewer.

Ask for the final code

Because these interviews work better with just two people, make sure to ask the candidate for a tarball of the code they wrote once they’re done. That allows you to be specific in talking about their interview with other people on your team.

Supplement with other interviews

Finally (and perhaps obviously), make sure that this coding interview is supplemented by other interviews that teach your team different aspects about the candidate. I’ve found the technical interview works much better when I know the things I don’t have to focus on.

Tags: coding, craft, interviews, programming,

One Response to “Coding while hiring”

  1. 2ks4

    I hope that in your postings for these kinds of jobs, you are very explicit. For example, “You will be required to demonstrate (basic/expert) knowledge of Django while participating in a live coding interview.” vs. simply listing Django under a requirements section. An excellent developer reading the latter style of posting may think, “I’m not familiar with Django, but have experience with similar frameworks and learn quickly so it isn’t a big deal.” Too often job postings are written without any consideration for the time and effort the applicant must put in to the application and interview process. By being explicit you are also being respectful of your applicants’ time.