The Myths of Tech Interviews

I recently ran across this article on from over a year ago. Here’s the punch line (or at least one of them):

“We looked at tens of thousands of interviews, and everyone who had done the interviews and what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. It’s a complete random mess…”

I expect (hope?) that the results aren’t at all surprising. After almost 20 years at Microsoft and hundreds of interviews, it’s exactly what I expected. Interviews, at best, are a guess at finding a good employee, but often serve the ego of the interviewer more than the needs of the company. The article makes note of that as well.

“On the hiring side, we found that brainteasers are a complete waste of time. How many golf balls can you fit into an airplane? How many gas stations in Manhattan? A complete waste of time. They don’t predict anything. They serve primarily to make the interviewer feel smart.”

I wish more interviewers paid attention to statistics or articles (or their peers) and stopped asking horrible interview questions, and really, really tried to see if they could come up with better approaches.

Why Bother?

So – why do we do interviews if they don’t work? Well, they do work – hopefully at least as a method of making sure you don’t hire a completely incapable person. While it’s hard to predict future performance based on an interview, I think they may be more effective at making sure you don’t hire a complete loser – but even this approach has flaws, as I frequently see managers pass on promising candidates for (perhaps) the wrong reasons out of fear of making a “bad hire”.

I do mostly “as appropriate” interviewer at Microsoft (this is the person on the interview loop who makes the ultimate hire / no hire decision on candidates based on previous interviews and their own questions). For college candidates or industry hires, one of the key questions I’m looking to answer is, “is this person worth investing 12-18 months of salary and benefits to see if they can cut it”. A hire decision is really nothing more than an agreement for a long audition. If I say yes, I’m making a (big) bet that the candidate will figure out how to be valuable within a year or so, and assume they will be “managed out” if not. I don’t know the stats on my hire decisions, but while my heart says I’m great, my head knows that I may be just throwing darts.

What Makes a Good Tech Employee?

If I had a secret formula for what made people successful in tech jobs, I’d share. But here’s what I look for anyway:

  1. Does the candidate like to learn? To me, knowing how to figure out how to do something is way more interesting than knowing how to do it in the first place. In fact, the skills you know today will probably be obsolete in 3-5 years anyway, so you better be able to give me examples about how you love to learn new things.
  2. Plays well with others – (good) software engineering is a collaborative process. I have no desire to hire people who want to sit in their office with the door closed all day while their worried team mates pass flat food under their door. Give me examples of solving problems with others.
  3. Is the candidate smart? By “smart”, I don’t mean you can solve puzzles or write some esoteric algorithm at my white board. I want to know if you can carry on an intelligent conversation and add value. I want to know your opinions and see how you back them up. Do you regurgitate crap from textbooks and twitter, or do you actually form your own ideas and thoughts?
  4. If possible, I’ll work with them on a real problem I’m facing and evaluate a lot of the above simultaneously. It’s a good method that I probably don’t use often enough (but will make a mental note now to do this more).

The above isn’t a perfect list (and leaves off the “can they do the job?” question, but I think someone who can do the above can at least stay employed.


  1. I interviewed more than a hundred candidates and basically agree with your “don’t hire a complete loser” goal.
    I think that we are missing an important piece of statistics though- how many false negative did we have (candidates that we should have hired but didn’t) and that it is as important as the false positives (complete loser we hired).
    I use some kind of variation on the list you mentioned, I found out that a good predictor for learning skills is someone that can show past work and learning and can actually explain it, a lot of the engineers we hired moved from technology to technology more than once over their careers and could explain them all.

  2. Alan and I both come out of Microsoft and I also spent ~10 years there as an As Appropriate interviewer so we’re largely in agreement about interviewing philosophies. I will quibble with the brain teaser example in the quote above because those are not logic puzzles they are modeling questions and are quite valuable in an interview. The key difference is that modeling problems do not have a single correct answer and they do not rely on a specific insight like a logic puzzle does. They provide the candidate with an excellent opportunity to demonstrate creativity and drive for results. For example in the golf ball question above you could provide answers like: calculate the number of golf balls based on the volume of the plane, fill the plane and weigh it, measure its take off velocity filled and empty, measure its fuel consumption filled and empty, measure resonance frequencies in the plane, etc. While many of these answers may presuppose a background in engineering or physics there are other problems that you can ask with less technical answers and provide the same insight into the candidates creativity and drive for results.

    Logic puzzles fall into what I call “revenge of the audio visual club” or “I had to answer this question so I am going to make someone else answer it. They have no predictive value for anything related to any position I ever filled.

    What I would emphasize from Alan’s post as well as the comment above is a focus on what is called “experiential interviewing”. Spend as much time as possible having the candidate talk about what they have done, what was successful, what was a failure, and what they learned from those experiences. You can tell so much about their intelligence, communication skills, commitment, how self aware they are, and their ability to learn from these discussions.

  3. My boss actually kept stats on his interviewing over the years. I think his stats might suggest that he’s better than a dart board, but without a control group, it is impossible to say for sure. In case your interested: Later he enumerated the various changes he has put in place after hiring someone who hasn’t work out or by careful consideration.

    One piece you fail to mention in interviewing is not all interviews are meant to filter out people. Some are to sell a person to join your company, particularly if you already know the person and their qualities. While this doesn’t happen as often it certainly does happen.

    As for smart, I think there is another piece that you probably look for but missed in your description. While one wants to hear opinions, you also want to make sure they are socially smart with their opinions. That isn’t to say they answer what they think you want to hear but rather than they aren’t overly aggressive with a different opinion. I have had candidates curse about questions that were asked in an interview. Defending a view or even ones own mistakes (E.G. Miss-spelled words in a resume) in a reasonable way is its own form of smart.

    In filtering out the the bad people, we can end up with mediocre people, but often I think that is just as much a failure in knowing what level of employee you are looking for and refusing to accept you might have to wait a long time to get a senior level employee. There just aren’t that many senior level testers out there, so if you expect senior level, it will take a while. The real problem is that if you hire for a Junior level testers and get one, are they willing to level up? This is a question of motivation, and while there are some interview questions for motivation, it is rather hard to know WHAT motivates a person much less how motivated they are in a particular example. Even if the candidate might not know. Only doing the job seems to prove that out.

  4. >How many golf balls can you fit into an airplane? How many gas stations in Manhattan?

    While I dislike brainteasers as much as the next person, I beg to differ with the examples given in the NYT article. The skill of Fermi estimation ( is a valuable tool in the software professional’s toolbox during the planning phases for a project, especially for cloud / network services when doing architecture design and capacity planning.

  5. Not to put too fine a point on it, University Admission Interviews were adopted in the 1930’s to keep down the number of Jewish students in the Ivy League universities. Test scores were capturing too many “non-u” people, and not enough “people like us”. Which, I’m afraid, points to the reason why “admission” interviews are stilled popular in industry.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: