Beyond Basic Statistics: Tips, Tricks, and Techniques Every Data Analyst Should Know

9MODELS, MURPHY’S LAW, AND PUBLIC HUMILIATION: REGRESSION RULES TO LIVE BY

I’m often reminded that I’m human. Sometimes, this reminder comes to me in the form of small mishaps—coffee spilled on a white carpet, an insensitive remark made to a friend, spinach stuck between my teeth—things I can correct with little embarrassment to myself. All too often, however, this reminder isn’t nearly so gentle. Sometimes it hits me like a brick to the forehead. Like when I’m standing in front of a room full of important people, declaring the brilliance of my statistical conclusions to anyone who will listen.

My first data analysis humiliation occurred when I was only a few weeks into my career as a research statistician. I was working for a biomedical research company, call it MajorMedicCorp. This company developed new devices for monitoring a patient’s blood chemistry. This company wanted to measure certain blood toxins noninvasively, meaning without drawing any blood. We knew we could measure how much life-giving oxygen was in the blood, simply by shining light through a patient’s finger and measuring what came out. My task was to use similar light measurements to predict the amount of this toxic substance.

I was fresh out of school with a Ph.D. and lots of fancy statistical techniques at my fingertips, and this was a linear regression problem. Linear regression is undergraduate stuff. By all accounts, this task should’ve been a breeze for me. Yet somehow, I still managed to turn this ...