IT'S BEEN OVER A DECADE SINCE THE FIRST WEBSITES REACHED REAL SCALE. We were there then, in those early days, watching our sites growing faster than anyone had seen before or knew how to manage. It was up to us figure out how to keep everything running, to make things happen, to get things done.
While everyone else was at the launch party, we were deep in the bowels of the datacenter racking and stacking the last servers. Then we sat at our desks late into the night, our faces lit with the glow of logfiles and graphs streaming by.
Our experiences were universal: Our software crashed or couldn't scale. The databases crashed and data was corrupted, while every server, disk, and switch failed in ways the manufacturer absolutely, positively said it wouldn't. Hackers attacked—first for fun and then for profit. And just when we got things working again, a new feature would be pushed out, traffic would spike, and everything would break all over again.
In the early days, we used what we could find because we had no budget. Then we grew from mismatched, scavenged machines hidden in closets to megawatt-scale datacenters spanning the globe filled with the cheapest machines we could find.
As we got to scale, we had to deal with the real world and its many dangers. Our datacenters caught fire, flooded, or were ripped apart by hurricanes. Our power failed. Generators didn't kick in—or started and then ran out of fuel—or were taken down when someone hit the Emergency Power Off. Cooling failed. Sprinklers leaked. Fiber was cut by backhoes and squirrels and strange creatures crawling along the seafloor.
Man, machine, and Mother Nature challenged us in every way imaginable and then surprised us in ways we never expected.
We worked from the instant our pagers woke us up or when a friend innocently inquired, "is the site down?" or when the CEO called scared and furious. We were always the first ones to know it was down and the last to leave when it was back up again.
Always.
Every day we got a little smarter, a little wiser, and learned a few more tricks. The scripts we wrote a decade ago have matured into tools and languages of their own, and whole industries have emerged around what we do. The knowledge, experiences, tools, and processes are growing into an art we call Web Operations.
We say that Web Operations is an art, not a science, for a reason. There are no standards, certifications, or formal schooling (at least not yet). What we do takes a long time to learn and longer to master, and everyone at every skill level must find his or her own style. There's no "right way," only what works (for now) and a commitment to doing it even better next time.
The web is changing the way we live and touches every person alive. As more and more people depend on the web, they depend on us.
Web Operations is work that matters.
—Jesse Robbins
The contributors to this book have donated their payments to the 826 Foundation, which helps kids learn to love reading at places like the Superhero Supply Company, the Greenwood Space Travel Supply Company, and the Liberty Street Robot Supply & Repair Shop.
Get Web Operations now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
