JUNOS Cookbook

Foreword

The early days at Juniper Networks were not for the faint of heart. Joining during the hiring rush of early 1997, I found that the cubes and offices of the small office in Santa Clara, California were already packed with experienced old hands—people whom I knew had been around the block once before and would not be shy of expressing themselves. Everyone had strong views on nearly every aspect of building a router from scratch. If you had the misfortune to sit next to a busy conference room, a good pair of headphones and large CD collection were required to drown out the arguments. Design meetings often became heated, and egos were occasionally bruised. Our friends from previous employers taunted us with predictions of doom.

Despite the arguments, we were all united and driven by one solitary goal: to win the competition to build the best Internet core router available. This was a serious challenge, considering the primary competition was a 300-pound gorilla in the form of Cisco Systems. Beating Cisco would require us to produce a router that tackled the perceived weaknesses in its core router platform. A Juniper Networks core router would have to provide line-rate performance (which, for the M40 router meant forwarding around 40 million packets per second), robust core routing protocols, and stable control software. In short, it had to make customers really want to use it.

The performance requirements meant that the network traffic had to be forwarded entirely in hardware. This was something that had never before been attempted for a core network router. As a result, the hardware design of the M40 looked like science fiction to Juniper recruits who had worked on other networking products. The entire forwarding path of the router was constructed from four Application Specific Integrated Circuits (ASICs), designed entirely by Juniper. These four ASICs (called A, B, C, and D to prevent loose lips from revealing their function) were huge, intricate, and enormously ambitious. A large design team of experienced engineers was assembled to implement the ASICs and partnered with another large verification team to check that the designs were functionally correct. Since Silicon Valley was littered with networking startups that had failed because of silicon design problems, there was enormous pressure on the ASIC teams to get it right first time. We all knew that a failed ASIC would probably sink the company.

Not that there was any less pressure on the software teams. Convincing customers to deploy a brand new—and essentially untried—core router into the very heart of their networks is an enormous task. A new router that crashes, forwards packets erratically, or just basically behaves weirdly won’t make any friends in the network operations team and will find itself unceremoniously removed from the network. The problem is that designing and implementing a core router that works completely reliably is a feat that has defeated many companies. And those were “simple” routers where the packets had been forwarded by software. In contrast, not only did the Juniper router require robust routing protocols that could scale to the largest networks, but it also had to have a robust software infrastructure on the CPU-based control boards that managed the fiendishly complicated packet-forwarding ASICs. Just like the ASIC team, the software team had to get it right the first time.

The JUNOS team started from a basic FreeBSD software base and reworked much of the network software in the kernel. New user daemons were written, and a carrier-grade routing protocol suite was implemented. The routing protocols had to be designed to scale to the largest networks and be robust enough to withstand wild fluctuations in the networks around them, something that the competing routers often struggled with. Thankfully, Juniper had a deep well of routing protocol talent available that could pool its cumulative knowledge to design high-quality routing protocol implementations. Potential customers still had to be convinced that the new protocol implementations would interoperate safely within their existing networks. To allow early evaluation, a fledgling JUNOS system appeared in the form of Olive, which was a standard rackmount PC pretending to be a JUNOS routing engine board. This prototype system was delivered to potential customers to give them a feel for the current state of the system and to allow the routing protocols to be debugged.

Juniper had outgrown the offices it occupied in Santa Clara and moved to Mountain View, just off of Highway 237. We didn’t trust the movers to shift the servers between sites and decided to move all the systems ourselves. At one point, we realized that all of Juniper’s primary software servers were loaded into just one car; paranoia dictated that we split them between two cars just in case something happened on the short drive to the new office. We drove gingerly to the new site once the rush hour had finished and breathed a huge sigh of relief when all the servers powered up again. We also got a surprise bonus when we arrived at the new site. The previous occupants of our new office block had left a huge rat’s nest of network cables in their old data center; they’d obviously decided that it was just too much work to untangle it. However, since money was tight, we refused to throw the huge bundle of cables out and spent the next couple of weeks teasing CAT5 cables out of the jumble during quiet moments. There were enough cables from the bundle to let us completely rewire the first software engineering lab for free.

Throughout 1997 and early 1998, all the Juniper engineering teams worked pretty much flat-out to finish the M40. The engineering labs were seldom quiet, and it was hard to tell the weekends from the weekdays by counting cars in the parking lot. The software teams designed and implemented a truly astonishing amount of code in a very short period of time. FreeBSD kernel extensions were added to provide support for chassis management and new Juniper network interfaces. A clean user interface was designed and implemented to provide a seamless interface to the system and prevent users from having to edit raw configuration files by hand. An entire embedded microkernel was written to manage the packet-forwarding engine boards in the system (a fully-loaded M40 would have nine PFE-related boards), which would allow users to exchange configuration and status messages with the routing engine and each other. Drivers for the embedded microkernel were written to manage the ASICs and to allow the route engine to configure the PFE. The size and complexity of the software required to manage just the various control boards eventually grew to rival the route engine itself.

The real headache for the software team was that the hardware wasn’t available to test with. It can take many months after a system is assembled in the engineering lab to get it to a usable state as a complete system. But Juniper couldn’t afford for us to spend six months in the lab; there just wasn’t enough money or time. The solution was to get extremely creative with test equipment, evaluation boards, and generic PCs before the final hardware was available. All sorts of emulation environments were developed to allow the new routing engine and embedded software to be debugged ahead of the actual hardware. For months, we used a motley collection of machines cobbled together from parts and equipment that emulated the final hardware. We didn’t really have to disguise the lab for external visitors—they wouldn’t have been able to guess that each ratty bundle of machines was a virtual M40.

The payback from this approach was enormous. When the hardware finally arrived, it took just one week in the engineering lab for the first network packets to be forwarded successfully! Considering the complexity of the routing engine and PFE interaction, this was a monumental achievement and meant that we could quickly verify that the hardware worked before shipping the systems to our early test customers in September of 1998.

Designing and implementing the first release of the JUNOS software was an unforgettable time. Although the reader may think I’ve concentrated way too much on the hardware, the JUNOS software is intrinsically the way it is because of the hardware. That it has gone through so many iterations since then, and continues to evolve with the advancement of Juniper routers, is the first item you should learn in this book.

The second thing that you should know is that although creating the JUNOS software really was a team effort, Aviva Garrett had the dubious task of documenting our efforts. In fact, she wrote the first manual. And then, as the manager of Juniper Networks technical publications, she led the effort from Version 1.0 until very recently, somewhere after 7.x. Now she has come back and worked on this marvelous book for an entire year, revisiting everything we once did and everything that has evolved since those early days. JUNOS Cookbook represents a full circle for the JUNOS software suite somehow, looping from those early, midday conference room marathons to today’s ability to route a large portion of the world’s network traffic. Aviva and her team of reviewers and technical experts have broken it all down into bite-size recipes and discussions that make today’s complex array of features seem like that simple, erudite version we created back in 1998. Enjoy it, and cheers.

—Scott Mackie

Former Distinguished Engineer, Juniper Networks February 2006