O'Reilly logo

Juniper MX Series by Harry Reynolds, Douglas Richard Hanks Jr.

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Switch and Control Board

At the heart of the MX Series is the Switch and Control Board (SCB). It’s the glue that brings everything together. The SCB has three primary functions: switch data between the line cards, control the chassis, and house the routing engine. The SCB is a single-slot card and has a carrier for the routing engine on the front. A SCB contains the following components:

  • An Ethernet switch for chassis management

  • Two switch fabrics

  • Control board (CB) and routing engine state machine for mastership arbitration

  • Routing engine carrier

Switch and control board components

Figure 1-32. Switch and control board components

Depending on the chassis and level of redundancy, the number of SCBs vary. The MX240 and MX480 require two SCBs for 1 + 1 redundancy, whereas the MX960 requires three SCBs for 2 + 1 redundancy.

Ethernet Switch

Each SCB contains a 24-port Gigabit Ethernet switch. This internal switch connects the two routing engines and all of the FPCs together. Each routing engine has two networking cards. The first NIC is connected to the local onboard Ethernet switch, whereas the second NIC is connected to the onboard Ethernet switch on the other SCB. This allows the two routing engines to have internal communication for features such as NSR, NSB, ISSU, and administrative functions such as copying files between the routing engines.

Each Ethernet switch has connectivity to each of the FPCs. This allows for the routing engines to communicate to the Junos microkernel onboard each of the FPCs. A good example would be when a packet needs to be processed by the routing engine. The FPC would need to send the packet across the SCB Ethernet switch and up to the master routing engine. Another good example is when the routing engine modifies the forwarding information base (FIB) and updates all of the PFEs with the new information.

MX-SCB Ethernet switch connectivity

Figure 1-33. MX-SCB Ethernet switch connectivity

It’s possible to view information about the Ethernet switch inside of the SCB. The command show chassis ethernet-switch will show which ports on the Ethernet switch are connected to which devices at a high level.

{master}
dhanks@R1-RE0> show chassis ethernet-switch

Displaying summary for switch 0
Link is good on GE port 1 connected to device: FPC1
  Speed is 1000Mb
  Duplex is full
  Autonegotiate is Enabled
  Flow Control TX is Disabled
  Flow Control RX is Disabled

Link is good on GE port 2 connected to device: FPC2
  Speed is 1000Mb
  Duplex is full
  Autonegotiate is Enabled
  Flow Control TX is Disabled
  Flow Control RX is Disabled

Link is good on GE port 12 connected to device: Other RE
  Speed is 1000Mb
  Duplex is full
  Autonegotiate is Enabled
  Flow Control TX is Disabled
  Flow Control RX is Disabled

Link is good on GE port 13 connected to device: RE-GigE
  Speed is 1000Mb
  Duplex is full
  Autonegotiate is Enabled
  Flow Control TX is Disabled
  Flow Control RX is Disabled
  Receive error count = 012032

The Ethernet switch will only be connected to FPCs that are online and routing engines. As you can see, R1-RE0 is showing that its Ethernet switch is connected to both FPC1 and FPC2. Let’s check the hardware inventory to make sure that this information is correct.

{master}
dhanks@R1-RE0> show chassis fpc
                     Temp  CPU Utilization (%)   Memory    Utilization (%)
Slot State            (C)  Total  Interrupt      DRAM (MB) Heap     Buffer
  0  Empty
  1  Online            35     21          0       2048       12         13
  2  Online            34     22          0       2048       11         16

{master}
dhanks@R1-RE0>

As you can see, FPC1 and FPC2 are both online. This matches the previous output from the show chassis ethernet-switch. Perhaps the astute reader noticed that the Ethernet switch port number is paired with the FPC location. For example, GE port 1 is connected to FPC1 and GE port 2 is connected to FPC2, so on and so forth all the way up to FPC11.

Although each Ethernet switch has 24 ports, only 14 are being used. GE ports 0 through 11 are reserved for FPCs, while GE ports 12 and 13 are reserved for connections to the routing engines.

Table 1-10. MX-SCB Ethernet switch port assignments

GE Port

Description

0

FPC0

1

FPC1

2

FPC2

3

FPC3

4

FPC4

5

FPC5

6

FPC6

7

FPC7

8

FPC8

9

FPC9

10

FPC10

11

FPC11

12

Other Routing Engine

13

Routing Engine GE

Note

One interesting note is that the show chassis ethernet-switch command is relative to where it’s executed. GE port 12 will always be the other routing engine. For example, when the command is executed from re0, the GE port 12 would be connected to re1 and GE port 13 would be connected to re0.

To view more detailed information about a particular GE port on the SCB Ethernet switch, you can use the command show chassis ethernet-switch statistics command. Let’s take a closer look at GE port 13, which is connected to the local routing engine.

{master}
dhanks@R1-RE0> show chassis ethernet-switch statistics 13

Displaying port statistics for switch 0
Statistics for port 13 connected to device RE-GigE:
  TX Packets 64 Octets        29023890
  TX Packets 65-127 Octets    101202929
  TX Packets 128-255 Octets   14534399
  TX Packets 256-511 Octets   239283
  TX Packets 512-1023 Octets  610582
  TX Packets 1024-1518 Octets  1191196
  TX Packets 1519-2047 Octets  0
  TX Packets 2048-4095 Octets  0
  TX Packets 4096-9216 Octets  0
  TX 1519-1522 Good Vlan frms  0
  TX Octets                   146802279
  TX Multicast Packets        4
  TX Broadcast Packets        7676958
  TX Single Collision frames  0
  TX Mult. Collision frames   0
  TX Late Collisions          0
  TX Excessive Collisions     0
  TX Collision frames         0
  TX PAUSEMAC Ctrl Frames     0
  TX MAC ctrl frames          0
  TX Frame deferred Xmns      0
  TX Frame excessive deferl   0
  TX Oversize Packets         0
  TX Jabbers                  0
  TX FCS Error Counter        0
  TX Fragment Counter         0
  TX Byte Counter             2858539809

<output truncated for brevity>

Although the majority of the traffic is communication between the two routing engines, exception traffic is also passed through the Ethernet switch. When an ingress PFE receives a packet that needs additional processing—such as a BGP update or SSH traffic destined to the router—the packet needs to be encapsulated and sent to the routing engine. The same is true if the routing engine is sourcing traffic that needs to be sent out an egress PFE.

Switch Fabric

The switch fabric connects all of the ingress and egress PFEs within the chassis to create a full mesh. Each SCB has two switch fabrics. Depending on the MX chassis, each switch fabric can have either one or two fabric planes.

The MX240 and MX480 support two SCBs for a total of four switch fabrics and eight fabric planes. The MX960 supports three SCBs for a total of six switch fabrics and six fabric planes.

This begs the question, what is a fabric plane? Think of the switch fabric as a fixed unit that can support N connections. When supporting 48 PFEs on the MX960, all of these connections on the switch fabric are completely consumed. Now think about what happens when you apply the same logic to the MX480. Each switch fabric now only has to support 24 PFEs, thus half of the connections aren’t being used. What happens on the MX240 and MX480 is that these unused connections are grouped together and another plane is created so that the unused connections can now be used. The benefit is that the MX240 and MX480 only require a single SCB to provide line rate throughput, thus only require an additional SCB for 1 + 1 SCB redundancy.

Table 1-11. MX-SCB fabric plane scale and redundancy assuming four PFEs per FPC

MX-SCB

MX240

MX480

MX960

PFEs

12

24

48

SCBs

2

2

3

Switch Fabrics

4

4

6

Fabric Planes

8

8

6

Spare Planes

4 (1 + 1 SCB redundancy)

4 (1 + 1 SCB redundancy)

2 (2 + 1 SCB redundancy)

MX240 and MX480 Fabric Planes

Given that the MX240 and MX480 only have to support a fraction of the number of PFEs as the MX960, we’re able to group together the unused connections on the switch fabric and create a second fabric plane per switch fabric. Thus we’re able to have two fabric planes per switch fabric, as shown in Figure 1-34.

Juniper MX240 and MX480 switch fabric planes

Figure 1-34. Juniper MX240 and MX480 switch fabric planes

As you can see, each control board has two switch fabrics: SF0 and SF1. Each switch fabric has two fabric planes. Thus the MX240 and MX480 have eight available fabric planes. This can be verified with the command show chassis fabric plane-location.

{master}
dhanks@R1-RE0> show chassis fabric plane-location
------------Fabric Plane Locations-------------
Plane 0                         Control Board 0
Plane 1                         Control Board 0
Plane 2                         Control Board 0
Plane 3                         Control Board 0
Plane 4                         Control Board 1
Plane 5                         Control Board 1
Plane 6                         Control Board 1
Plane 7                         Control Board 1

{master}
dhanks@R1-RE0>

Because the MX240 and MX480 only support two SCBs, they support 1 + 1 SCB redundancy. By default, SCB0 is in the Online state and processes all of the forwarding. SCB1 is in the Spare state and waits to take over in the event of a SCB failure. This can be illustrated with the command show chassis fabric summary.

{master}
dhanks@R1-RE0> show chassis fabric summary
Plane   State    Uptime
 0      Online   18 hours, 24 minutes, 57 seconds
 1      Online   18 hours, 24 minutes, 52 seconds
 2      Online   18 hours, 24 minutes, 51 seconds
 3      Online   18 hours, 24 minutes, 46 seconds
 4      Spare    18 hours, 24 minutes, 46 seconds
 5      Spare    18 hours, 24 minutes, 41 seconds
 6      Spare    18 hours, 24 minutes, 41 seconds
 7      Spare    18 hours, 24 minutes, 36 seconds

{master}
dhanks@R1-RE0>

As expected, planes 0 to 3 are Online and planes 4 to 7 are Spare. Another useful tool from this command is the Uptime. The Uptime column displays how long the SCB has been up since the last boot. Typically, each SCB will have the same uptime as the system itself, but it’s possible to hot-swap SCBs during a maintenance; the new SCB would then show a smaller uptime than the others.

MX960 Fabric Planes

The MX960 is a different beast because of the PFE scale involved. It has to support twice the number of PFEs as the MX480, while maintaining the same line rate performance requirements. An additional SCB is mandatory to support these new scaling and performance requirements.

Juniper MX960 switch fabric planes

Figure 1-35. Juniper MX960 switch fabric planes

Unlike the MX240 and MX480, the switch fabrics only support a single fabric plane because all available links are required to create a full mesh between all 48 PFEs. Let’s verify this with the command show chassis fabric plane-location.

{master}
dhanks@MX960> show chassis fabric plane-location
------------Fabric Plane Locations-------------
Plane 0                         Control Board 0
Plane 1                         Control Board 0
Plane 2                         Control Board 1
Plane 3                         Control Board 1
Plane 4                         Control Board 2
Plane 5                         Control Board 2

{master}
dhanks@MX960>

As expected, things seem to line up nicely. We see there are two switch fabrics per control board. The MX960 supports up to three SCBs providing 2 + 1 SCB redundancy. At least two SCBs are required for basic line rate forwarding, and the third SCB provides redundancy in case of a SCB failure. Let’s take a look at the command show chassis fabric summary.

{master}
dhanks@MX960> show chassis fabric summary
Plane   State    Uptime
 0      Online   18 hours, 24 minutes, 22 seconds
 1      Online   18 hours, 24 minutes, 17 seconds
 2      Online   18 hours, 24 minutes, 12 seconds
 3      Online   18 hours, 24 minutes, 6 seconds
 4      Spare    18 hours, 24 minutes, 1 second
 5      Spare    18 hours, 23 minutes, 56 seconds

{master}
dhanks@MX960>

Everything looks good. SCB0 and SCB1 are Online, whereas the redundant SCB2 is standing by in the Spare state. If SCB0 or SCB1 fails, SCB2 will immediately transition to the Online state and allow the router to keep forwarding traffic at line rate.

J-Cell

As packets move through the MX from one PFE to another, they need to traverse the switch fabric. Before the packet can be placed onto the switch fabric, it first must be broken into J-cells. A J-cell is a 64-byte fixed-width unit.

Cellification of variable length packets

Figure 1-36. Cellification of variable length packets

The benefit of J-cells is that it’s much easier for the router to process, buffer, and transmit fixed-width data. When dealing with variable length packets with different types of headers, it adds inconsistency to the memory management, buffer slots, and transmission times. The only drawback when segmenting variable data into a fixed-width unit is the waste, referred to as “cell tax.” For example, if the router needed to segment a 65-byte packet, it would require two J-cells: the first J-cell would be fully utilized, the second J-cell would only carry 1 byte, and the other 63 bytes of the J-cell would go unused.

Note

For those of you old enough (or savvy enough) to remember ATM, go ahead and laugh.

J-Cell Format

There are some additional fields in the J-cell to optimize the transmission and processing:

  • Request source and destination address

  • Grant source and destination address

  • Cell type

  • Sequence number

  • Data (64 bytes)

  • Checksum

Each PFE has an address that is used to uniquely identify it within the fabric. When J-cells are transmitted across the fabric a source and destination address is required, much like the IP protocol. The sequence number and cell type aren’t used by the fabric, but instead are important only to the destination PFE. The sequence number is used by the destination PFE to reassemble packets in the correct order. The cell type identifies the cell as one of the following: first, middle, last, or single cell. This information assists in the reassembly and processing of the cell on the destination PFE.

J-Cell Flow

As the packet leaves the ingress PFE, the Trio chipset will segment the packet into J-cells. Each J-cell will be sprayed across all available fabric links. The following illustration represents a MX960 fully loaded with 48 PFEs and 3 SCBs. The example packet flow is from left to right.

Juniper MX960 fabric spray and reordering across the MX-SCB

Figure 1-37. Juniper MX960 fabric spray and reordering across the MX-SCB

J-cells will be sprayed across all available fabric links. Keep in mind that only PLANE0 through PLANE3 are Online, whereas PLANE4 and PLANE5 are Standby.

Request and Grant

Before the J-cell can be transmitted to the destination PFE, it needs to go through a three-step request and grant process:

  1. The source PFE will send a request to the destination PFE.

  2. The destination PFE will respond back to the source PFE with a grant.

  3. The source PFE will transmit the J-cell.

The request and grant process guarantees the delivery of the J-cell through the switch fabric. An added benefit of this mechanism is the ability to quickly discover broken paths within the fabric and provide a method of flow control.

Juniper MX fabric request and grant process

Figure 1-38. Juniper MX fabric request and grant process

As the J-cell is placed into the switch fabric, it’s placed into one of two fabric queues: high or low. In the scenario where there are multiple source PFEs trying to send data to a single destination PFE, it’s going to cause the destination PFE to be oversubscribed. One tool that’s exposed to the network operator is the fabric priority knob in the class of service configuration. When you define a forwarding class, you’re able to set the fabric priority. By setting the fabric priority to high for a specific forwarding class, it will ensure that when a destination PFE, is congested, the high-priority traffic will be delivered. This is covered more in detail in Chapter 5.

MX Switch Control Board

The MX SCB is the first-generation switch fabric for the MX240, MX480, and MX960. This MX SCB was designed to work with the first-generation DPC line cards. As described previously, the MX SCB provides line-rate performance with full redundancy.

The MX240 and MX480 provide 1 + 1 MX SCB redundancy when used with the DPC line cards. The MX960 provides 2 + 1 MX SCB redundancy when used with the DPC line cards.

Each of the fabric planes on the first-generation SCB is able to process 20 Gbps of bandwidth. The MX240 and MX480 use eight fabric planes across two SCBs, whereas the MX960 uses six fabric planes across three SCBs. Because of the fabric plane virtualization, the aggregate fabric bandwidth between the MX240, MX480, and MX960 is different.

Table 1-12. First-Generation SCB bandwidth

Model

SCBs

Switch Fabrics

Fabric Planes

Fabric Bandwidth per Slot

MX240

2

4

8

160 Gbps

MX480

2

4

8

160 Gbps

MX960

3

6

6

120 Gbps

MX SCB and MPC Caveats

The only caveat is that the first-generation MX SCBs are not able to provide line-rate redundancy with some of the new-generation MPC line cards. When the MX SCB is used with the newer MPC line cards, it places additional bandwidth requirements onto the switch fabric. The additional bandwidth requirements come at a cost of oversubscription and a loss of redundancy.

Note

The new-generation Enhanced MX SCB is required to provide line-rate fabric bandwidth with full redundancy for high-density MPC line cards such as the MPC-3D-16x10GE-SFPP.

MX240 and MX480

As described previously, the MX240 and MX480 have a total of eight fabric planes when using two MX SCBs. When the MX SCB and MPCs are being used on the MX240 and MX480, there’s no loss in performance and all MPCs are able to operate at line rate. The only drawback is that all fabric planes are in use and are Online.

Let’s take a look at a MX240 with the first-generation MX SCBs and new-generation MPC line cards.

{master}
dhanks@R1-RE0> show chassis hardware | match FPC
FPC 1            REV 15   750-031088   ZB7956            MPC Type 2 3D Q
FPC 2            REV 25   750-031090   YC5524            MPC Type 2 3D EQ

{master}
dhanks@R1-RE0> show chassis hardware | match SCB
CB 0             REV 03   710-021523   KH6172            MX SCB
CB 1             REV 10   710-021523   ABBM2781          MX SCB

{master}
dhanks@R1-RE0> show chassis fabric summary
Plane   State    Uptime
 0      Online   10 days, 4 hours, 47 minutes, 47 seconds
 1      Online   10 days, 4 hours, 47 minutes, 47 seconds
 2      Online   10 days, 4 hours, 47 minutes, 47 seconds
 3      Online   10 days, 4 hours, 47 minutes, 47 seconds
 4      Online   10 days, 4 hours, 47 minutes, 47 seconds
 5      Online   10 days, 4 hours, 47 minutes, 46 seconds
 6      Online   10 days, 4 hours, 47 minutes, 46 seconds
 7      Online   10 days, 4 hours, 47 minutes, 46 seconds

As we can see, R1 has the first-generation MX SCBs and new-generation MPC2 line cards. In this configuration, all eight fabric planes are Online and processing J-cells.

If a MX SCB fails on a MX240 or MX480 using the new-generation MPC line cards, the router’s performance will degrade gracefully. Losing one of the two MX SCBs would result in a loss of half of the router’s performance.

MX960

In the case of the MX960, it has six fabric planes when using three MX SCBs. When the first-generation MX SCBs are used on a MX960 router, there isn’t enough fabric bandwidth to provide line-rate performance for the MPC-3D-16X10GE-SPFF or MPC3-3D line cards. However, with the MPC1 and MPC2 line cards, there’s enough fabric capacity to operate at line rate, \ except when used with the 4x10G MIC.

Let’s take a look at a MX960 with a first-generation MX SCB and second-generation MPC line cards.

dhanks@MX960> show chassis hardware | match SCB
CB 0             REV 03.6 710-013385   JS9425            MX SCB
CB 1             REV 02.6 710-013385   JP1731            MX SCB
CB 2             REV 05   710-013385   JS9744            MX SCB

dhanks@MX960> show chassis hardware | match FPC
FPC 2            REV 14   750-031088   YH8454            MPC Type 2 3D Q
FPC 5            REV 29   750-031090   YZ6139            MPC Type 2 3D EQ
FPC 7            REV 29   750-031090   YR7174            MPC Type 2 3D EQ

dhanks@MX960> show chassis fabric summary
Plane   State    Uptime
 0      Online   11 hours, 21 minutes, 30 seconds
 1      Online   11 hours, 21 minutes, 29 seconds
 2      Online   11 hours, 21 minutes, 29 seconds
 3      Online   11 hours, 21 minutes, 29 seconds
 4      Online   11 hours, 21 minutes, 28 seconds
 5      Online   11 hours, 21 minutes, 28 seconds

As you can see, the MX960 has three of the first-generation MX SCB cards. There’s also three second-generation MPC line cards. Taking a look at the fabric summary, we can surmise that all six fabric planes are Online. When using high-speed MPCs and MICs, the oversubscription is approximately 4:3 with the first-generation MX SCB. Losing a MX SCB with the new-generation MPC line cards would cause the MX960 to gracefully degrade performance by a third.

Enhanced MX Switch Control Board

The second-generation Enhanced MX Switch Control Board (SCBE) doubles performance from the previous MX SCB. The SBCE was designed to be used specifically with the new-generation MPC line cards to provide full line-rate performance and redundancy without a loss of bandwidth.

Table 1-13. Second-generation SCBE bandwidth

Model

SCBs

Switch Fabrics

Fabric Planes

Fabric Bandwidth Per Slot

MX240

2

4

8

320 Gbps

MX480

2

4

8

320 Gbps

MX960

3

6

6

240 Gbps

MX240 and MX480

When the SCBE is used with the MX240 and MX480, only one SCBE is required for full line-rate performance and redundancy.

Let’s take a look at a MX480 with two SCBEs and 100G MPC3 line cards.

dhanks@paisa> show chassis hardware | match SCB
CB 0             REV 14   750-031391   ZK8231            Enhanced MX SCB
CB 1             REV 14   750-031391   ZK8226            Enhanced MX SCB

dhanks@paisa> show chassis hardware | match FPC
FPC 0            REV 24   750-033205   ZJ6553            MPC Type 3
FPC 1            REV 21   750-033205   ZG5027            MPC Type 3

dhanks@paisa> show chassis fabric summary
Plane   State    Uptime
 0      Online   5 hours, 54 minutes, 51 seconds
 1      Online   5 hours, 54 minutes, 45 seconds
 2      Online   5 hours, 54 minutes, 45 seconds
 3      Online   5 hours, 54 minutes, 40 seconds
 4      Spare    5 hours, 54 minutes, 40 seconds
 5      Spare    5 hours, 54 minutes, 35 seconds
 6      Spare    5 hours, 54 minutes, 35 seconds
 7      Spare    5 hours, 54 minutes, 30 seconds

Much better. You can see that there are two SCBEs as well 100G MPC3 line cards. When taking a look at the fabric summary, we see that all eight fabric planes are present. The big difference is that now four of the planes are Online while the other four are Spare. These new SCBEs are providing line-rate fabric performance as well as 1 + 1 SCB redundancy.

Because the MX SCBE is twice the performance of the previous MX SCB, the MX960 can now go back to the original 2 + 1 SCB for full line-rate performance and redundancy.

MX960

Let’s check out a MX960 using three MX SCBEs and 100G MPC3 line cards.

dhanks@bellingham> show chassis hardware | match SCB
CB 0             REV 10   750-031391   ZB9999            Enhanced MX SCB
CB 1             REV 10   750-031391   ZC0007            Enhanced MX SCB
CB 2             REV 10   750-031391   ZC0001            Enhanced MX SCB

dhanks@bellingham> show chassis hardware | match FPC
FPC 0            REV 14.3.09 750-033205 YY8443           MPC Type 3
FPC 3            REV 12.3.09 750-033205 YR9438           MPC Type 3
FPC 4            REV 27   750-033205   ZL5997            MPC Type 3
FPC 5            REV 27   750-033205   ZL5968            MPC Type 3
FPC 11           REV 12.2.09 750-033205 YW7060           MPC Type 3

dhanks@bellingham> show chassis fabric summary
Plane   State    Uptime
 0      Online   6 hours, 7 minutes, 6 seconds
 1      Online   6 hours, 6 minutes, 57 seconds
 2      Online   6 hours, 6 minutes, 52 seconds
 3      Online   6 hours, 6 minutes, 46 seconds
 4      Spare    6 hours, 6 minutes, 41 seconds
 5      Spare    6 hours, 6 minutes, 36 seconds

What a beautiful sight. We have three MX SCBEs in addition to five 100G MPC3 line cards. As discussed previously, the MX960 has six fabric planes. We can see that four of the fabric planes are Online, whereas the other two are Spare. We now have line-rate fabric performance plus 2 + 1 MX SCBE redundancy.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required