Juniper EX VC, HA and QinQ.



The scenario.

A customer needed to connect their main office to the datacenter. They wanted everything to be redundant. They also wanted to be able to create their own vlans between their main office and their rackspace without issuing a change.
Since their main office was pretty close to the datacenter, we proposed to place two EX4200 virtual-chassis. One VC in the datacenter and one VC in their main office.
These VC’s were then connected together via two dark fibers. The dark fibers each followed a different physical route.
On the customer side, we provided a Q-in-Q tunnel over an aggregated Ethernet interface. Through the use of this Q-in-Q tunnel, they were able to create their own vlans without intervention from our side.
Besides this advantage, there is another advantage for us. Future services can be delivered across the solution in vlans outside the Q-in-Q tunnel. We can deliver our standard services in vlans of our choosing without consulting the customer. For example, we know that they are also going to connect other remote office to their main office and they could eventually be interested in even more services.
When connecting other offices to the main office, we can simply extend these vlan across their switches without getting into a conflict on what vlan to use, making life easier for us.
As far as the other services are concerned, we can keep our solution standard and we will not have to ‘design’ something when they request a new service. We simply extend another vlan of our choosing, or even another Q-in-Q tunnel, to deliver that particular service.


scenario
This scenario offered connectivity without any single-point of failure and flexibility for the customer since they could create their own vlans etc.

Virtual chassis technology.

Juniper offers the virtual chassis technology. By using this technology, the two switches that are deployed on a location, will turn into one logical switch. Some of the advantages of using a VC are; greater availability, reduced opex, better performance and scalability and lower latency. Only the first two are of real interest in our scenario.

The other advantage in this scenario is that the dark-fibers (which are quite expensive) can be combined to 1 logical link (an AE interface) distributed over 2 switches. When there are no errors, the customer will have 2Gbit of throughput without relying on 1 piece of hardware and without using some form of spanning-tree.

When you deploy EX4200’s as a VC, you are creating one logical device. This logical device can consist of 10 switches. If you need a lot of 10Gbit ports, you could even mix the EX4200 with the EX4500/EX4550 model.

In a VC, each device can have 1 of 3 roles:
- Master role: runs Junos in master role, manages the member switches, runs the management processes and the control protocols. This switch will represent the VC to the rest of the network.
- Backup role: this switch can assume the master role when the master fails. Junos is run in a backup role.
- Linecard role: runs only a subset of Junos.

In our scenario we only have two devices. This means the two devices will be master and backup; there will be no linecard-switches. Since we want to deterministically control the role of each switch and we want to assign the member-number, we will make use of the pre-provisioned configuration option.

Since our VC’s will consist of only 2 members, we will include ‘no-split-detection’ in the configuration of the VC. When a device fails, the other RE (master or backup) could interpret this as a VC-split. As a result, the RE can put itself in an inactive state. When the RE was functioning as a backup, damages may be limited. But when the RE was functioning as a master, you will be left with nothing.
The configuration snippet to configure the switches in a pre-provisioned manner:

set virtual-chassis preprovisioned
set virtual-chassis no-split-detection
set virtual-chassis member 0 role routing-engine
set virtual-chassis member 0 serial-number GX0223654487
set virtual-chassis member 1 role routing-engine
set virtual-chassis member 1 serial-number GX0256987745



High availability for the virtual chassis.

Junos offers several high-availability knobs to ensure operational continuity during the downtime of a component.

In our scenario will use GRES, NSR and NSB.



GRES:

scenario

When 2 switches form a virtual-chassis, there is a master and a backup switch. When the master switch fails, and GRES is not enabled, several things will happen on the backup switch before it becomes active:

- PFE is restarted by the new master
- all hardware and interfaces are discovered and rpd is restarted

GRES ensures a higher degree of operation continuity when there is a hardware failure. When GRES is enabled, the backup switch will run an additional process called ‘ksyncd’. This process synchronizes the backup RE with the master RE and preserves interface and kernel information.

When GRES is enabled and the master switch fails, the PFE disconnects from the old master RE and reconnect to the new master. This happens after a dead interval of 2 seconds. The switchover-time is greatly reduced through GRES. There will be packet loss however.

GRES is configured with the following two commands:

set system commit synchronize (ensures that a commit is executed on a backup RE as well)
set chassis redundancy graceful-switchover

In order to further minimize packet loss, NSR and NSB need to be configured on the backup RE.



NSR and NSB:

NSR and NSB are extensions to GRES. Where GRES makes sure that the PFE is continuously running and that interface and kernel information is preserved, it will not start the ‘ethernet-switching’ or ‘routing’ subsystem on the backup switch. This means that the forwarding table needs to be repopulated with information when the backup assumes the role of master. To counter this, you can activate NSR and NSB.

NSR will start the rpd on the backup RE and save routing protocol information on the backup RE. NSB will achieve similar results on layer 2. The backup RE will start running the eswd process, which is the Ethernet switching process.


Example output without NSB;

scenario


With NSB and NSR activated:

scenario


What you can see here is that with the functionality of NSB and NSR activated, the backup RE;
- is running the ‘eswd’ and ‘rpd’ in addition to ‘ksyncd’ process for GRES
- has the FT populated with MAC addresses and static routes

NSR and NSB require these two commands:

set ethernet-switching-options nonstop-bridging
set routing-options nonstop-routing


The AE interface:

The AE-interface can be configured across two different hardware switches on the virtual chassis. In this scenario, building an AE interface between the two VC’s offers us the following benefits:
- more and more efficient throughput and physical link redundancy
- a loop free redundant path without the use of spanning-tree
- a redundant path with a link on both members

By using the ae interface in this setup, on both the WAN-link and the customer interface, an entire switch can fail and the customer would still have (albeit half the bandwidth) connectivity. The configuration (on both VC’s) is very straightforward. We will configure a customer port (the ingress side of the Q-in-Q tunnel) and a WAN port (a trunk with additional vlans).

The customer port:

set interfaces ae0 description q_in_q_customer_side
set interfaces ae0 aggregated-ether-options link-speed 1g
set interfaces ae0 mtu 9000
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 unit 0 family ethernet-switching port-mode access
set interfaces ae0 unit 0 family ethernet-switching vlan members 4000

The WAN port:

set interfaces ae10 description wan_port
set interfaces ae10 aggregated-ether-options link-speed 1g
set interfaces ae10 mtu 9000
set interfaces ae10 aggregated-ether-options lacp active
set interfaces ae10 unit 0 family ethernet-switching port-mode trunk
set interfaces ae10 unit 0 family ethernet-switching vlan members 10
set interfaces ae10 unit 0 family ethernet-switching vlan members 4000

Last thing is to configure the VC to be able to handle two AE-interfaces;

set chassis aggregated-devices ethernet device-count 1



The Q-in-Q vlan:

A lot of customers like having a layer 2 link What’s even better is a layer 2 link on which they can send their own vlans. In this case, two dark fibers are used, so there are no restrictions (sometimes, carriers we use restrict us from sending tags across the wan).

By deploying dot1q-tunneling, we can enable the customer to be able to create an own set of vlans. This was something they wanted because they were, amongst others, doing VMware which requires a layer 2 vlan end-to-end for certain features (vMotion for example). A non-technical reason was that they did not want to issue a change every time they created a vlan.

The dot1q-tunneling is nothing more than a ‘tunnel’ carrying frames with the vlan-id that represents the q-in-q vlan and additional vlans, created by the customer in our case.

scenario

Since the EX4200 switch supports q-in-q vlans we enabled it for them. We used vlan-id 4000 as the q-in-q vlan. We enabled it to tunnel all possible vlans. In order to be more transparent to the customer, we also turned on the layer2-protocol-tunneling knob.
With this statement, the EX-switch will transport certain layer 2 protocols across the tunnel. In this customers case, it meant that they would run lldp between their switches and that they could tunnel their stp-bpdu’s (per their own request…).

For the configuration of our scenario, we’ll divide everything up in 3 steps:

1. configuring the q-in-q vlan

set ethernet-switching-options dot1q-tunneling ether-type 0x8100
set vlans q-in-q vlan-id 4000
set vlans q-in-q dot1q-tunneling customer-vlans 1-4094
set vlans q-in-q dot1q-tunneling layer2-protocol-tunneling all

2. configuring the customer side of the q-in-q tunnel
In our case, we have configured an ae interface towards the customer. To enable the customer to send his own vlans across the dot1q-tunnel, the port needs to be configured as an access-port:
set interfaces ae0 unit 0 family ethernet-switching port-mode access
set interfaces ae0 unit 0 family ethernet-switching vlan members 4000

3. configuring the transit link that carries the q-in-q vlan
The link between the two VC’s will carry frames in the q-in-q vlan with two tags, tag 4000 and the vlans the customer wants to use.
The interface will be configured as a trunk in order to be able to create additional vlans for other services:

set interfaces ae10 unit 0 family ethernet-switching port-mode trunk
set interfaces ae10 unit 0 family ethernet-switching vlan members 4000


The complete configuration used in our example:

set system commit synchronize
set chassis redundancy graceful-switchover
set chassis aggregated-devices ethernet device-count 1

set ethernet-switching-options nonstop-bridging
set ethernet-switching-options dot1q-tunneling ether-type 0x8100

set routing-options nonstop-routing

set interfaces ae0 description q_in_q_customer_side
set interfaces ae0 aggregated-ether-options link-speed 1g
set interfaces ae0 mtu 9000
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 unit 0 family ethernet-switching port-mode access
set interfaces ae0 unit 0 family ethernet-switching vlan members 4000

set interfaces ae10 description wan_port
set interfaces ae10 aggregated-ether-options link-speed 1g
set interfaces ae10 mtu 9000
set interfaces ae10 aggregated-ether-options lacp active
set interfaces ae10 unit 0 family ethernet-switching port-mode trunk
set interfaces ae10 unit 0 family ethernet-switching vlan members 10
set interfaces ae10 unit 0 family ethernet-switching vlan members 4000

set vlans q-in-q vlan-id 4000
set vlans q-in-q dot1q-tunneling customer-vlans 1-4094
set vlans q-in-q dot1q-tunneling layer2-protocol-tunneling all

set virtual-chassis preprovisioned
set virtual-chassis no-split-detection
set virtual-chassis member 0 role routing-engine
set virtual-chassis member 0 serial-number GX0223654487
set virtual-chassis member 1 role routing-engine
set virtual-chassis member 1 serial-number GX0256987745

4-8-2014