Monday, 12 November 2012

High Availability - SRX LAG & Dual Fabric Configuration

In this post I expand on the SRX Active/Active HA configuration to include Link Aggregation and Dual Fabric Links. This exercise will build directly on the configuration and concepts from the previous two posts.
High Availability - SRX Active/Passive Configuration
High Availability - SRX Active/Active Configuration

The diagram below outlines the physical connections. Additional interfaces (ge-0/0/10) were utilized on each node for the network. Also a second fabric link was added utilizing ge-0/0/3 on each node. This exercise assumes the specific firewall rules are configured for needed access between nodes. Also this exercise will contain configuration from the previous two posts (such as enabling the chassis cluster, previous interfaces and redundancy groups etc.).

The diagram below outlines the logical connection. The additional links will be used by RETH2, for this reason the link aggregation settings are configured under the RETH interface. The additional fabric ports will be configured under the fab0 and fab1 interfaces.

Link Aggregation Configuration

1. Existing Configuration Recap 
This exercise picks up where the last post (High Availability - SRX Active/Active Configuration) left off. Again, please reference these two post to see details on the previously done configuration.
High Availability - SRX Active/Passive Configuration
High Availability - SRX Active/Active Configuration

2. Add additional ports into RETH interface 
Link aggregation allows multiple ports to appear as one logical link for additional redundancy and bandwidth. A Link Aggregation Group (LAG) can be configured in two modes. Static mode and Dynamic (LACP) mode. In static mode the ports are specified in the configuration and as long as link is up the physical port will participate in the LAG. When using Link Aggregation Control Protocol (LACP) a higher level protocol manages the physical ports in the LAG. The following commands would add additional ports to RETH2 interface and therefore will create a Static LAG on the SRX.
It is also important to mention that interfaces ge-0/0/10 and ge-0/0/12 need to be in a unique LAG on the switch, also ge-5/0/10 and ge-5/0/12 need to be in a separate unique LAG on the switch. This was a little confusing to me at first as the SRX chassis cluster configuration has all four interfaces (ge-0/0/10, ge-0/0/12, ge-5/0/10 and ge-5/0/12) under the same RETH interface. There is obviously some 'under the hood' logic going on to have these four links work with link aggregation and the chassis cluster failover. It is important to remember that a single LAG on the switch for all four ports (ge-0/0/10, ge-0/0/12, ge-5/0/10 and ge-5/0/12) is not a supported configuration in this type of configuration.

3. Optionally configure LACP
To configure link aggregation as a dynamic LAG (using LACP) the commands below are needed in addition to the ones in Step#2. The switch connected to RETH2's interfaces (ge-0/0/10, ge-0/0/12, ge-5/0/10 and ge-5/0/12) will also need to have LACP configured. The switch is configured in LACP active mode and the SRX is configured in passive mode. At least one side has to be configured in active mode, I have also done configurations in the past where both end points were configured in active mode and I have not had any issues because of this (maybe there are reasons not to do this? If so please leave comments). LACP also has two types of timers (slow periodic and fast periodic). Fast periodic is the default setting (when using LACP on the SRX) and sends LACP updates every second. The reason timers are not configured below is because the SRX uses fast periodic (1 second) updates by default. 

4. Configure Interface Monitoring
Just as we did on the other ports we will configure interface monitoring on the new interfaces ge-0/0/10 and ge-5/0/10. If we assign a weight of 255 any single link failure will cause the redundancy group to failover.
Another option would be to assign a new weight to all of the physical interfaces configured under the logical interface RETH2. In this example all of the physical interfaces are assigned a weight of 128. In this example two physical links would need to fail to cause the redundancy group to failover. Both are viable configuration options.

5. Configure Minimum-Links
Optionally minimum-links can also be configured under the RETH. The logic behind minimum links is that you can specify a minimum number of links that must be up for the RETH to be up. In the case of SRX Chassis Cluster minimum links is only monitored on the active node. If using this command it is very important that interface monitoring is configured to fail the redundancy group when minimum links disables the RETH. If these settings do not match traffic can potentially be black holed. If interface monitoring is configured for interfaces ge-0/0/10, ge-0/0/12, ge-5/0/10 and ge-5/0/12 and all interfaces have a weight of 128 (as it is in the previous screenshot) the correct configuration of minimum links would be '1'. If minimum links was set to '2' and any single interface was disconnected then the RETH2 interface would be disabled AND redundancy group would not failover. This would result in an outage for any traffic utilizing RETH2.

The screenshot below outlines the command to configure minimum-links. In this specific case this configuration wouldn't be needed as the default minimum-links is already 1, but this can be used for a syntax reference.

VLAN Tagging/Sub Interface Configuration
Sub interfaces can be used if you want to support more than one VLAN per RETH (and essentially more than one VLAN per port). This is valuable if you have a large number of VLANs to support as you do not need to use a large number of ports. I thought it would make sense to add this topic here as it makes even more sense when thinking about doing link aggregation. To demonstrate this RETH2 will be reconfigured to support VLAN tagging, and an additional network will be created ( The diagram below outlines the physical topology.

The diagram below outlines the logical configuration, and depicts how multiple VLANs and IP addresses are configured under RETH2.

1. Delete the old IP configuration
The following screenshot outlines the commands used to remove the old IP address configuration under the logical interface RETH2.

2. Configure Sub Interfaces
The following screenshot outlines the commands used to configured sub interfaces under logical interface RETH2. It should also be noted that the switch connecting to port ge-0/0/10, ge-0/0/12, ge-5/0/10 and ge-5/0/12 would also need to be reconfigured as 'trunk' ports.

3. Update Logical Interfaces under Security Zones
Logical interfaces are assigned to security zones. In the previous step we deleted logical interface RETH2.0 and created two new logical interfaces; RETH2.2 and RETH2.3. The screenshot below outlines the commands used to update this under the security zones.

Dual Fabric Link Configuration
The fabric link is used by the control plane to serve two main purposes. The first is to sync data plane specific parameters such as session state, this is done by sending messages called Real Time Objects (RTO). The second is as a path for data traffic to flow from one SRX device to another. In the previous two exercises only one fabric link was configured and utilized, however up to two fabric links can be configured. When two links are configured the first link will be used for RTOs while the second will be used for data traffic that needs to traverse from one SRX to another. The order of the ports configured is important and must match the physical cabling. The screenshot below outlines the commands for configuring both fabric links. Interface ge-0/0/2 must be connected to interface ge-5/0/2, and interface ge-0/0/3 must be connected to ge-5/0/3. This seems straight forward but can be easy missed when different ports are used on node0 and node1.

This post will conclude my posts on high availability. I originally was going to do two more posts;
1. Layer 2 Switching across SRX Chassis Cluster (supported on SRX240 and SRX650 running 11.1 or later).
2. SRX Chassis Cluster example of Dual Active/Active ISP. (an overview of Chassis Cluster with multiple routing instances to support active/active dual ISP connections with inbound NAT).

The Layer 2 switching across chassis clusters does not seem very common and it is not listed in the JNCIE-SEC exam blueprint. The active/active dual ISP post might have more relevance under the NAT category. I would be interested to hear back from anyone who would like to see these posts. If there is interest I will make sure to do them.

I would also be interested to hear if a lot of people use the 'mininum-links' command when using link aggregation on RETH interfaces. It seems to me like interface monitoring commands would suffice for any configuration needs.  I have had some good discussion with ,  and  on this topic. I am curious of any type of configuration or situation that would need minimum-links (that I might be overlooking). So feel free to jump in on twitter or leave comments here.


  1. Hi Stefan,

    excellent post, thank you. I found it while doing some research on SRX LAGs on chassis clusters.

    Question for you: I don't quite get the paragraph about configuring minimum-links. I am still trying to figure out how exacty this plays together with the interface-monitor feature.

    Would you care to elaborate a little more about it?


  2. Hi Sascha,

    Thanks for the feedback.

    From my perspective it appears that minimum-links and interface monitoring were not designed with each other in mind. By that I mean min-links was designed to to meet a requirement for LAG configuration (which is not specific to SRX or chassis clustering) and interface monitoring was designed to meet a requirement for Chassis Cluster.

    Minimum-links basically allows you to set a min-link threshold on the LAG, if this threshold is crossed then the logical AE (Aggregate Ethernet) interface is disabled. The problem is that this is all that happens no chassis cluster failover happens or no other logic ties into the chassis cluster failover.

    To summarize, although it is possible I don't see any relevancy to using minimum-links in an SRX Chassis Cluster using LAGs/RETHs. I would stick to only using interface monitoring (or IP monitoring which is supported in newer Junos versions).

    Does that help?


  3. Thanks Stefan,

    so what happens if the minimum-links threshold is reached and the corresponding reth interface goes down. Wouldn't that trigger a failover?

    By the way, I think I read in the release notes for Junos 12.1 (or 11.4, don't quite remember) that you can now put reth interfaces into the interface-monitor config statement.

    Too bad Juniper isn't very clear about how minimum-links and interface-monitor act together (if at all). I wish their documentation was better in that regard.


    1. I don't think the RETH going down alone would trigger a failover as the redundancy group node priority is what triggers the failover. Interface monitoring, for example, will modify the failed nodes priority to ensure the failover happens (once the interface monitoring threshold is reached). I could be overlooking something here but this is how I understand the basic redundancy group failover takes place.

      I have been mostly working with the 11.1 code as it is one of versions listed on the JNCIE-SEC exam blue print so you could be correct regarding using RETHs in interface monitoring configuration. It would be something worth configuring in the lab to see how it behaves.


  4. Hi
    Just interested to know what happens if the layer 2 switch has to shut down (maybe for upgrade). So if we loose reth1, will it mean that clients which have default route to will loose all connectivity?

    1. Hi Pramod,

      In this example basic switches were used to connect various networks. One switch with multiple VLANs could have also been used. You are correct in assuming that if a switch went down for an upgrade that traffic would be impacted.

      This design could be further enhanced by using Juniper's Virtual Chassis technology so that there would also be L2 switch resiliency. Here is a link to a Virtual Chassis document.

      Keep in mind you could also use another switching vendor and any L2 reliency technology they offer (ie. Generic Switch Stacking, Cisco MLAG/VPC or Avaya SMLT)


  5. Great article. It would be helpful to have your thoughts on L2 switching across a cluster since there's not much info on it and for the longest time, SRX did not have such capability when clustered. Thanks


  6. Excellent article.

    Do have a question however. We have a cluster configured and we monitor the interfaces to achieve a failover when the interfaces go down.

    However, show chassis cluster interfaces show some interfaces as won, even though they are up.

    Interface Monitoring:
    Interface Weight Status Redundancy-group
    ge-2/0/23 255 Down 1
    ge-2/0/21 255 Down 1
    ge-2/0/20 255 Down 1
    ge-11/0/20 255 Up 1
    ge-11/0/23 255 Up 1
    ge-11/0/22 255 Up 1
    ge-11/0/21 255 Up 1
    ge-2/0/22 255 Up 1

    saidk@FW-SRX-EHV> show interfaces ge-2/0/23
    Physical interface: ge-2/0/23, Enabled, Physical link is Up
    Interface index: 195, SNMP ifIndex: 542
    Link-level type: Ethernet, MTU: 1518, Link-mode: Full-duplex, Speed: 1000mbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled, Auto-negotiation: Enabled,

    Ever saw this? I wonder what can cause the down status even though the physical link and the RETH it belongs to are up.

  7. Great article with explanation and we want more article like these which you have been post , please post your other

    1. Layer 2 Switching across SRX Chassis Cluster (supported on SRX240 and SRX650 running 11.1 or later).
    2. SRX Chassis Cluster example of Dual Active/Active ISP. (an overview of Chassis Cluster with multiple routing instances to support active/active dual ISP connections with inbound NAT).

    with details diagram understanding like you have done above wonderfull , Could you give more example of traffic flow with Red colour with all possible condition in the scenario then we can put our valuable question with other understandings which we have or may be those are our confusions and will be clear with your explanations.

    You are doing great job, I have never seen any site for juniper on the Internet.