Half-MLAG design

Hello there,

Recently I was providing class to a customer, when at some point we went off the topic to discuss their brand new Extreme-based infrastructure they are about to move to. This doesn't happen much often, as we have very experienced VARs as partners, so I would like to cherish the moment by making note on this. One of the ideas of the deployment was to utilize MLAG on Core-ToR connections.

Just to briefly introduce MLAG to people that might not feel familiar with the name, I would call it a mechanism to provide dual-homing of uplinks from a downstream device (like server, ToR switch or else) to a doubled peer (like ToR, aggregation or core switches) for redundancy with no single point of failure at upstream node level. From a downstream device perspective, you create an aggregated connection upwards, but physically the links are split to both MLAG peers - and those peers cooperate so everything works fine as it was really a single L2 device.

The customer was really kind to show me their Extreme Management Center maps and let get impressed about the size of the infrastructure. A lot of purple switches with X690 stacks for redundant core. However, one of the maps was showing kind of uncommon topology, more similar to a ring rather than up/down and diagonal connections between Top-of-Racks (2 per each rack) and Core nodes as you have when you google for MLAG example diagrams:

I was curious to see if it's just because of disabled LLDP/EDP on those ports (that's how XMC builds a topology map, along with CDP) or perhaps the ports were down... The customer claimed it was made on purpose by the reseller and some Extreme-experienced people were involved. At first I was like "no wai" but then I serious'd. Why was it configured that way? Did they try to create MLAG with both uplinks starting from different ToRs but forgot to stack them or is that some super-unique approach I didn't met before? My self-confidence began to struggle. I didn't wan't to upset the customer and I also didn't wan't to make bad reputation for a reseller, who could have some reason behind that config. It was really confusing to either say "that's fine" or "that's totally wrong".

We have displayed the switches' configurations for better understanding and something uncommon appeared to be there. Please take a look at relevant configuration portions:

ToR-1

create vlan "ISC"
configure vlan ISC tag 1022
create vlan "vlan10"
configure vlan vlan10 tag 10

configure ports 61 display-string ToR-2
configure ports 65 display-string Core-1
configure ports 69 display-string ToR-2

enable sharing 61 grouping 61,69 algorithm address-based L2 lacp
enable sharing 65 grouping 65 algorithm address-based L3_L4 lacp

configure vlan ISC add ports 61 untagged
configure vlan vlan10 add ports 1-10,61,65 tagged
configure vlan ISC ipaddress 1.1.1.1 255.255.255.252

create mlag peer "ToR-2"
configure mlag peer "ToR-2" ipaddress 1.1.1.2 vr VR-Default
enable mlag port 65 peer "ToR-2" id 1

---
ToR-2

create vlan "ISC"
configure vlan ISC tag 1022
create vlan "vlan10"
configure vlan vlan10 tag 10

configure ports 61 display-string ToR-1
configure ports 65 display-string Core-2
configure ports 69 display-string ToR-1

enable sharing 61 grouping 61,69 algorithm address-based L2 lacp
enable sharing 65 grouping 65 algorithm address-based L3_L4 lacp

configure vlan ISC add ports 61 untagged
configure vlan vlan10 add ports 1-10,61,65 tagged
configure vlan ISC ipaddress 1.1.1.2 255.255.255.252

create mlag peer "ToR-1"
configure mlag peer "ToR-1" ipaddress 1.1.1.1 vr VR-Default
enable mlag port 65 peer "ToR-1" id 1

---
Core-1

create vlan "ISC"
configure vlan ISC tag 1022
create vlan "vlan10"
configure vlan vlan10 tag 10

configure ports 1:117 display-string Core-2
configure ports 2:97 display-string ToR-1
configure ports 2:117 display-string Core-2

enable sharing 2:97 grouping 2:97 algorithm address-based L3_L4 lacp
enable sharing 1:117 grouping 1:117,2:117 algorithm address-based L2 lacp

configure vlan vlan10 add ports 1:4,1:96,1:117,2:24 tagged
configure vlan ISC add ports 1:117 untagged
configure vlan ISC ipaddress 1.1.1.1 255.255.255.252

create mlag peer "Core-2"
configure mlag peer "Core-2" ipaddress 1.1.1.2 vr VR-Default
enable mlag port 2:97 peer "Core-2" id 297

---
Core-2

create vlan "ISC"
configure vlan ISC tag 1022
create vlan "vlan10"
configure vlan vlan10 tag 10

configure ports 1:117 display-string Core-1
configure ports 2:97 display-string ToR-2
configure ports 2:117 display-string Core-1

enable sharing 2:97 grouping 2:97 algorithm address-based L3_L4 lacp
enable sharing 1:117 grouping 1:117,2:117 algorithm address-based L2 lacp

configure vlan vlan10 add ports 1:4,1:96,1:117,2:24 tagged
configure vlan ISC add ports 1:117 untagged
configure vlan ISC ipaddress 1.1.1.2 255.255.255.252

create mlag peer "Core-1"
configure mlag peer "Core-1" ipaddress 1.1.1.1 vr VR-Default
enable mlag port 2:97 peer "Core-1" id 297

Can you see that? Both core nodes and both ToR switches are configured the same way! So it looks like the goal was to get two-way MLAG for ultimate redundancy of connections. However, there are no diagonal links like it should be in the most typical two-way MLAG deployment scenario. So how will this work right now? I've decided to spend some time on analysis to better understand will it work anyhow or not at all.

Thanks to a feature called FDB Checkpointing, which is essential to MLAG, you might have this scenario working. What does FDB Checkpointing do? Once a MLAG peer see an incoming packet from a downstream device (or a ToR MLAG cluster as with intended architecture) it will not only store the source MAC address in the FDB due to learning mechanism, but it will also inform its peer so it can also store the same MAC address in its own FDB. So whenever a second peer gets a frame destined to that MAC, it can forward it locally, not necessarily by forwarding it to the switch that has seen the MAC address first. Without FDB checkpointing it looks like there would be only left or right side learning possible, so the connectivity will be heavily disturbed until all available source MAC addresses are learnt. Thankfully, peers do talk and exchange information between themselves with joy. But is that it? Does FDB Checkpointing mean everything is fine and we can leave the design as it is? Definitely no.

Because of no double connections from any node to another tier (like from ToR switch to Core MLAG peers) there is no load sharing mechanism available as only an individual switch (or stack) can utilize LACP load sharing algorithms on egress. Thus, all traffic from/to ToR-1 will go via left uplink only, and all traffic from/to ToR-2 will go via right link only. That makes things pretty clear, you can't benefit MLAG here (which principle is to have downstream device equipped with aggregated link for a bigger pipe, yet terminated to two upstream peers for redundancy). Same applies to Core switches perspective with ToRs as MLAG peers.

Another flaw of such implementation is that your redundancy is limited. Think of that Core-1 to ToR-1 link that can go down. If it went down, all traffic from ToR-1 to any of the Core nodes would have to go through ToR-2 (like in EAPS which is not a best option for DC with such connections IMHO), so ToR-2 would be double-loaded with the traffic. Same applies to Core switches perspective with ToRs as MLAG peers. Of course it depends on the traffic pattern whether it could be painful or not; if you have a lot of east-west traffic between ToRs that might be fine, but otherwise you should be sure of the design... or just fix it for full two-way MLAG redundancy. If you had diagonal links as well, you would benefit both load sharing and not overload one peer in case of a link failure.

I would recommend to utilize different VLANs and IPs for every single ISC connection and having ISC-ports tagged, just for clearance (ISC VLANs are not reachable across switches).

How should full-blown two-way MLAG look like, then? What was missing here for sure, were diagonal connections. More than that, there is always a great opportunity to utilize MLAG also on servers-to-ToR connections if you have more than one ToR switch and wish to keep it that way without stacking. You can refer to EXOS User Guide for details, I'll just put a config example that corrects briefly what was at some distance from optimum here.The image below doesn't contain MLAG or VLAN graphical representation for clarity.

ToR-1

create vlan "ISC"
configure vlan ISC tag 1011
create vlan "vlan10"
configure vlan vlan10 tag 10

configure ports 1 display-string Server-1
configure ports 3 display-string Server-2
configure ports 61 display-string ToR-2
configure ports 65 display-string Core-1
configure ports 66 display-string Core-2
configure ports 69 display-string ToR-2

enable sharing 61 grouping 61,69 algorithm address-based L2 lacp
enable sharing 65 grouping 65,66 algorithm address-based L3_L4 lacp

configure vlan ISC add ports 61 untagged
configure vlan vlan10 add ports 1,3,61,65 tagged
configure vlan ISC ipaddress 1.1.1.11 255.255.255.252

create mlag peer "ToR-2"
configure mlag peer "ToR-2" ipaddress 1.1.1.12 vr VR-Default
enable mlag port 65 peer "ToR-2" id 1
enable mlag port 1 peer "ToR-2" id 101
enable mlag port 3 peer "ToR-2" id 102

---
ToR-2

create vlan "ISC"
configure vlan ISC tag 1011
create vlan "vlan10"
configure vlan vlan10 tag 10

configure ports 1 display-string Server-1
configure ports 3 display-string Server-2
configure ports 61 display-string ToR-1
configure ports 65 display-string Core-2
configure ports 66 display-string Core-1
configure ports 69 display-string ToR-1

enable sharing 61 grouping 61,69 algorithm address-based L2 lacp
enable sharing 65 grouping 65,66 algorithm address-based L3_L4 lacp

configure vlan ISC add ports 61 untagged
configure vlan vlan10 add ports 1,3,61,65 tagged
configure vlan ISC ipaddress 1.1.1.12 255.255.255.252

create mlag peer "ToR-1"
configure mlag peer "ToR-1" ipaddress 1.1.1.11 vr VR-Default
enable mlag port 65 peer "ToR-1" id 1
enable mlag port 1 peer "ToR-1" id 101
enable mlag port 3 peer "ToR-1" id 102

---
Core-1

create vlan "ISC"
configure vlan ISC tag 1022
create vlan "vlan10"
configure vlan vlan10 tag 10

configure ports 1:117 display-string Core-2
configure ports 2:97 display-string ToR-1
configure ports 2:98 display-string ToR-2
configure ports 2:117 display-string Core-2

enable sharing 2:97 grouping 2:97,2:98 algorithm address-based L3_L4 lacp
enable sharing 1:117 grouping 1:117,2:117 algorithm address-based L2 lacp

configure vlan vlan10 add ports 1:4,1:96,1:117,2:24 tagged
configure vlan ISC add ports 1:117 untagged
configure vlan ISC ipaddress 1.1.1.1 255.255.255.252

create mlag peer "Core-2"
configure mlag peer "Core-2" ipaddress 1.1.1.2 vr VR-Default
enable mlag port 2:97 peer "Core-2" id 97

---
Core-2

create vlan "ISC"
configure vlan ISC tag 1022
create vlan "vlan10"
configure vlan vlan10 tag 10

configure ports 1:117 display-string Core-1
configure ports 2:97 display-string ToR-2
configure ports 2:98 display-string ToR-1
configure ports 2:117 display-string Core-1

enable sharing 2:97 grouping 2:97,2:98 algorithm address-based L3_L4 lacp
enable sharing 1:117 grouping 1:117,2:117 algorithm address-based L2 lacp

configure vlan vlan10 add ports 1:4,1:96,1:117,2:24 tagged
configure vlan ISC add ports 1:117 untagged
configure vlan ISC ipaddress 1.1.1.2 255.255.255.252

create mlag peer "Core-1"
configure mlag peer "Core-1" ipaddress 1.1.1.1 vr VR-Default
enable mlag port 2:97 peer "Core-1" id 97

Finally, I let the customer know, that the network is in fact operational, but they have something like two-way MLAG in failover state, or better called half-MLAG as there was nothing to failover from. Such scenario gives you a bit of redundancy, as there are two links between Core and ToR architecture tiers, but MLAG doesn't feed you anything with its power here. You could stack on both tiers and aggregate the links or do EAPS or MSTP and the result would be more-less the same. But I wouldn't call that deployment a good practice MLAG example and I ask you the same. Or perhaps you disagree? Please let me know in the comments. Thanks!