MC-LAG Configuration on Asterfusion Enterprise SONiC Distribution Switch

1 Purpose

This article mainly explains the MC-LAG solution and configuration of Asterfusion enterprise SONiC switches.

2 Model and Version

Model:
CX-N series switches
Version:
AsterNOSv3.1

3 MC-LAG introduction

MC-LAG (Multi Chassis Link Aggregation Group) is a mechanism to achieve cross-device link aggregation. By performing cross-device link aggregation between one device and two other devices, ordinary links are retained. It provides the advantages of route aggregation while providing device-level redundancy.

MC-LAG provides a horizontal virtualization technology that virtualizes two physical devices into a single logical device. This virtualized “single device” and its connected uplink or downlink device achieve “one-to-one” link aggregation. As shown below:

The working modes of the device MC-LAG introduced in this article are: control plane active and backup mode, data plane active-active mode, that is:
On the control plane, it is necessary to distinguish between primary equipment and standby equipment, primary links and standby links.
The data plane adopts dual-active mode, and the two devices independently decide to forward the data flow.

4 Asterfusion Enterprise SONiC Distribution switch MC-LAG configuration

Asterfusion switches run the enterprise-level SONiC (AsterNOS) system and ability to flexibly support MC-LAG networking.

4.1 Configure MC-LAG

mclag domain domain-id

session-timeout timeout

# Create an MC-LAG domain and enter the view. Currently, only one domain creation is supported, with a range of 1-4095.
# Timeout period, in seconds, ranges from 3 to 3600, with a default of 15 seconds. The interval for heartbeat detection messages should be less than one-third of the MC-LAG session timeout period. The session timeout period should be a multiple of the heartbeat detection message interval.

Example:

sonic(config)# mclag domain 10
sonic(mclag-domain)# session-timeout 15

4.2 Configure peer-link

vlan vlan-id
interface link-aggregation lag-id
mode static
commit
switchport trunk vlan vlan-id

interface ethernet interface-name
link-aggregation-group lag-id [port-priority port-priority]
startup-delay delay


mclag domain domain-id

peer-link link-aggregation name
commit

# Configure a dedicated VLAN, ranging from 1 to 4095.
# Create an aggregation group,1-9999.
# Configure the aggregation mode as static.
# Commit configuration.
# Specify a dedicated VLAN and add it to the business VLAN.
# enter interface view.
# Add it to the corresponding LAG, with an optional priority setting from 0 to 65535, defaulting to 255.
# Configure interface delay, defaulting to 150 seconds, and recommend setting a delay of 145 seconds on the physical interface where the peer-link resides.
# Enter the mclag domain, and the id is the previously configured value.
# Specify peer-link.
# Commit configuration.

Example:

sonic(config)# vlan 10
sonic(config-vlan-10)# exit
sonic(config)# interface link-aggregation 10
sonic(config-lagif-10)# mode static
sonic(config-lagif-10)# commit
sonic(config-lagif-10)# switchport trunk vlan 10
sonic(config-lagif-10)# exit
sonic(config)# interface ethernet 0/10
sonic(config-if-0/10)# link-aggregation-group 10
sonic(config-if-0/10)# startup-delay 100
sonic(config-if-0/10)# exit
sonic(config)# mclag domain 10
sonic(mclag-domain)# peer-link link-aggregation 10
sonic(mclag-domain)# commit

4.3 Configure the heartbeat detection link

Note: The heartbeat detection link is used to forward MC-LAG control packets. It can be shared with the peer-link or use a separate physical link. When dual-active detection is enabled, the heartbeat detection link must be shared with the peer-link; otherwise, the functionality will be disabled.

4.3.1 Using a separate physical link to configure heartbeat detection 

interface ethernet interface-name
ip address A.B.C.D/M
mclag domain domain-id
peer-address A.B.C.D
local-address A.B.C.D

heartbeat-interval interval

commit

# Enter interface view.
# Configure interface IP address.
# Enter mclag domain view.
# Configure peer IP address for heartbeat detection link.
# Configure local IP address for heartbeat detection link.
# Configure MC-LAG heartbeat detection packet interval in seconds, range 1~60, default is 1 second.
# Commit configuration.

Example:

sonic(config)# interface ethernet 0/11
sonic(config-if-0/11)# ip address 10.0.0.11/24
sonic(config-if-0/11)# exit
sonic(config)# mclag domain 10
sonic(mclag-domain)# peer-address 10.0.0.12
sonic(mclag-domain)# local-address 10.0.0.11
sonic(mclag-domain)# heartbeat-interval 1
sonic(mclag-domain)# commit

4.3.2 Using the peer-link to configure the heartbeat detection link

interface vlan vlan-id
ip address A.B.C.D/M
mclag domain domain-id
peer-address A.B.C.D
local-address A.B.C.D
commit

# Enter peer-link dedicated vlanif configuration view.
# Configure IP address for vlanif.
# Enter mclag domain view.
# Configure peer IP address for heartbeat detection link.
# Configure local IP address for heartbeat detection link.
# Commit configuration.

Example:

sonic(config)# vlan 10
sonic(config-vlan-10)# ip address 10.0.0.11/24
sonic(config- vlan-10)# exit
sonic(config)# mclag domain 10
sonic(mclag-domain)# peer-address 10.0.0.12
sonic(mclag-domain)# local-address 10.0.0.11
sonic(mclag-domain)# commit

4.4 Configure MC-LAG member interfaces

Note: It is recommended to use low-speed interfaces as MC-LAG member interfaces. To enhance system reliability, it is advised to use dynamic aggregation for cross-device aggregation groups and enable short LACP timeout. Both member physical interfaces of the two devices deploying MC-LAG must have identical port IDs for proper aggregation.

vlan vlan-id
interface link-aggregation lag-id

lacp fast-rate
commit
switchport trunk vlan vlan-id
mclag domain domain-id
member lag lag-id

# Create a business VLAN, range 1 ~ 4094.
# Enter LAG interface configuration view and create a downstream aggregation group, range 1 ~ 9999.
# Enable LACP short timeout.
# Commit configuration.
# Add business VLAN to trunk.
# Enter mclag domain view.
# Add MC-LAG member interface.

Example:

sonic(config)# vlan 10
sonic(config-vlan-10)# exit
sonic(config)# interface link-aggregation 10
sonic(config-lagif-10)# lacp fast-rate
sonic(config-lagif-10)# commit
sonic(config-lagif-10)# switchport trunk vlan 10
sonic(config-lagif-10)# exit
sonic(config)# mclag domain 10
sonic(mclag-domain)# member lag 10

4.5 Configure Monitor Link group

Note: It is recommended to configure a Monitor Link group on the primary and secondary devices when deploying MC-LAG. The upstream interface should be configured as ‘uplink’ and the downstream interface as ‘downlink’. Enabling this feature ensures that if the upstream interface goes down, the downstream interface will also go down in response, ensuring rapid convergence in the event of a topology failure.

monitor-link-group group-name [delay-time]


interface ethernet interface-name
monitor-link group-name uplink
interface ethernet interface-name
monitor-link group-name downlink

# Create a monitor-link group. delay-time is an optional parameter that indicates the delay time in seconds before the downlink comes up after the uplink comes up. The default value is 0 seconds.
# Enter interface view.
# Configure the interface as uplink.
# Enter interface view.
# Configure the MC-LAG member interface as downlink.

Example:

sonic(config)# monitor-link-group monitor1
sonic(config)# interface ethernet 0/48
sonic(config-if-0/48)# monitor-link monitor1 uplink
sonic(config-if-0/48)# exit
sonic(config)# interface ethernet 0/0
sonic(config-if-0/0)# monitor-link monitor1 downlink

5 MC-LAG typical configuration case

5.1 Requirements

A server named server1, running critical applications for users, requires stable and reliable network connectivity. Currently, a network is being set up using 4 Asterfusion Sonic switches. Two leaf devices are configured with MC-LAG to connect to server1. The objective is to test whether server1 will lose network connectivity if one of the links is disconnected during the test. During the testing process, data transmission between server1 and server2 will be conducted.

5.2 Topology

5.3 Testing environment

5.3.1 Hardware

5.3.2 Software

5.3.3 Management IP

5.3.4 Device communication IP

5.4 Preparation before testing

Ensure each device is correctly connected according to the topology diagram and verify that the iperf3 testing software is installed correctly.

5.5 Configuration step

Step 1 Configure the IP address and gateway for server1’s bond0 interface

[admin@Server1~]# sudo vim /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
BROADCAST=100.0.10.255
IPADDR=100.0.10.200
NETMASK=255.255.255.0
NETWORK=100.0.10.0
ONBOOT=yes
USERCTL=no
GATEWAY=100.0.10.201
BONDING_OPTS=”miimon=100 mode=4 lacp_rate=fast xmit_hash_policy=layer2+3 fail_over_mac=1″
MASTER=yes
[admin@Server1~]# sudo vim /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=”eth0″
BOOTPROTO=none
ONBOOT=”yes”
MASTER=bond0
SLAVE=yes
HWADDR=”b8:59:9f:42:36:68″
[admin@Server1~]# sudo vim /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=”eth1″
BOOTPROTO=none
ONBOOT=”yes”
MASTER=bond0
SLAVE=yes
HWADDR=”b8:59:9f:42:36:69″
[admin@Server1~]# sudo systemctl restart network

Step 2 Configure the IP address and gateway for server2’s eth0 interface

[admin@Server1~]# sudo ifconfig eth0 100.0.20.200/24 up
[admin@Server1~]# sudo route add default gw 100.0.20.201

Step 3 Config Spine switch IP address

sonic# configure terminal
sonic(config)# interface ethernet 0/0
sonic(config-if-0/0)# ip address 10.0.10.201/24
sonic(config-if-0/0)# interface ethernet 0/4
sonic(config-if-0/4)# ip address 10.0.11.201/24
sonic(config-if-0/4)# interface ethernet 0/8
sonic(config-if-0/8)# ip address 10.0.12.201/24
sonic(config-if-0/8)# exit
sonic(config)# interface loopback 0
sonic(config-loif-0)# ip address 172.16.0.1/32

Step 4 Configure IP addresses and MC-LAG on three Leaf devices

Leaf1

sonic# configure terminal
sonic(config)# interface ethernet 0/48
sonic(config-if-0/48)# ip address 10.0.10.1/24
sonic(config-if-0/48)# interface loopback 0
sonic(config-loif-0)# ip address 172.16.0.2/32
sonic(config-loif-0)# exit
sonic(config)# vlan 10
sonic(config-vlan-10)# exit
sonic(config)# vlan 30
sonic(config-vlan-30)# exit
sonic(config)# interface link-aggregation 10
sonic(config-lagif-10)# exit
sonic(config)# interface ethernet 0/0
sonic(config-if-0/0)# speed 10000
sonic(config-if-0/0)# link-aggregation-group 10
sonic(config-if-0/0)# exit
sonic(config)# interface ethernet 0/4
sonic(config-if-0/4)# speed 10000
sonic(config-if-0/4)# switchport trunk vlan 10
sonic(config-if-0/4)# switchport trunk vlan 30
sonic(config-if-0/4)# exit
sonic(config)# interface link-aggregation 10
sonic(config-lagif-10)# switchport trunk vlan 10
sonic(config-lagif-10)# exit
sonic(config)# vlan 10
sonic(config-vlan-10)# ip address 100.0.10.201/24
sonic(config-vlan-10)# exit
sonic(config)# vlan 30
sonic(config-vlan-30)# ip address 11.0.0.11/24
sonic(config-vlan-10)# exit
sonic(config)# monitor-link-group monitor1
sonic(config)# interface ethernet 0/48
sonic(config-if-0/48)# monitor-link monitor1 uplink
sonic(config-if-0/48)# exit
sonic(config)# interface ethernet 0/0
sonic(config-if-0/0)# monitor-link monitor1 downlink
sonic(config-if-0/0)# exit
sonic(config)# mclag domain 10
sonic(mclag-domain)# peer-link ethernet 0/4
sonic(mclag-domain)# local-address 11.0.0.11
sonic(mclag-domain)# peer-address 11.0.0.12
sonic(mclag-domain)# member lag 10
sonic(mclag-domain)# commit
sonic(mclag-domain)# interface vlan 10
sonic(config-vlan-10)# mac-address 00:11:22:33:44:55
sonic(config-vlan-10)# exit
sonic(config)# write

Leaf2

sonic# configure terminal
sonic(config)# interface ethernet 0/48
sonic(config-if-0/48)# ip address 10.0.11.1/24
sonic(config-if-0/48)# interface loopback 0
sonic(config-loif-0)# ip address 172.16.0.3/32
sonic(config-loif-0)# exit
sonic(config)# vlan 10
sonic(config-vlan-10)# exit
sonic(config)# vlan 30
sonic(config-vlan-30)# exit
sonic(config)# interface link-aggregation 10
sonic(config-lagif-10)# exit
sonic(config)# interface ethernet 0/0
sonic(config-if-0/0)# speed 10000
sonic(config-if-0/0)# link-aggregation-group 10
sonic(config-if-0/0)# exit
sonic(config)# interface ethernet 0/4
sonic(config-if-0/4)# speed 10000
sonic(config-if-0/4)# switchport trunk vlan 10
sonic(config-if-0/4)# switchport trunk vlan 30
sonic(config-if-0/4)# exit
sonic(config)# interface link-aggregation 10
sonic(config-lagif-10)# switchport trunk vlan 10
sonic(config-lagif-10)# exit
sonic(config)# vlan 10
sonic(config-vlan-10)# ip address 100.0.10.201/24
sonic(config-vlan-10)# exit
sonic(config)# vlan 30
sonic(config-vlan-30)# ip address 11.0.0.12/24
sonic(config-vlan-10)# exit
sonic(config)# monitor-link-group monitor1
sonic(config)# interface ethernet 0/48
sonic(config-if-0/48)# monitor-link monitor1 uplink
sonic(config-if-0/48)# exit
sonic(config)# interface ethernet 0/0
sonic(config-if-0/0)# monitor-link monitor1 downlink
sonic(config-if-0/0)# exit
sonic(config)# mclag domain 10
sonic(mclag-domain)# peer-link ethernet 0/4
sonic(mclag-domain)# local-address 11.0.0.12
sonic(mclag-domain)# peer-address 11.0.0.11
sonic(mclag-domain)# member lag 10
sonic(mclag-domain)# commit
sonic(mclag-domain)# interface vlan 10
sonic(config-vlan-10)# mac-address 00:11:22:33:44:55
sonic(config-vlan-10)# exit
sonic(config)# write

Leaf3

sonic# configure terminal
sonic(config)# interface ethernet 0/48
sonic(config-if-0/48)# ip address 10.0.12.1/24
sonic(config-if-0/48)# interface ethernet 0/0
sonic(config-if-0/0)# speed 10000
sonic(config-if-0/0)# ip address 100.0.20.201/24
sonic(config-if-0/0)# interface loopback 0
sonic(config-loif-0)# ip address 172.16.0.4/32
sonic(config-loif-0)# exit
sonic(config)# write

Step 5 Configure BGP on Spine device

sonic# configure terminal
sonic(config)# router bgp 65001
sonic(config-router)# bgp router-id 171.16.0.1
sonic(config-router)# no bgp ebgp-requires-policy
sonic(config-router)# neighbor 172.16.0.2 remote-as 65002
sonic(config-router)# neighbor 172.16.0.3 remote-as 65002
sonic(config-router)# neighbor 172.16.0.4 remote-as 65004
sonic(config-router)# address-family l2vpn evpn
sonic(config-router)# neighbor 172.16.0.2 activate
sonic(config-router)# neighbor 172.16.0.3 activate
sonic(config-router)# neighbor 172.16.0.4 activate
sonic(config-router)# advertise-all-vni
sonic(config)# write
sonic(config)# reload

Step 6 Configure BGP on three Leaf devices

Leaf1

sonic(config)# router bgp 65002
sonic(config-router)# bgp router-id 171.16.0.2
sonic(config-router)# no bgp ebgp-requires-policy
sonic(config-router)# neighbor 172.16.0.1 remote-as 65001
sonic(config-router)# neighbor 172.16.0.3 remote-as 65002
sonic(config-router)# address-family ipv4 unicast
sonic(config-router-af)# network 172.16.0.2/32
sonic(config-router-af)# exit
sonic(config)# router bgp 65002
sonic(config-router)# address-family l2vpn evpn
sonic(config-router)# neighbor 172.16.0.1 activate
sonic(config-router)# neighbor 172.16.0.3 activate
sonic(config-router)# advertise-all-vni
sonic(config)# write
sonic(config)# reload

Leaf2

sonic(config)# router bgp 65002
sonic(config-router)# bgp router-id 171.16.0.3
sonic(config-router)# no bgp ebgp-requires-policy
sonic(config-router)# neighbor 172.16.0.1 remote-as 65001
sonic(config-router)# neighbor 172.16.0.2 remote-as 65002
sonic(config-router)# address-family ipv4 unicast
sonic(config-router-af)# network 172.16.0.3/32
sonic(config-router-af)# exit
sonic(config)# router bgp 65002
sonic(config-router)# address-family l2vpn evpn
sonic(config-router)# neighbor 172.16.0.1 activate
sonic(config-router)# neighbor 172.16.0.2 activate
sonic(config-router)# advertise-all-vni
sonic(config)# write
sonic(config)# reload

Leaf3

sonic(config)# router bgp 65004
sonic(config-router)# bgp router-id 171.16.0.4
sonic(config-router)# no bgp ebgp-requires-policy
sonic(config-router)# neighbor 172.16.0.1 remote-as 65001
sonic(config-router)# address-family ipv4 unicast
sonic(config-router-af)# network 172.16.0.4/32
sonic(config-router-af)# exit
sonic(config)# router bgp 65004
sonic(config-router)# address-family l2vpn evpn
sonic(config-router)# neighbor 172.16.0.1 activate
sonic(config-router)# advertise-all-vni
sonic(config)# write
sonic(config)# reload

Step 7 Use server1 to send traffic to server2 and Check The Traffic

[admin@Server2~]# iperf3 -s
[admin@Server1~]# iperf3 -c 100.0.20.200 -l 20k -b 100G -M 9000 -t 1000

Check Spine device traffic

Step 8 Bring down interface 0/48 on leaf1 and Check The Traffic

6 Conclusion

In this case, after the mc-lag link failure, servers were able to communicate normally with each other. Even after the mc-lag link was restored, servers continued to communicate normally. This demonstrates that the mc-lag network composed of Asterfusion Sonic switches can meet users’ requirements for high network stability and reliability.

Asterfusion CX-N & CX-M SONiC switches as the links below

CX-N series switches
CX-M series switches

If you have more technical questions, feel free to propose a ticket on our
https://help.cloudswit.ch/portal/en/signin