This is a new series where I’ll share some tips I’ve learned over the previous week or two. Mostly, this is going to involve valuable lessons I’ve learned after a mistake or two, or three.
No, the LACP bonds aren’t all right
Have a misbehaving LACP bond? If you want to double check the networking team’s work for your LACP bonds, cat out /proc/net/bonding/<bond_ifname>, and take a look at the Aggregator IDs used on each interface.
If your bond is set up to go to a single switch, or to two switches that are taking part in MLAG, the aggregator IDs should be the same on all interfaces in that bond.
Symptoms include packets arriving on a bond interface only when one of the child interfaces is up. tcpdump shows the packets arriving at the child interface of the bond, but not the bond itself.
Useful article: https://access.redhat.com/solutions/631283
Another, showing when aggregator IDs can be different: https://unix.stackexchange.com/questions/82569/bonds-vs-aggregators
Dude, where’s my asynchronously routed packets?
Got a situation where you have two interfaces on a host, on separate subnets with their own gateways, and packets from the outside world could arrive on either interface? It could be a recipe for asynchronous routing problems. If the packet arrives on interface A, but the reverse path leaves via interface B, the packet will be dropped by default.
Useful article: https://access.redhat.com/solutions/53031
One option, other than weakening the reverse path check, is to use a policy-based route to ensure path symmetry; i.e. that packets leave on the interface they come in on. Create your own routing table, shunt any packets coming from interface A’s IP address into it, then put a interface A’s default gateway into the table:
# create a new iproute2 table echo "200 interfaceA" >> /etc/iproute/rt_tables # 10.10.10.2 is interface A's IP address - anything with a source IP of 10.10.10.2 jumps to the table "interfaceA" # i.e. it doesn't use the main routing table. ip rule add from 10.10.10.2 to all lookup interfaceA_table # set a default gateway of 10.10.10.1 for the table "interfaceA". ip route add default via 10.10.10.1 table interfaceA_table
Add the rule and routes to /etc/sysconfig/network-scripts/rule-<ifname> and route-<ifname>, as they won’t persist over reboots without them.
One last useful article: https://blog.scottlowe.org/2013/05/29/a-quick-introduction-to-linux-policy-routing/