Tips from the Trenches, part 1

This is a new series where I’ll share some tips I’ve learned over the previous week or two. Mostly, this is going to involve valuable lessons I’ve learned after a mistake or two, or three.

No, the LACP bonds aren’t all right

Have a misbehaving LACP bond? If you want to double check the networking team’s work for your LACP bonds, cat out /proc/net/bonding/<bond_ifname>, and take a look at the Aggregator IDs used on each interface.

If your bond is set up to go to a single switch, or to two switches that are taking part in MLAG, the aggregator IDs should be the same on all interfaces in that bond.

Symptoms include packets arriving on a bond interface only when one of the child interfaces is up. tcpdump shows the packets arriving at the child interface of the bond, but not the bond itself.

Useful article: https://access.redhat.com/solutions/631283

Another, showing when aggregator IDs can be different: https://unix.stackexchange.com/questions/82569/bonds-vs-aggregators

Dude, where’s my asynchronously routed packets?

Got a situation where you have two interfaces on a host, on separate subnets with their own gateways, and packets from the outside world could arrive on either interface? It could be a recipe for asynchronous routing problems. If the packet arrives on interface A, but the reverse path leaves via interface B, the packet will be dropped by default.

Useful article: https://access.redhat.com/solutions/53031

Another: https://access.redhat.com/discussions/1186373

One option, other than weakening the reverse path check, is to use a policy-based route to ensure path symmetry; i.e. that packets leave on the interface they come in on. Create your own routing table, shunt any packets coming from interface A’s IP address into it, then put a interface A’s default gateway into the table:

# create a new iproute2 table
echo "200 interfaceA" >> /etc/iproute/rt_tables

# 10.10.10.2 is interface A's IP address - anything with a source IP of 10.10.10.2 jumps to the table "interfaceA"
# i.e. it doesn't use the main routing table.
ip rule add from 10.10.10.2 to all lookup interfaceA_table

# set a default gateway of 10.10.10.1 for the table "interfaceA".
ip route add default via 10.10.10.1 table interfaceA_table

Add the rule and routes to /etc/sysconfig/network-scripts/rule-<ifname> and route-<ifname>, as they won’t persist over reboots without them.

One last useful article: https://blog.scottlowe.org/2013/05/29/a-quick-introduction-to-linux-policy-routing/

Leave a Reply

Your email address will not be published. Required fields are marked *