Tips from the Trenches, part 1

This is a new series where I’ll share some tips I’ve learned over the previous week or two. Mostly, this is going to involve valuable lessons I’ve learned after a mistake or two, or three.

No, the LACP bonds aren’t all right

Have a misbehaving LACP bond? If you want to double check the networking team’s work for your LACP bonds, cat out /proc/net/bonding/<bond_ifname>, and take a look at the Aggregator IDs used on each interface.

If your bond is set up to go to a single switch, or to two switches that are taking part in MLAG, the aggregator IDs should be the same on all interfaces in that bond.

Symptoms include packets arriving on a bond interface only when one of the child interfaces is up. tcpdump shows the packets arriving at the child interface of the bond, but not the bond itself.

Useful article:

Another, showing when aggregator IDs can be different:

Dude, where’s my asynchronously routed packets?

Got a situation where you have two interfaces on a host, on separate subnets with their own gateways, and packets from the outside world could arrive on either interface? It could be a recipe for asynchronous routing problems. If the packet arrives on interface A, but the reverse path leaves via interface B, the packet will be dropped by default.

Useful article:


One option, other than weakening the reverse path check, is to use a policy-based route to ensure path symmetry; i.e. that packets leave on the interface they come in on. Create your own routing table, shunt any packets coming from interface A’s IP address into it, then put a interface A’s default gateway into the table:

# create a new iproute2 table
echo "200 interfaceA" >> /etc/iproute/rt_tables

# is interface A's IP address - anything with a source IP of jumps to the table "interfaceA"
# i.e. it doesn't use the main routing table.
ip rule add from to all lookup interfaceA_table

# set a default gateway of for the table "interfaceA".
ip route add default via table interfaceA_table

Add the rule and routes to /etc/sysconfig/network-scripts/rule-<ifname> and route-<ifname>, as they won’t persist over reboots without them.

One last useful article:

Leave a Reply

Your email address will not be published. Required fields are marked *