[Discuss] vlans and arp

[me at mattgillen.net (Matthew Gillen)]

I'm pulling my hair out.  I have a fedora box (serv1) that is my network 
management hub.  It has a physical interface with no address, and a 
series of vlans configured (via NetworkManager).

This is plugged into a netgear switch that sends vlan tagged packets to 
serv1.  This works great.  (side note: this is a great way to have one 
dhcpd serve a lot of VLANs: the client requests come in to the vlan 
interfaces, and then a single dhcpd instance knows which subnet it 
should issue an address for)

A botched upgrade made me finally want to separate the net management 
from my file server.  So I have a little NUC-like box that I'm trying to 
set up as a replacement for serv1.

serv2 is RHEL9.  Tried to set it up the same way using NetworkManager: a 
base interface with no address, and VLAN interfaces.

The netgear switch has the same vlan config for the ports serv1 and 
serv2 are using (ie every vlan is using tagging).

On serv2, a tcpdump of the base interface (enp1s0) sees VLAN-tagged 
traffic for several vlans, as expected.  A tcpdump of the VLAN interface 
shows the untagged traffic for that VLAN, again as expected.

The problem is serv1 never sees any arp requests or replies from serv2. 
serv2 sees ARP coming in from serv1, and sends a reply.  tcpdump on 
serv2 shows the reply being sent (both the untagged version going out 
the VLAN interface, and the VLAN-tagged version going out the base 
interface.  But serv1 never sees anything.  It's like the outgoing 
arp-reply gets dropped on the floor sometime after the tcpdump on serv2 
sees it.

What would make arp only work in one direction like that?  My google-fu 
turned up some stuff about ARP-flux, but that seems like a different 
problem (my arp things appear to be routed to the correct interface, 
they're just getting dropped somewhere).

Any ideas?

Thanks
Matt