I have an update! On two moderately loaded CCRs, I have deployed a non-bridged solution. All of the Q-in-Q sub-interfaces are now their own broadcast domains. As I use static ip addresses, it is possible to assign a specific ip address to each AVC:
948 address=180.181.80.1/32 network=180.181.86.111 interface=AVC0000XYZXYZ actual-interface=AVC0000XYZXYZ
Yes, that works ... (the route below is auto-generated, the ARP entry comes from the DHCP "Assign ARP" feature.)
[terry@CCR2.NSW] /ip route> print detail where dst-address ="180.181.86.111/32" Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 0 ADC dst-address=180.181.86.111/32 pref-src=180.181.80.1 gateway=AVC0000XYZXYZ gateway-status=AVC0000XYZXYZ reachable distance=0 scope=10 [terry@CCR2.NSW] /ip route>
[terry@CCR2.NSW] /ip arp> print detail where address ="180.181.86.111" Flags: X - disabled, I - invalid, H - DHCP, D - dynamic, P - published, C - complete 0 HDC address=180.181.86.111 mac-address=10:BE:F5:CD:93:4B interface=AVC0000XYZXYZ published=no [terry@CCR2.NSW] /ip arp>
The sub-interface has proxy-arp turned up, so the AVC gets to ARP all of the 180.180.80.0/20 without issue. (The CCR hands out it's MAC address, problems solved.) Right now, I have 950 interfaces, 450 active clients and 2% CPU load on the busiest CCR. Almost every single byte is moved using FP. So far, the only issue is -- I need to wait around 1 second between adding an interface and making a /ip/dhcp-server on the interface -- the DHCP instance will go "red" aka invalid if added too fast. I have some concerns about ARP poisoning given the CCR now has proxy-arp running, but otherwise so far so good. --- http://about.me/terry.sweetser On 29/08/16 15:23, Terry Sweetser (SkyMesh) wrote:
Hello MK People,
I have now several CCR 36core deployed routers, with Q-in-Q interfaces numbering in 4,000 range.
They're in a single bridge group and DHCP broadcasts and ipv6 multicasts have become a major issue.
A single DHCP reply generates an ipv4 broadcast to 4000 interfaces in the bridge group.
The group members are all on the same split horizon, so no port to port traffic occurs.
When ipv6 ND fires up some discoveries, again N*4000+ packets go out.
Given these are 600 bytes, and N*4000 PPS, N being number of open requests or discoveries, there's multiples of 10Mbps of traffic going out.
I'm loath to redo the bridge concept, it would mean burning up a /30 per site for "their own subnet" space.
PPPoE, as an answer to urgent support cases has been used, but for 10,000 sites this is just not possible.
Currently, I have a bridge rate limit in place to "slow down" (a la tar pit) floods of DHCP requests.
Where to from here?
Is there some new or interesting Cisco-like "unnumbered" scheme possible?
Can I try to move IPv6 to the individual interfaces, and drop IPv6 across the bridge?
Can I filter out all bridge DHCP and attach 4000 DHCP servers to the interfaces? (No joke.)