L3 over QinQ and a Bridge -- too far?
Hello Mikrotikians, I've been looking at this one for a while now. I have several hundred QinQ vlans heading into a single bridge group. ipv4 and ipv6 from the bridge interface into the individual links is working well. None of the ipv4 addresses can actually connect to or ping each other. Flags: X - disabled, R - running 0 R name="bridge136" mtu=auto actual-mtu=1500 l2mtu=1992 *arp=proxy-arp* mac-address=D4:CA:6D:01:38:C7 protocol-mode=none priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s transmit-hold-count=6 ageing-time=5m Flags: X - disabled, I - inactive, D - dynamic 0 interface=XXXXXX bridge=bridge136 priority=0x80 path-cost=10 edge=auto *point-to-point=yes* external-fdb=auto horizon=1 auto-isolate=no A bridge too far? _So, I'm looking for solution where the IPv* arrives at bridge136 and then goes straight back out the correct port.__ _ -- http://about.me/terry.sweetser
Hi Terry, Any reason why you have proxy ARP enabled? I don't think you will want the bridge to be answering any ARP requests other than to its own address/es (do you?) Also, I'm not certain what is the effect of point-to-point mode on bridge port, but I can't imagine that you would need to force it in these cases. Since you highlighted both attributes, I suspect that you have some focus on them already - can you add any comment on why you have selected them? Cheers Mike.
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Terry Sweetser Sent: Wednesday, 22 April 2015 8:22 AM To: public@talk.mikrotik.com.au Subject: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hello Mikrotikians,
I've been looking at this one for a while now.
I have several hundred QinQ vlans heading into a single bridge group.
ipv4 and ipv6 from the bridge interface into the individual links is working well.
None of the ipv4 addresses can actually connect to or ping each other.
Flags: X - disabled, R - running 0 R name="bridge136" mtu=auto actual-mtu=1500 l2mtu=1992 *arp=proxy-arp* mac-address=D4:CA:6D:01:38:C7 protocol-mode=none priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s transmit-hold-count=6 ageing-time=5m
Flags: X - disabled, I - inactive, D - dynamic 0 interface=XXXXXX bridge=bridge136 priority=0x80 path-cost=10 edge=auto *point-to-point=yes* external-fdb=auto horizon=1 auto- isolate=no
A bridge too far?
_So, I'm looking for solution where the IPv* arrives at bridge136 and then goes straight back out the correct port.__ _ -- http://about.me/terry.sweetser
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Mike, Yes, I'm of the opinion I have the proxy-arp and p2p issues muddled up somehow. Changing to "enabled" for arp on the bridge makes no difference to the issue, less arp traffic is always good. I've also got "horizon" set as well, to "1" on all ports. So, what to try instead of point-to-point? http://about.me/terry.sweetser ------------------------------------------------------------------------ *From:* Mike Everest <mailto:%3Ca%3Emike@duxtel.com%3C/a%3E> *Sent:* Wednesday, April 22, 2015 8:51AM *To:* Terrence Sweetser, 'mikrotik Australia Public List' <mailto:%3Ca%3Eterry@skymesh.net.au%3C/a%3E,%20%3Ca%3Epublic@talk.mikrotik.com.au%3C/a%3E> *Subject:* RE: [MT-AU Public] L3 over QinQ and a Bridge -- too far? Hi Terry, Any reason why you have proxy ARP enabled? I don't think you will want the bridge to be answering any ARP requests other than to its own address/es (do you?) Also, I'm not certain what is the effect of point-to-point mode on bridge port, but I can't imagine that you would need to force it in these cases. Since you highlighted both attributes, I suspect that you have some focus on them already - can you add any comment on why you have selected them? Cheers Mike.
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Terry Sweetser Sent: Wednesday, 22 April 2015 8:22 AM To: public@talk.mikrotik.com.au Subject: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hello Mikrotikians,
I've been looking at this one for a while now.
I have several hundred QinQ vlans heading into a single bridge group.
ipv4 and ipv6 from the bridge interface into the individual links is
working
well.
None of the ipv4 addresses can actually connect to or ping each other.
Flags: X - disabled, R - running 0 R name="bridge136" mtu=auto actual-mtu=1500 l2mtu=1992 *arp=proxy-arp* mac-address=D4:CA:6D:01:38:C7 protocol-mode=none priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s transmit-hold-count=6 ageing-time=5m
Flags: X - disabled, I - inactive, D - dynamic 0 interface=XXXXXX bridge=bridge136 priority=0x80 path-cost=10 edge=auto *point-to-point=yes* external-fdb=auto horizon=1 auto- isolate=no
A bridge too far?
_So, I'm looking for solution where the IPv* arrives at bridge136 and then goes straight back out the correct port.__ _ -- http://about.me/terry.sweetser
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Terry, Am I correct in assuming you are trying to achieve PVLAN functionality ? This currently does not specifically exist on RouterOS, but there is a work around that achieves the same result. The trick is to drop packets from forwarding on the bridge, and use bridge NAT to re-write arp requests so all traffic between clients passes via the Mikrotik bridge MAC address e.g. /interface bridge filter add action=drop chain=forward comment="Drop everything" in-bridge=bridge4 out-bridge=bridge4 /interface bridge nat add action=arp-reply chain=dstnat comment="Re-write MAC in ARP responses" in-bridge=bridge4 mac-protocol=arp to-arp-reply-mac-address=D4:CA:6D:01:3B:80 Where bridge4 is the bridge you are doing this on and "D4:CA:6D:01:3B:80" is the MAC address of the bridge. I requested a "pvlan" arp option on bridges and Mikrotik came back that they will add this in RouterOS v7 which will negate the need for this workaround. Regards, Andrew On Wed, Apr 22, 2015 at 11:16 AM, Terry Sweetser < terry+mikrotik@skymesh.net.au> wrote:
Hi Mike,
Yes, I'm of the opinion I have the proxy-arp and p2p issues muddled up somehow.
Changing to "enabled" for arp on the bridge makes no difference to the issue, less arp traffic is always good.
I've also got "horizon" set as well, to "1" on all ports.
So, what to try instead of point-to-point?
http://about.me/terry.sweetser
------------------------------------------------------------------------
*From:* Mike Everest <mailto:%3Ca%3Emike@duxtel.com%3C/a%3E> *Sent:* Wednesday, April 22, 2015 8:51AM *To:* Terrence Sweetser, 'mikrotik Australia Public List' <mailto:% 3Ca%3Eterry@skymesh.net.au%3C/a%3E,%20%3Ca%3Epublic@talk.mikrotik.com.au %3C/a%3E> *Subject:* RE: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hi Terry,
Any reason why you have proxy ARP enabled? I don't think you will want the bridge to be answering any ARP requests other than to its own address/es (do you?)
Also, I'm not certain what is the effect of point-to-point mode on bridge port, but I can't imagine that you would need to force it in these cases.
Since you highlighted both attributes, I suspect that you have some focus on them already - can you add any comment on why you have selected them?
Cheers
Mike.
-----Original Message-----
From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Terry Sweetser Sent: Wednesday, 22 April 2015 8:22 AM To: public@talk.mikrotik.com.au Subject: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hello Mikrotikians,
I've been looking at this one for a while now.
I have several hundred QinQ vlans heading into a single bridge group.
ipv4 and ipv6 from the bridge interface into the individual links is
working
well.
None of the ipv4 addresses can actually connect to or ping each other.
Flags: X - disabled, R - running 0 R name="bridge136" mtu=auto actual-mtu=1500 l2mtu=1992 *arp=proxy-arp* mac-address=D4:CA:6D:01:38:C7 protocol-mode=none priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s transmit-hold-count=6 ageing-time=5m
Flags: X - disabled, I - inactive, D - dynamic 0 interface=XXXXXX bridge=bridge136 priority=0x80 path-cost=10 edge=auto *point-to-point=yes* external-fdb=auto horizon=1 auto- isolate=no
A bridge too far?
_So, I'm looking for solution where the IPv* arrives at bridge136 and then goes straight back out the correct port.__ _ -- http://about.me/terry.sweetser
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Andrew, Initially, the consideration is security -- I don't want L2 broadcast traffic to travel between client endpoints. Setting horizon "1" on all of the AVCs ensures that, but then ARP no longer gets to sends queries from AVC to AVC. Removing the horizon causes a massive ARP storm between 600+ interfaces all sending broadcasts. I've also discovered that several brands of cheap CPE router do not respond well to seeing 600+ mac addresses from the WAN port. Removing the point-to-point setting and swapping between "enabled" and "proxy-arp" has not changed the client to client routing situation. Starting to think a bridge group of this size is not a good idea. http://about.me/terry.sweetser ------------------------------------------------------------------------ *From:* Andrew Thrift <mailto:%3Ca%3Eandrew@networklabs.co.nz%3C/a%3E> *Sent:* Wednesday, April 22, 2015 10:02AM *To:* Terrence Sweetser, 'mikrotik Australia Public List' <mailto:%3Ca%3Eterry@skymesh.net.au%3C/a%3E,%20%3Ca%3Epublic@talk.mikrotik.com.au%3C/a%3E> *Subject:* Re: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hi Terry,
Am I correct in assuming you are trying to achieve PVLAN functionality ?
This currently does not specifically exist on RouterOS, but there is a work around that achieves the same result.
The trick is to drop packets from forwarding on the bridge, and use bridge NAT to re-write arp requests so all traffic between clients passes via the Mikrotik bridge MAC address e.g.
/interface bridge filter add action=drop chain=forward comment="Drop everything" in-bridge=bridge4 out-bridge=bridge4
/interface bridge nat add action=arp-reply chain=dstnat comment="Re-write MAC in ARP responses" in-bridge=bridge4 mac-protocol=arp to-arp-reply-mac-address=D4:CA:6D:01:3B:80
Where bridge4 is the bridge you are doing this on and "D4:CA:6D:01:3B:80" is the MAC address of the bridge.
I requested a "pvlan" arp option on bridges and Mikrotik came back that they will add this in RouterOS v7 which will negate the need for this workaround.
Regards,
Andrew
On Wed, Apr 22, 2015 at 11:16 AM, Terry Sweetser < terry+mikrotik@skymesh.net.au> wrote:
Hi Mike,
Yes, I'm of the opinion I have the proxy-arp and p2p issues muddled up somehow.
Changing to "enabled" for arp on the bridge makes no difference to the issue, less arp traffic is always good.
I've also got "horizon" set as well, to "1" on all ports.
So, what to try instead of point-to-point?
http://about.me/terry.sweetser
------------------------------------------------------------------------
*From:* Mike Everest <mailto:%3Ca%3Emike@duxtel.com%3C/a%3E> *Sent:* Wednesday, April 22, 2015 8:51AM *To:* Terrence Sweetser, 'mikrotik Australia Public List' <mailto:% 3Ca%3Eterry@skymesh.net.au%3C/a%3E,%20%3Ca%3Epublic@talk.mikrotik.com.au %3C/a%3E> *Subject:* RE: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hi Terry,
Any reason why you have proxy ARP enabled? I don't think you will want the bridge to be answering any ARP requests other than to its own address/es (do you?)
Also, I'm not certain what is the effect of point-to-point mode on bridge port, but I can't imagine that you would need to force it in these cases.
Since you highlighted both attributes, I suspect that you have some focus on them already - can you add any comment on why you have selected them?
Cheers
Mike.
-----Original Message-----
From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Terry Sweetser Sent: Wednesday, 22 April 2015 8:22 AM To: public@talk.mikrotik.com.au Subject: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hello Mikrotikians,
I've been looking at this one for a while now.
I have several hundred QinQ vlans heading into a single bridge group.
ipv4 and ipv6 from the bridge interface into the individual links is
working
well.
None of the ipv4 addresses can actually connect to or ping each other.
Flags: X - disabled, R - running 0 R name="bridge136" mtu=auto actual-mtu=1500 l2mtu=1992 *arp=proxy-arp* mac-address=D4:CA:6D:01:38:C7 protocol-mode=none priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s transmit-hold-count=6 ageing-time=5m
Flags: X - disabled, I - inactive, D - dynamic 0 interface=XXXXXX bridge=bridge136 priority=0x80 path-cost=10 edge=auto *point-to-point=yes* external-fdb=auto horizon=1 auto- isolate=no
A bridge too far?
_So, I'm looking for solution where the IPv* arrives at bridge136 and then goes straight back out the correct port.__ _ -- http://about.me/terry.sweetser
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Terry, If you use the trick I posted above it will solve both problems for you. The clients will only see 1 mac address as we are using the bridge NAT policy to re-write the replied MAC in all ARP requests making all Layer3 traffic between clients go via your headend router, and all traffic directly between clients on the bridge will be blocked by the filter policy. Regards, Andrew On Wed, Apr 22, 2015 at 3:27 PM, Terry Sweetser < terry+mikrotik@skymesh.net.au> wrote:
Hi Andrew,
Initially, the consideration is security -- I don't want L2 broadcast traffic to travel between client endpoints.
Setting horizon "1" on all of the AVCs ensures that, but then ARP no longer gets to sends queries from AVC to AVC.
Removing the horizon causes a massive ARP storm between 600+ interfaces all sending broadcasts.
I've also discovered that several brands of cheap CPE router do not respond well to seeing 600+ mac addresses from the WAN port.
Removing the point-to-point setting and swapping between "enabled" and "proxy-arp" has not changed the client to client routing situation.
Starting to think a bridge group of this size is not a good idea.
http://about.me/terry.sweetser
------------------------------------------------------------------------
*From:* Andrew Thrift <mailto:%3Ca%3Eandrew@networklabs.co.nz%3C/a%3E> *Sent:* Wednesday, April 22, 2015 10:02AM *To:* Terrence Sweetser, 'mikrotik Australia Public List' <mailto:% 3Ca%3Eterry@skymesh.net.au%3C/a%3E,%20%3Ca%3Epublic@talk.mikrotik.com.au %3C/a%3E> *Subject:* Re: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hi Terry,
Am I correct in assuming you are trying to achieve PVLAN functionality ?
This currently does not specifically exist on RouterOS, but there is a work around that achieves the same result.
The trick is to drop packets from forwarding on the bridge, and use bridge NAT to re-write arp requests so all traffic between clients passes via the Mikrotik bridge MAC address e.g.
/interface bridge filter add action=drop chain=forward comment="Drop everything" in-bridge=bridge4 out-bridge=bridge4
/interface bridge nat add action=arp-reply chain=dstnat comment="Re-write MAC in ARP responses" in-bridge=bridge4 mac-protocol=arp to-arp-reply-mac-address=D4:CA:6D:01:3B:80
Where bridge4 is the bridge you are doing this on and "D4:CA:6D:01:3B:80" is the MAC address of the bridge.
I requested a "pvlan" arp option on bridges and Mikrotik came back that they will add this in RouterOS v7 which will negate the need for this workaround.
Regards,
Andrew
On Wed, Apr 22, 2015 at 11:16 AM, Terry Sweetser < terry+mikrotik@skymesh.net.au> wrote:
Hi Mike,
Yes, I'm of the opinion I have the proxy-arp and p2p issues muddled up somehow.
Changing to "enabled" for arp on the bridge makes no difference to the issue, less arp traffic is always good.
I've also got "horizon" set as well, to "1" on all ports.
So, what to try instead of point-to-point?
http://about.me/terry.sweetser
------------------------------------------------------------------------
*From:* Mike Everest <mailto:%3Ca%3Emike@duxtel.com%3C/a%3E> *Sent:* Wednesday, April 22, 2015 8:51AM *To:* Terrence Sweetser, 'mikrotik Australia Public List' <mailto:% 3Ca%3Eterry@skymesh.net.au%3C/a%3E,%20%3Ca%3Epublic@talk.mikrotik.com.au %3C/a%3E> *Subject:* RE: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hi Terry,
Any reason why you have proxy ARP enabled? I don't think you will want the bridge to be answering any ARP requests other than to its own address/es (do you?)
Also, I'm not certain what is the effect of point-to-point mode on bridge port, but I can't imagine that you would need to force it in these cases.
Since you highlighted both attributes, I suspect that you have some focus on them already - can you add any comment on why you have selected them?
Cheers
Mike.
-----Original Message-----
From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Terry Sweetser Sent: Wednesday, 22 April 2015 8:22 AM To: public@talk.mikrotik.com.au Subject: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hello Mikrotikians,
I've been looking at this one for a while now.
I have several hundred QinQ vlans heading into a single bridge group.
ipv4 and ipv6 from the bridge interface into the individual links is
working
well.
None of the ipv4 addresses can actually connect to or ping each other.
Flags: X - disabled, R - running 0 R name="bridge136" mtu=auto actual-mtu=1500 l2mtu=1992 *arp=proxy-arp* mac-address=D4:CA:6D:01:38:C7 protocol-mode=none priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s transmit-hold-count=6 ageing-time=5m
Flags: X - disabled, I - inactive, D - dynamic 0 interface=XXXXXX bridge=bridge136 priority=0x80 path-cost=10 edge=auto *point-to-point=yes* external-fdb=auto horizon=1 auto- isolate=no
A bridge too far?
_So, I'm looking for solution where the IPv* arrives at bridge136 and then goes straight back out the correct port.__ _ -- http://about.me/terry.sweetser
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________
Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hello People! I have tried these ideas below again, and still can't get to a working solution. The CCR logs the NAT rule "as working", but no packets exit the CCR -- quick sniffer sees no reply. Any more ideas?
[terry@CCR1.QLD] > /interface bridge filter export # sep/09/2015 13:09:04 by RouterOS 6.27 # software id = YXCC-M0LZ # /interface bridge filter add action=drop chain=forward comment="Remove Broadcasts between clients ..." in-bridge=bridge136 log-prefix=DROP out-bridge=bridge136
[terry@CCR1.QLD] > /interface bridge nat export # sep/09/2015 13:09:10 by RouterOS 6.27 # software id = YXCC-M0LZ # /interface bridge nat add action=arp-reply chain=dstnat comment="Force AVC to AVC traffic to use CCR ..." in-bridge=bridge136 log-prefix=NAT mac-protocol=arp to-arp-reply-mac-address=D4:CA:6D:01:38:C7
--- http://about.me/terry.sweetser ------------------------------------------------------------------------ *From:* Andrew Thrift <mailto:andrew@networklabs.co.nz> *Sent:* Wednesday, April 22, 2015 1:43PM *To:* Terence C Sweetser, 'Mikrotik Australia Public List' <mailto:terry@skymesh.net.au,%20public@talk.mikrotik.com.au> *Subject:* Re: [MT-AU Public] L3 over QinQ and a Bridge -- too far? Hi Terry, If you use the trick I posted above it will solve both problems for you. The clients will only see 1 mac address as we are using the bridge NAT policy to re-write the replied MAC in all ARP requests making all Layer3 traffic between clients go via your headend router, and all traffic directly between clients on the bridge will be blocked by the filter policy. Regards, Andrew On Wed, Apr 22, 2015 at 3:27 PM, Terry Sweetser < terry+mikrotik@skymesh.net.au> wrote:
Hi Andrew,
Initially, the consideration is security -- I don't want L2 broadcast traffic to travel between client endpoints.
Setting horizon "1" on all of the AVCs ensures that, but then ARP no longer gets to sends queries from AVC to AVC.
Removing the horizon causes a massive ARP storm between 600+ interfaces all sending broadcasts.
I've also discovered that several brands of cheap CPE router do not respond well to seeing 600+ mac addresses from the WAN port.
Removing the point-to-point setting and swapping between "enabled" and "proxy-arp" has not changed the client to client routing situation.
Starting to think a bridge group of this size is not a good idea.
http://about.me/terry.sweetser
------------------------------------------------------------------------
*From:* Andrew Thrift<mailto:%3Ca%3Eandrew@networklabs.co.nz%3C/a%3E> *Sent:* Wednesday, April 22, 2015 10:02AM *To:* Terrence Sweetser, 'mikrotik Australia Public List'<mailto:% 3Ca%3Eterry@skymesh.net.au%3C/a%3E,%20%3Ca%3Epublic@talk.mikrotik.com.au %3C/a%3E> *Subject:* Re: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hi Terry,
Am I correct in assuming you are trying to achieve PVLAN functionality ?
This currently does not specifically exist on RouterOS, but there is a work around that achieves the same result.
The trick is to drop packets from forwarding on the bridge, and use bridge NAT to re-write arp requests so all traffic between clients passes via the Mikrotik bridge MAC address e.g.
/interface bridge filter add action=drop chain=forward comment="Drop everything" in-bridge=bridge4 out-bridge=bridge4
/interface bridge nat add action=arp-reply chain=dstnat comment="Re-write MAC in ARP responses" in-bridge=bridge4 mac-protocol=arp to-arp-reply-mac-address=D4:CA:6D:01:3B:80
Where bridge4 is the bridge you are doing this on and "D4:CA:6D:01:3B:80" is the MAC address of the bridge.
I requested a "pvlan" arp option on bridges and Mikrotik came back that they will add this in RouterOS v7 which will negate the need for this workaround.
Regards,
Andrew
On Wed, Apr 22, 2015 at 11:16 AM, Terry Sweetser < terry+mikrotik@skymesh.net.au> wrote:
Hi Mike,
Yes, I'm of the opinion I have the proxy-arp and p2p issues muddled up somehow.
Changing to "enabled" for arp on the bridge makes no difference to the issue, less arp traffic is always good.
I've also got "horizon" set as well, to "1" on all ports.
So, what to try instead of point-to-point?
http://about.me/terry.sweetser
------------------------------------------------------------------------
*From:* Mike Everest<mailto:%3Ca%3Emike@duxtel.com%3C/a%3E> *Sent:* Wednesday, April 22, 2015 8:51AM *To:* Terrence Sweetser, 'mikrotik Australia Public List'<mailto:% 3Ca%3Eterry@skymesh.net.au%3C/a%3E,%20%3Ca%3Epublic@talk.mikrotik.com.au %3C/a%3E> *Subject:* RE: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hi Terry,
Any reason why you have proxy ARP enabled? I don't think you will want the bridge to be answering any ARP requests other than to its own address/es (do you?)
Also, I'm not certain what is the effect of point-to-point mode on bridge port, but I can't imagine that you would need to force it in these cases.
Since you highlighted both attributes, I suspect that you have some focus on them already - can you add any comment on why you have selected them?
Cheers
Mike.
-----Original Message-----
From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Terry Sweetser Sent: Wednesday, 22 April 2015 8:22 AM To:public@talk.mikrotik.com.au Subject: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hello Mikrotikians,
I've been looking at this one for a while now.
I have several hundred QinQ vlans heading into a single bridge group.
ipv4 and ipv6 from the bridge interface into the individual links is
working well.
None of the ipv4 addresses can actually connect to or ping each other.
Flags: X - disabled, R - running 0 R name="bridge136" mtu=auto actual-mtu=1500 l2mtu=1992 *arp=proxy-arp* mac-address=D4:CA:6D:01:38:C7 protocol-mode=none priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s transmit-hold-count=6 ageing-time=5m
Flags: X - disabled, I - inactive, D - dynamic 0 interface=XXXXXX bridge=bridge136 priority=0x80 path-cost=10 edge=auto *point-to-point=yes* external-fdb=auto horizon=1 auto- isolate=no
A bridge too far?
_So, I'm looking for solution where the IPv* arrives at bridge136 and then goes straight back out the correct port.__ _ -- http://about.me/terry.sweetser
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
I'm confused.. :-}
None of the ipv4 addresses can actually connect to or ping each other.
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Terry Sweetser Sent: Wednesday, 22 April 2015 1:28 PM To: public@talk.mikrotik.com.au Subject: Re: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hi Andrew,
Initially, the consideration is security -- I don't want L2 broadcast
travel between client endpoints.
Setting horizon "1" on all of the AVCs ensures that, but then ARP no longer gets to sends queries from AVC to AVC.
Removing the horizon causes a massive ARP storm between 600+ interfaces all sending broadcasts.
I've also discovered that several brands of cheap CPE router do not respond well to seeing 600+ mac addresses from the WAN port.
Removing the point-to-point setting and swapping between "enabled" and "proxy-arp" has not changed the client to client routing situation.
Starting to think a bridge group of this size is not a good idea.
http://about.me/terry.sweetser
------------------------------------------------------------------------
*From:* Andrew Thrift <mailto:%3Ca%3Eandrew@networklabs.co.nz%3C/a%3E> *Sent:* Wednesday, April 22, 2015 10:02AM *To:* Terrence Sweetser, 'mikrotik Australia Public List' <mailto:%3Ca%3Eterry@skymesh.net.au%3C/a%3E,%20%3Ca%3Epublic@tal k.mikrotik.com.au%3C/a%3E> *Subject:* Re: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hi Terry,
Am I correct in assuming you are trying to achieve PVLAN functionality ?
This currently does not specifically exist on RouterOS, but there is a work around that achieves the same result.
The trick is to drop packets from forwarding on the bridge, and use bridge NAT to re-write arp requests so all traffic between clients passes via the Mikrotik bridge MAC address e.g.
/interface bridge filter add action=drop chain=forward comment="Drop everything" in-bridge=bridge4 out-bridge=bridge4
/interface bridge nat add action=arp-reply chain=dstnat comment="Re-write MAC in ARP responses" in-bridge=bridge4 mac-protocol=arp to-arp-reply-mac-address=D4:CA:6D:01:3B:80
Where bridge4 is the bridge you are doing this on and "D4:CA:6D:01:3B:80" is the MAC address of the bridge.
I requested a "pvlan" arp option on bridges and Mikrotik came back that they will add this in RouterOS v7 which will negate the need for this workaround.
Regards,
Andrew
On Wed, Apr 22, 2015 at 11:16 AM, Terry Sweetser < terry+mikrotik@skymesh.net.au> wrote:
Hi Mike,
Yes, I'm of the opinion I have the proxy-arp and p2p issues muddled up somehow.
Changing to "enabled" for arp on the bridge makes no difference to the issue, less arp traffic is always good.
I've also got "horizon" set as well, to "1" on all ports.
So, what to try instead of point-to-point?
http://about.me/terry.sweetser
--------------------------------------------------------------------- ---
*From:* Mike Everest <mailto:%3Ca%3Emike@duxtel.com%3C/a%3E> *Sent:* Wednesday, April 22, 2015 8:51AM *To:* Terrence Sweetser, 'mikrotik Australia Public List' <mailto:%
3Ca%3Eterry@skymesh.net.au%3C/a%3E,%20%3Ca%3Epublic@talk.mikrotik. com
.au %3C/a%3E> *Subject:* RE: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hi Terry,
Any reason why you have proxy ARP enabled? I don't think you will want the bridge to be answering any ARP requests other than to its own address/es (do you?)
Also, I'm not certain what is the effect of point-to-point mode on bridge port, but I can't imagine that you would need to force it in
I read that as the problem, not as "that works, check!" ;-) Thus my strange remarks previous! :-D Cheers! Mike. traffic to these
cases.
Since you highlighted both attributes, I suspect that you have some focus on them already - can you add any comment on why you have selected them?
Cheers
Mike.
-----Original Message-----
From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Terry Sweetser Sent: Wednesday, 22 April 2015 8:22 AM To: public@talk.mikrotik.com.au Subject: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hello Mikrotikians,
I've been looking at this one for a while now.
I have several hundred QinQ vlans heading into a single bridge group.
ipv4 and ipv6 from the bridge interface into the individual links is
working
well.
None of the ipv4 addresses can actually connect to or ping each other.
Flags: X - disabled, R - running 0 R name="bridge136" mtu=auto actual-mtu=1500 l2mtu=1992 *arp=proxy-arp* mac-address=D4:CA:6D:01:38:C7 protocol-mode=none priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s transmit-hold-count=6 ageing-time=5m
Flags: X - disabled, I - inactive, D - dynamic 0 interface=XXXXXX bridge=bridge136 priority=0x80 path-cost=10 edge=auto *point-to-point=yes* external-fdb=auto horizon=1 auto- isolate=no
A bridge too far?
_So, I'm looking for solution where the IPv* arrives at bridge136 and then goes straight back out the correct port.__ _ -- http://about.me/terry.sweetser
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.co m.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com .au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Mike, Yes, confusing. :P I've run full circle on this today -- Andrew's hints have sent me in the right direction. Essentially, I'll be throwing out the bridge grouping at some stage, and attaching ARP and ND to individual sub interfaces. T -- http://about.me/terry.sweetser ------------------------------------------------------------------------ *From:* Mike Everest <mailto:%3Ca%3Emike@duxtel.com%3C/a%3E> *Sent:* Wednesday, April 22, 2015 4:16PM *To:* Terrence Sweetser, 'mikrotik Australia Public List' <mailto:%3Ca%3Eterry@skymesh.net.au%3C/a%3E,%20%3Ca%3Epublic@talk.mikrotik.com.au%3C/a%3E> *Subject:* Re: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
I'm confused.. :-}
None of the ipv4 addresses can actually connect to or ping each other. I read that as the problem, not as "that works, check!" ;-)
Thus my strange remarks previous! :-D
Cheers!
Mike.
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Terry Sweetser Sent: Wednesday, 22 April 2015 1:28 PM To: public@talk.mikrotik.com.au Subject: Re: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hi Andrew,
Initially, the consideration is security -- I don't want L2 broadcast traffic to travel between client endpoints.
Setting horizon "1" on all of the AVCs ensures that, but then ARP no longer gets to sends queries from AVC to AVC.
Removing the horizon causes a massive ARP storm between 600+ interfaces all sending broadcasts.
I've also discovered that several brands of cheap CPE router do not respond well to seeing 600+ mac addresses from the WAN port.
Removing the point-to-point setting and swapping between "enabled" and "proxy-arp" has not changed the client to client routing situation.
Starting to think a bridge group of this size is not a good idea.
http://about.me/terry.sweetser
------------------------------------------------------------------------
*From:* Andrew Thrift <mailto:%3Ca%3Eandrew@networklabs.co.nz%3C/a%3E> *Sent:* Wednesday, April 22, 2015 10:02AM *To:* Terrence Sweetser, 'mikrotik Australia Public List' <mailto:%3Ca%3Eterry@skymesh.net.au%3C/a%3E,%20%3Ca%3Epublic@tal k.mikrotik.com.au%3C/a%3E> *Subject:* Re: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hi Terry,
Am I correct in assuming you are trying to achieve PVLAN functionality ?
This currently does not specifically exist on RouterOS, but there is a work around that achieves the same result.
The trick is to drop packets from forwarding on the bridge, and use bridge NAT to re-write arp requests so all traffic between clients passes via the Mikrotik bridge MAC address e.g.
/interface bridge filter add action=drop chain=forward comment="Drop everything" in-bridge=bridge4 out-bridge=bridge4
/interface bridge nat add action=arp-reply chain=dstnat comment="Re-write MAC in ARP responses" in-bridge=bridge4 mac-protocol=arp to-arp-reply-mac-address=D4:CA:6D:01:3B:80
Where bridge4 is the bridge you are doing this on and "D4:CA:6D:01:3B:80" is the MAC address of the bridge.
I requested a "pvlan" arp option on bridges and Mikrotik came back that they will add this in RouterOS v7 which will negate the need for this workaround.
Regards,
Andrew
On Wed, Apr 22, 2015 at 11:16 AM, Terry Sweetser < terry+mikrotik@skymesh.net.au> wrote:
Hi Mike,
Yes, I'm of the opinion I have the proxy-arp and p2p issues muddled up somehow.
Changing to "enabled" for arp on the bridge makes no difference to the issue, less arp traffic is always good.
I've also got "horizon" set as well, to "1" on all ports.
So, what to try instead of point-to-point?
http://about.me/terry.sweetser
--------------------------------------------------------------------- ---
*From:* Mike Everest <mailto:%3Ca%3Emike@duxtel.com%3C/a%3E> *Sent:* Wednesday, April 22, 2015 8:51AM *To:* Terrence Sweetser, 'mikrotik Australia Public List' <mailto:%
.au %3C/a%3E> *Subject:* RE: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hi Terry,
Any reason why you have proxy ARP enabled? I don't think you will want the bridge to be answering any ARP requests other than to its own address/es (do you?)
Also, I'm not certain what is the effect of point-to-point mode on bridge port, but I can't imagine that you would need to force it in
3Ca%3Eterry@skymesh.net.au%3C/a%3E,%20%3Ca%3Epublic@talk.mikrotik. com these cases.
Since you highlighted both attributes, I suspect that you have some focus on them already - can you add any comment on why you have selected them?
Cheers
Mike.
-----Original Message-----
From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Terry Sweetser Sent: Wednesday, 22 April 2015 8:22 AM To: public@talk.mikrotik.com.au Subject: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
Hello Mikrotikians,
I've been looking at this one for a while now.
I have several hundred QinQ vlans heading into a single bridge group.
ipv4 and ipv6 from the bridge interface into the individual links is
working
well.
None of the ipv4 addresses can actually connect to or ping each other.
Flags: X - disabled, R - running 0 R name="bridge136" mtu=auto actual-mtu=1500 l2mtu=1992 *arp=proxy-arp* mac-address=D4:CA:6D:01:38:C7 protocol-mode=none priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s transmit-hold-count=6 ageing-time=5m
Flags: X - disabled, I - inactive, D - dynamic 0 interface=XXXXXX bridge=bridge136 priority=0x80 path-cost=10 edge=auto *point-to-point=yes* external-fdb=auto horizon=1 auto- isolate=no
A bridge too far?
_So, I'm looking for solution where the IPv* arrives at bridge136 and then goes straight back out the correct port.__ _ -- http://about.me/terry.sweetser
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.co m.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com .au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
I have some updates for you all ... As you're aware, I have several thousand IPoE clients into a single bridge group per CCR. Client to internet, internet to client -- works very well. Firstly, the bridge group is the best solution and works under these circumstances: [1] all arp from the CCR is "reply-only", as set on the bridge interface, and a DHCP server is set up on this interface, adds ARP when granting a lease; [2] every single port is set for horizon "1" so that no BC from any of the clients makes it to any other client, this has proved very important, many cheap CPE routers respond badly to DHCP and other broadcasts on the WAN port; [3] 'Agent Circuit ID' checks are being done via RADIUS, so all clients have a "sticky" address, and "authenticated" IPoE allocations. Now, after some feedback from the list, I know that I can use a Bridge Filter rule to block "client" to "client" traffic, given how detrimental it can be. Sadly, I still don't have a working for solution for the "Layer2 NAT", the rule appears to be ignored under 6.27 and 6.28 /interface bridge nat add action=arp-reply chain=dstnat comment="Re-write MAC in ARP responses" in-bridge=bridgeXXX mac-protocol=arp to-arp-reply-mac-address=XX:XX:XX:XX:XX:XX So, any more suggestions? -- about.me/terry.sweetser
Hi Terry, Have you consulted the packet flow diagram to make sure that packets can get to the right point for mac-Nat processing? http://wiki.mikrotik.com/wiki/Manual:Packet_Flow Depending on how the firewall and bridge rules are set up, packets might be changed already by the time you expect to process them, so can miss your match rules. If you are sure that the feature is not working in some revision (e.g. same configuration works in version x, doesn't work in version y) then please try to records steps to reproduce the problem, and submit to support@mikrotik, or via your distributor support if you'd like some additional assistance. Cheers! Mike.
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Terry Sweetser Sent: Saturday, 2 May 2015 10:19 AM To: public@talk.mikrotik.com.au Subject: Re: [MT-AU Public] L3 over QinQ and a Bridge -- too far?
I have some updates for you all ...
As you're aware, I have several thousand IPoE clients into a single bridge group per CCR.
Client to internet, internet to client -- works very well.
Firstly, the bridge group is the best solution and works under these circumstances:
[1] all arp from the CCR is "reply-only", as set on the bridge interface, and a DHCP server is set up on this interface, adds ARP when granting a lease;
[2] every single port is set for horizon "1" so that no BC from any of the clients makes it to any other client, this has proved very important, many cheap CPE routers respond badly to DHCP and other broadcasts on the WAN port;
[3] 'Agent Circuit ID' checks are being done via RADIUS, so all clients have a "sticky" address, and "authenticated" IPoE allocations.
Now, after some feedback from the list, I know that I can use a Bridge Filter rule to block "client" to "client" traffic, given how detrimental it can be.
Sadly, I still don't have a working for solution for the "Layer2 NAT", the rule appears to be ignored under 6.27 and 6.28
/interface bridge nat add action=arp-reply chain=dstnat comment="Re-write MAC in ARP responses" in-bridge=bridgeXXX mac-protocol=arp to-arp-reply- mac-address=XX:XX:XX:XX:XX:XX
So, any more suggestions?
-- about.me/terry.sweetser
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Terry, This is working perfectly for us on 6.5 and 6.19 in production. There is always the possibility there has been a regression in later releases though.. Maybe something is slightly different in your configuration. Do you have a test bridge you are trialling this config on ? Have you tried using the bridge forwarding block rule rather than split horizon ? If you want another set of eyes I can do a TeamViewer session with you and have a look over it. On Sat, May 2, 2015 at 12:18 PM, Terry Sweetser < terry+mikrotik@skymesh.net.au> wrote:
I have some updates for you all ...
As you're aware, I have several thousand IPoE clients into a single bridge group per CCR.
Client to internet, internet to client -- works very well.
Firstly, the bridge group is the best solution and works under these circumstances:
[1] all arp from the CCR is "reply-only", as set on the bridge interface, and a DHCP server is set up on this interface, adds ARP when granting a lease;
[2] every single port is set for horizon "1" so that no BC from any of the clients makes it to any other client, this has proved very important, many cheap CPE routers respond badly to DHCP and other broadcasts on the WAN port;
[3] 'Agent Circuit ID' checks are being done via RADIUS, so all clients have a "sticky" address, and "authenticated" IPoE allocations.
Now, after some feedback from the list, I know that I can use a Bridge Filter rule to block "client" to "client" traffic, given how detrimental it can be.
Sadly, I still don't have a working for solution for the "Layer2 NAT", the rule appears to be ignored under 6.27 and 6.28
/interface bridge nat add action=arp-reply chain=dstnat comment="Re-write MAC in ARP responses" in-bridge=bridgeXXX mac-protocol=arp to-arp-reply-mac-address=XX:XX:XX:XX:XX:XX
So, any more suggestions?
-- about.me/terry.sweetser
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
participants (4)
-
Andrew Thrift -
Mike Everest -
Terry Sweetser -
Terry Sweetser (SkyMesh)