Hi guys, thought I would post this information to see if anybody had experienced the same issue. We have an RB1100AHx2 in a data centre that connects to a CCR in another data centre, unfortunately we can't do QINQ between the DC's so we run an EOIP tunnel to connect two vlans for a customer through to there, all has been working fine for about 2 years. The other day, just out of the blue, the customer said their link had gone down, after a number of hours troubleshooting, convinced that it was a problem on site, we discovered that the EOIP tunnel just stopped passing traffic. Nothing had changed on those two routers for months but all of a sudden they just stopped working. The CCR has some other EOIP tunnels on it which were still working fine, but the 1100 was refusing to work. We checked firewall rules, vlan ID's, tunnel ID's, everything, and there were no issues, the tunnel just refused to pass data. I tried changing the tunnel ID and it made no difference, tried re-adding the vlans to it, nothing, disabled and enabled it, nothing made a difference. Out of desperation we created a new EOIP tunnel on that router with a new ID and pointed it to the existing EOIP interface at the other end and it came up straight away. So apart from a big WTF, has anybody experienced this before ? Regards Paul
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Paul Julian Sent: Tuesday, 10 May 2016 9:23 AM To: public@talk.mikrotik.com.au Subject: [MT-AU Public] EOIP Epic Fail
Hi guys, thought I would post this information to see if anybody had experienced the same issue.
We have an RB1100AHx2 in a data centre that connects to a CCR in another data centre, unfortunately we can't do QINQ between the DC's so we run an EOIP tunnel to connect two vlans for a customer through to there, all has been working fine for about 2 years.
The other day, just out of the blue, the customer said their link had gone down, after a number of hours troubleshooting, convinced that it was a problem on site, we discovered that the EOIP tunnel just stopped passing traffic. Nothing had changed on those two routers for months but all of a sudden
just stopped working. The CCR has some other EOIP tunnels on it which were still working fine, but the 1100 was refusing to work.
We checked firewall rules, vlan ID's, tunnel ID's, everything, and there were no issues, the tunnel just refused to pass data.
I tried changing the tunnel ID and it made no difference, tried re-adding
I've seen something similar several years ago where IP address would simply stop working: an ordinary address on an interface would become unreachable, even from the localhost. The only way to recover would be to remove the address, then re-add. That was way back in some early v3 routerOS I think, and after upgrades, I just forgot that it ever happened - until now ;) So I guess I'm leaning toward software bug - what routerOS version is it? Apart from waiting for it to happen again and try to get some quick diagnostics knocked out (at least grap some pcap data) there's probably not much more to do than wait - and seek feedback from others, of course, for which I'm also interested to hear about! Cheers! Mike. they the
vlans to it, nothing, disabled and enabled it, nothing made a difference.
Out of desperation we created a new EOIP tunnel on that router with a new ID and pointed it to the existing EOIP interface at the other end and it came up straight away.
So apart from a big WTF, has anybody experienced this before ?
Regards Paul _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Paul Julian Sent: Tuesday, 10 May 2016 9:23 AM To: public@talk.mikrotik.com.au Subject: [MT-AU Public] EOIP Epic Fail
Hi guys, thought I would post this information to see if anybody had experienced the same issue.
We have an RB1100AHx2 in a data centre that connects to a CCR in another data centre, unfortunately we can't do QINQ between the DC's so we run an EOIP tunnel to connect two vlans for a customer through to there, all has been working fine for about 2 years.
The other day, just out of the blue, the customer said their link had gone down, after a number of hours troubleshooting, convinced that it was a problem on site, we discovered that the EOIP tunnel just stopped passing traffic. Nothing had changed on those two routers for months but all of a sudden
just stopped working. The CCR has some other EOIP tunnels on it which were still working fine, but the 1100 was refusing to work.
We checked firewall rules, vlan ID's, tunnel ID's, everything, and there were no issues, the tunnel just refused to pass data.
I tried changing the tunnel ID and it made no difference, tried re-adding
V3 and some V4 ROS variants would sometimes require an ip address or EoIP tunel deleted and re added for no apparent reason. We haven't noticed this behaviour in any of the recent versions though. We tend to try and run a c-tag(s) inside an s-tag where QinQ with all 0x8100 ether type fails across a third party network segment. Not because we have anything against EoIP, but the overhead is so much lower Kind regards Murray Southwell Tasmanet Pty Ltd Manager - Network Services murray@tasmanet.com.au 0361652511 0418145979 -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Mike Everest Sent: Tuesday, 10 May 2016 9:52 AM To: 'MikroTik Australia Public List' Subject: Re: [MT-AU Public] EOIP Epic Fail I've seen something similar several years ago where IP address would simply stop working: an ordinary address on an interface would become unreachable, even from the localhost. The only way to recover would be to remove the address, then re-add. That was way back in some early v3 routerOS I think, and after upgrades, I just forgot that it ever happened - until now ;) So I guess I'm leaning toward software bug - what routerOS version is it? Apart from waiting for it to happen again and try to get some quick diagnostics knocked out (at least grap some pcap data) there's probably not much more to do than wait - and seek feedback from others, of course, for which I'm also interested to hear about! Cheers! Mike. they the
vlans to it, nothing, disabled and enabled it, nothing made a difference.
Out of desperation we created a new EOIP tunnel on that router with a new ID and pointed it to the existing EOIP interface at the other end and it came up straight away.
So apart from a big WTF, has anybody experienced this before ?
Regards Paul _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Paul Julian Sent: Tuesday, 10 May 2016 9:23 AM To: public@talk.mikrotik.com.au Subject: [MT-AU Public] EOIP Epic Fail
Hi guys, thought I would post this information to see if anybody had experienced the same issue.
We have an RB1100AHx2 in a data centre that connects to a CCR in another data centre, unfortunately we can't do QINQ between the DC's so we run an EOIP tunnel to connect two vlans for a customer through to there, all has been working fine for about 2 years.
The other day, just out of the blue, the customer said their link had gone down, after a number of hours troubleshooting, convinced that it was a problem on site, we discovered that the EOIP tunnel just stopped passing traffic. Nothing had changed on those two routers for months but all of a sudden
just stopped working. The CCR has some other EOIP tunnels on it which were still working fine, but the 1100 was refusing to work.
We checked firewall rules, vlan ID's, tunnel ID's, everything, and there were no issues, the tunnel just refused to pass data.
I tried changing the tunnel ID and it made no difference, tried re-adding
Hi Mike, it's 6.27 on that router, all IP's respond to ping and everything else on that router is working 100% except the EOIP tunnel. That router does a lot of work and is one of our busier routers in the DC, it's such a strange thing that I can only put down to corruption or something. Regards Paul -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Mike Everest Sent: Tuesday, 10 May 2016 9:52 AM To: 'MikroTik Australia Public List' Subject: Re: [MT-AU Public] EOIP Epic Fail I've seen something similar several years ago where IP address would simply stop working: an ordinary address on an interface would become unreachable, even from the localhost. The only way to recover would be to remove the address, then re-add. That was way back in some early v3 routerOS I think, and after upgrades, I just forgot that it ever happened - until now ;) So I guess I'm leaning toward software bug - what routerOS version is it? Apart from waiting for it to happen again and try to get some quick diagnostics knocked out (at least grap some pcap data) there's probably not much more to do than wait - and seek feedback from others, of course, for which I'm also interested to hear about! Cheers! Mike. they the
vlans to it, nothing, disabled and enabled it, nothing made a difference.
Out of desperation we created a new EOIP tunnel on that router with a new ID and pointed it to the existing EOIP interface at the other end and it came up straight away.
So apart from a big WTF, has anybody experienced this before ?
Regards Paul _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
I've had a *lot* of trouble with EOIP tunnels randomly stopping working. It has got to the point that when an EOIP tunnel drops, I just create a new one and move the VLAN's across to the new one *first*, before I look for anything else. That said, I haven't had any issues in 4-5 months, so I assume whatever it was, was fixed somewhere around 6.33/6.34-ish? On 10 May 2016 at 09:22, Paul Julian <paul@oxygennetworks.com.au> wrote:
Hi guys, thought I would post this information to see if anybody had experienced the same issue.
We have an RB1100AHx2 in a data centre that connects to a CCR in another data centre, unfortunately we can't do QINQ between the DC's so we run an EOIP tunnel to connect two vlans for a customer through to there, all has been working fine for about 2 years.
The other day, just out of the blue, the customer said their link had gone down, after a number of hours troubleshooting, convinced that it was a problem on site, we discovered that the EOIP tunnel just stopped passing traffic. Nothing had changed on those two routers for months but all of a sudden they just stopped working. The CCR has some other EOIP tunnels on it which were still working fine, but the 1100 was refusing to work.
We checked firewall rules, vlan ID's, tunnel ID's, everything, and there were no issues, the tunnel just refused to pass data.
I tried changing the tunnel ID and it made no difference, tried re-adding the vlans to it, nothing, disabled and enabled it, nothing made a difference.
Out of desperation we created a new EOIP tunnel on that router with a new ID and pointed it to the existing EOIP interface at the other end and it came up straight away.
So apart from a big WTF, has anybody experienced this before ?
Regards Paul _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
-- Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder
Bit late to the party on this thread, but I just wanted to confirm I have seen this issue on occasion as well, mostly on later 5.2x and early 6.0x versions Same symptoms, tunnels would stop out of the blue, and the only thing that would bring them back is deleting and recreating them, same details and everything. Haven't seen this for a good while (9-12 months) so assume it was fixed. However, I'm beginning to notice another strange EOIP issue. If a tunnel is configured on a device that provides internet access through NAT, even if the tunnel is disabled, not a part of any bridge etc, it seems to interfere with client's internet access in a strange way, for example particular sites will not load, some elements don't display properly etc, very similar symptoms to a MTU issue. No traffic passes over the tunnel (as its disabled), and I do not see how it could be affecting traffic at all. Removing the tunnel from the router complete fixes the problem straight away. Is this something anyone else has seen or am I going crazy? -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Damien Gardner Jnr Sent: Tuesday, 10 May 2016 10:32 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] EOIP Epic Fail I've had a *lot* of trouble with EOIP tunnels randomly stopping working. It has got to the point that when an EOIP tunnel drops, I just create a new one and move the VLAN's across to the new one *first*, before I look for anything else. That said, I haven't had any issues in 4-5 months, so I assume whatever it was, was fixed somewhere around 6.33/6.34-ish? On 10 May 2016 at 09:22, Paul Julian <paul@oxygennetworks.com.au> wrote:
Hi guys, thought I would post this information to see if anybody had experienced the same issue.
We have an RB1100AHx2 in a data centre that connects to a CCR in another data centre, unfortunately we can't do QINQ between the DC's so we run an EOIP tunnel to connect two vlans for a customer through to there, all has been working fine for about 2 years.
The other day, just out of the blue, the customer said their link had gone down, after a number of hours troubleshooting, convinced that it was a problem on site, we discovered that the EOIP tunnel just stopped passing traffic. Nothing had changed on those two routers for months but all of a sudden they just stopped working. The CCR has some other EOIP tunnels on it which were still working fine, but the 1100 was refusing to work.
We checked firewall rules, vlan ID's, tunnel ID's, everything, and there were no issues, the tunnel just refused to pass data.
I tried changing the tunnel ID and it made no difference, tried re-adding the vlans to it, nothing, disabled and enabled it, nothing made a difference.
Out of desperation we created a new EOIP tunnel on that router with a new ID and pointed it to the existing EOIP interface at the other end and it came up straight away.
So apart from a big WTF, has anybody experienced this before ?
Regards Paul _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
-- Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Wow that sounds like a good one, I reckon there is some dodgy code somewhere that hooks EOIP into the bridge stuff in some occasions by accident, it's strange indeed ! Regards Paul -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Shaffi Khan Sent: Tuesday, 17 May 2016 5:12 PM To: MikroTik Australia Public List Subject: Re: [MT-AU Public] EOIP Epic Fail Bit late to the party on this thread, but I just wanted to confirm I have seen this issue on occasion as well, mostly on later 5.2x and early 6.0x versions Same symptoms, tunnels would stop out of the blue, and the only thing that would bring them back is deleting and recreating them, same details and everything. Haven't seen this for a good while (9-12 months) so assume it was fixed. However, I'm beginning to notice another strange EOIP issue. If a tunnel is configured on a device that provides internet access through NAT, even if the tunnel is disabled, not a part of any bridge etc, it seems to interfere with client's internet access in a strange way, for example particular sites will not load, some elements don't display properly etc, very similar symptoms to a MTU issue. No traffic passes over the tunnel (as its disabled), and I do not see how it could be affecting traffic at all. Removing the tunnel from the router complete fixes the problem straight away. Is this something anyone else has seen or am I going crazy? -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Damien Gardner Jnr Sent: Tuesday, 10 May 2016 10:32 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] EOIP Epic Fail I've had a *lot* of trouble with EOIP tunnels randomly stopping working. It has got to the point that when an EOIP tunnel drops, I just create a new one and move the VLAN's across to the new one *first*, before I look for anything else. That said, I haven't had any issues in 4-5 months, so I assume whatever it was, was fixed somewhere around 6.33/6.34-ish? On 10 May 2016 at 09:22, Paul Julian <paul@oxygennetworks.com.au> wrote:
Hi guys, thought I would post this information to see if anybody had experienced the same issue.
We have an RB1100AHx2 in a data centre that connects to a CCR in another data centre, unfortunately we can't do QINQ between the DC's so we run an EOIP tunnel to connect two vlans for a customer through to there, all has been working fine for about 2 years.
The other day, just out of the blue, the customer said their link had gone down, after a number of hours troubleshooting, convinced that it was a problem on site, we discovered that the EOIP tunnel just stopped passing traffic. Nothing had changed on those two routers for months but all of a sudden they just stopped working. The CCR has some other EOIP tunnels on it which were still working fine, but the 1100 was refusing to work.
We checked firewall rules, vlan ID's, tunnel ID's, everything, and there were no issues, the tunnel just refused to pass data.
I tried changing the tunnel ID and it made no difference, tried re-adding the vlans to it, nothing, disabled and enabled it, nothing made a difference.
Out of desperation we created a new EOIP tunnel on that router with a new ID and pointed it to the existing EOIP interface at the other end and it came up straight away.
So apart from a big WTF, has anybody experienced this before ?
Regards Paul _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
-- Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Not sure if it is related, but I have had issues in 6.2x releases where OpenVPN tunnels (client on Mikrotik, Linux with official OpenVPN on the server end) randomly drop and won't reconnect no matter what I do - but if I upgrade to a newer version (even without touching the config), they would come good again straight away. I've been running 6.34.2 for a while now without any issues. -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Paul Julian Sent: Tuesday, 10 May 2016 9:23 AM To: public@talk.mikrotik.com.au Subject: [MT-AU Public] EOIP Epic Fail Hi guys, thought I would post this information to see if anybody had experienced the same issue. We have an RB1100AHx2 in a data centre that connects to a CCR in another data centre, unfortunately we can't do QINQ between the DC's so we run an EOIP tunnel to connect two vlans for a customer through to there, all has been working fine for about 2 years. The other day, just out of the blue, the customer said their link had gone down, after a number of hours troubleshooting, convinced that it was a problem on site, we discovered that the EOIP tunnel just stopped passing traffic. Nothing had changed on those two routers for months but all of a sudden they just stopped working. The CCR has some other EOIP tunnels on it which were still working fine, but the 1100 was refusing to work. We checked firewall rules, vlan ID's, tunnel ID's, everything, and there were no issues, the tunnel just refused to pass data. I tried changing the tunnel ID and it made no difference, tried re-adding the vlans to it, nothing, disabled and enabled it, nothing made a difference. Out of desperation we created a new EOIP tunnel on that router with a new ID and pointed it to the existing EOIP interface at the other end and it came up straight away. So apart from a big WTF, has anybody experienced this before ? Regards Paul _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
participants (6)
-
Damien Gardner Jnr -
Mike Everest -
Murray Southwell -
Paul Julian -
Shaffi Khan -
Thomas Jackson