BGP with ISP and IXP traffic variation
Hi All, I have just completed implementing BGP with Megaport and our ISP. I am seeing an usual pattern only with my outbound traffic. See https://malfunction.faed.name/ for a graph. I can't easily see what 1000's of users are doing but it seems like some routes are changing between ISP and IXP, then changing back causing this pattern. Can anyone suggest how to investigate this further? Or perhaps this is normal if anyone has experience in this. Thanks in advance, Mal *Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au Av-Comm Office: +61 2 9939 4377 Mobile+61 424 957 053 Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au [image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.
Hi Malcolm, Since this is outbound, it is the routes on your device that define where the traffic goes. Have you checked the uptime of the BGP peers to ensure they aren't dropping for any reason? If you view your route list for a few minutes, do you see lots of fluctuations of which routes are active? If, like most organisations, you have higher capacity on an IX than you have on your IP transit, and you want to ensure that your traffic prefers that path where possible, the easiest way to prefer that path is to set the LOCAL PREF of the routes learned via Megaport to anything above 100 (the default). This value propagates via iBGP peers so if you have multiple POPs, they are preferred across your whole network. If you don't want that, then other settings such as prepending routes learned via IP transit may be more practical. Regards, Philip Loenneker | Senior Network Engineer | TasmaNet -----Original Message----- From: Public <public-bounces@talk.mikrotik.com.au> On Behalf Of Malcolm Faed Sent: Friday, 6 September 2019 6:39 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: [MT-AU Public] BGP with ISP and IXP traffic variation Hi All, I have just completed implementing BGP with Megaport and our ISP. I am seeing an usual pattern only with my outbound traffic. See https://malfunction.faed.name/ for a graph. I can't easily see what 1000's of users are doing but it seems like some routes are changing between ISP and IXP, then changing back causing this pattern. Can anyone suggest how to investigate this further? Or perhaps this is normal if anyone has experience in this. Thanks in advance, Mal *Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au Av-Comm Office: +61 2 9939 4377 Mobile+61 424 957 053 Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au [image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Thanks Philip, The route numbers are not changing significantly +-10, and the peer uptime has not reset. I think then the solution for me then would be to configure the Megaport In filter as follows: *add action=accept chain="In_Filter_Megaport IX" comment="Set LOCAL PREF to 200 to favour the IXP routes over ISP" set-bgp-local-pref=200* Would you happen to know if this would take immediate affect, or only on new routes learned? Thanks and regards, *Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au Av-Comm Office: +61 2 9939 4377 Mobile+61 424 957 053 Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au [image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. On Mon, 9 Sep 2019 at 09:19, Philip Loenneker < Philip.Loenneker@tasmanet.com.au> wrote:
Hi Malcolm,
Since this is outbound, it is the routes on your device that define where the traffic goes. Have you checked the uptime of the BGP peers to ensure they aren't dropping for any reason? If you view your route list for a few minutes, do you see lots of fluctuations of which routes are active?
If, like most organisations, you have higher capacity on an IX than you have on your IP transit, and you want to ensure that your traffic prefers that path where possible, the easiest way to prefer that path is to set the LOCAL PREF of the routes learned via Megaport to anything above 100 (the default). This value propagates via iBGP peers so if you have multiple POPs, they are preferred across your whole network. If you don't want that, then other settings such as prepending routes learned via IP transit may be more practical.
Regards, Philip Loenneker | Senior Network Engineer | TasmaNet
-----Original Message----- From: Public <public-bounces@talk.mikrotik.com.au> On Behalf Of Malcolm Faed Sent: Friday, 6 September 2019 6:39 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: [MT-AU Public] BGP with ISP and IXP traffic variation
Hi All,
I have just completed implementing BGP with Megaport and our ISP.
I am seeing an usual pattern only with my outbound traffic. See https://malfunction.faed.name/ for a graph.
I can't easily see what 1000's of users are doing but it seems like some routes are changing between ISP and IXP, then changing back causing this pattern.
Can anyone suggest how to investigate this further? Or perhaps this is normal if anyone has experience in this.
Thanks in advance, Mal
*Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm
Office: +61 2 9939 4377 Mobile+61 424 957 053
Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au
[image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/>
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Malcolm, It may be worth running an ongoing traceroute to somewhere to see if the route changes at any particular interval - but if you have connection tracking enabled, that may make the routes persist per session regardless of the routes changing. If you don't need it, it's generally a good idea to turn off connection tracking to improve performance, but depending on your firewall/mangle/NAT rules you may not have that option. Adding the route filter rule you suggested will work, and will apply to all existing routes too - you won't need to drop the peer or anything like that. However, once you add the rule, it will add it at the end of the list by default. If you want to move it, you will need to disable it, then move it, then enable it - if you just move it, it may not function as expected. Also be mindful that you have "accept" as the action - if you have other rules you want to have apply too, you could put this new rule at the top of the list with an action of "passthrough" so that you don't lose your existing functionality. Regards, Philip Loenneker | Senior Network Engineer | TasmaNet -----Original Message----- From: Public <public-bounces@talk.mikrotik.com.au> On Behalf Of Malcolm Faed Sent: Monday, 9 September 2019 10:41 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] BGP with ISP and IXP traffic variation Thanks Philip, The route numbers are not changing significantly +-10, and the peer uptime has not reset. I think then the solution for me then would be to configure the Megaport In filter as follows: *add action=accept chain="In_Filter_Megaport IX" comment="Set LOCAL PREF to 200 to favour the IXP routes over ISP" set-bgp-local-pref=200* Would you happen to know if this would take immediate affect, or only on new routes learned? Thanks and regards, *Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au Av-Comm Office: +61 2 9939 4377 Mobile+61 424 957 053 Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au [image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. On Mon, 9 Sep 2019 at 09:19, Philip Loenneker < Philip.Loenneker@tasmanet.com.au> wrote:
Hi Malcolm,
Since this is outbound, it is the routes on your device that define where the traffic goes. Have you checked the uptime of the BGP peers to ensure they aren't dropping for any reason? If you view your route list for a few minutes, do you see lots of fluctuations of which routes are active?
If, like most organisations, you have higher capacity on an IX than you have on your IP transit, and you want to ensure that your traffic prefers that path where possible, the easiest way to prefer that path is to set the LOCAL PREF of the routes learned via Megaport to anything above 100 (the default). This value propagates via iBGP peers so if you have multiple POPs, they are preferred across your whole network. If you don't want that, then other settings such as prepending routes learned via IP transit may be more practical.
Regards, Philip Loenneker | Senior Network Engineer | TasmaNet
-----Original Message----- From: Public <public-bounces@talk.mikrotik.com.au> On Behalf Of Malcolm Faed Sent: Friday, 6 September 2019 6:39 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: [MT-AU Public] BGP with ISP and IXP traffic variation
Hi All,
I have just completed implementing BGP with Megaport and our ISP.
I am seeing an usual pattern only with my outbound traffic. See https://malfunction.faed.name/ for a graph.
I can't easily see what 1000's of users are doing but it seems like some routes are changing between ISP and IXP, then changing back causing this pattern.
Can anyone suggest how to investigate this further? Or perhaps this is normal if anyone has experience in this.
Thanks in advance, Mal
*Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm
Office: +61 2 9939 4377 Mobile+61 424 957 053
Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au
[image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/>
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hey Malcolm, If you have the time/resources it might be interesting to set up a flow monitor, I have been using Elastiflow https://github.com/robcowart/elastiflow You should be able to log the flows and see which IP/ASN they're destined for as well as the interface (I don't think mikrotik includes interface names, just a numerical id starting 0 from memory) that they leave the router on. Two things that come to mind are CDN's that might try and influence your routing to load balance (via MED or more specific routes), being able to see more details on the traffic flows should get you a better idea of what prefixes are actually changing Cheers, Kyle | kyle pharo | CTO | nz +64 21 064 6723 | | kyle@tangelo.com | www.tangelo.com | | https://www.peeringdb.com/net/10453 | -----Original Message----- From: Public <public-bounces@talk.mikrotik.com.au> On Behalf Of Malcolm Faed Sent: Friday, 6 September 2019 20:39 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: [MT-AU Public] BGP with ISP and IXP traffic variation Hi All, I have just completed implementing BGP with Megaport and our ISP. I am seeing an usual pattern only with my outbound traffic. See https://malfunction.faed.name/ for a graph. I can't easily see what 1000's of users are doing but it seems like some routes are changing between ISP and IXP, then changing back causing this pattern. Can anyone suggest how to investigate this further? Or perhaps this is normal if anyone has experience in this. Thanks in advance, Mal *Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au Av-Comm Office: +61 2 9939 4377 Mobile+61 424 957 053 Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au [image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Thanks Philip and Kyle, Adding the filter rule (with passthrough) has certainly helped. The bias is more toward megaport by about 15Mbps (Thanks Philip! If we meet at the next MUM I will buy you a beer.) Interestingly I am still seeing 10-15Mbps oscillation between the ISP and Megaport. Picture here... https://malfunction.faed.name/2019/09/outbound-bgp-traffic-oscillation.html I had disabled connection tracking prior to implementing the IX and moved my NAT clients etc onto a separate router. For this to occur some routes must be changing to / from the ISP and IXP. I know it is not much traffic in the greater scheme of things, but would be nice to understand it further. @Kyle I will look into flow monitoring. It has been on my list of things to do! Regards, *Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au Av-Comm Office: +61 2 9939 4377 Mobile+61 424 957 053 Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au [image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. On Mon, 9 Sep 2019 at 13:29, Kyle Pharo <kyle@tangelo.com> wrote:
Hey Malcolm,
If you have the time/resources it might be interesting to set up a flow monitor, I have been using Elastiflow https://github.com/robcowart/elastiflow
You should be able to log the flows and see which IP/ASN they're destined for as well as the interface (I don't think mikrotik includes interface names, just a numerical id starting 0 from memory) that they leave the router on.
Two things that come to mind are CDN's that might try and influence your routing to load balance (via MED or more specific routes), being able to see more details on the traffic flows should get you a better idea of what prefixes are actually changing
Cheers,
Kyle
| kyle pharo | CTO | nz +64 21 064 6723 | | kyle@tangelo.com | www.tangelo.com | | https://www.peeringdb.com/net/10453 |
-----Original Message----- From: Public <public-bounces@talk.mikrotik.com.au> On Behalf Of Malcolm Faed Sent: Friday, 6 September 2019 20:39 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: [MT-AU Public] BGP with ISP and IXP traffic variation
Hi All,
I have just completed implementing BGP with Megaport and our ISP.
I am seeing an usual pattern only with my outbound traffic. See https://malfunction.faed.name/ for a graph.
I can't easily see what 1000's of users are doing but it seems like some routes are changing between ISP and IXP, then changing back causing this pattern.
Can anyone suggest how to investigate this further? Or perhaps this is normal if anyone has experience in this.
Thanks in advance, Mal
*Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm
Office: +61 2 9939 4377 Mobile+61 424 957 053
Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au
[image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/>
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Malcolm, You're welcome :) RouterOS does have debug options for both BGP and Route - you may find that enabling them will reveal any changes to the route table that could be causing the behaviour you're seeing. Regards, Philip Loenneker | Senior Network Engineer | TasmaNet -----Original Message----- From: Public <public-bounces@talk.mikrotik.com.au> On Behalf Of Malcolm Faed Sent: Tuesday, 10 September 2019 12:20 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] BGP with ISP and IXP traffic variation Thanks Philip and Kyle, Adding the filter rule (with passthrough) has certainly helped. The bias is more toward megaport by about 15Mbps (Thanks Philip! If we meet at the next MUM I will buy you a beer.) Interestingly I am still seeing 10-15Mbps oscillation between the ISP and Megaport. Picture here... https://malfunction.faed.name/2019/09/outbound-bgp-traffic-oscillation.html I had disabled connection tracking prior to implementing the IX and moved my NAT clients etc onto a separate router. For this to occur some routes must be changing to / from the ISP and IXP. I know it is not much traffic in the greater scheme of things, but would be nice to understand it further. @Kyle I will look into flow monitoring. It has been on my list of things to do! Regards, *Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au Av-Comm Office: +61 2 9939 4377 Mobile+61 424 957 053 Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au [image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. On Mon, 9 Sep 2019 at 13:29, Kyle Pharo <kyle@tangelo.com> wrote:
Hey Malcolm,
If you have the time/resources it might be interesting to set up a flow monitor, I have been using Elastiflow https://github.com/robcowart/elastiflow
You should be able to log the flows and see which IP/ASN they're destined for as well as the interface (I don't think mikrotik includes interface names, just a numerical id starting 0 from memory) that they leave the router on.
Two things that come to mind are CDN's that might try and influence your routing to load balance (via MED or more specific routes), being able to see more details on the traffic flows should get you a better idea of what prefixes are actually changing
Cheers,
Kyle
| kyle pharo | CTO | nz +64 21 064 6723 | | kyle@tangelo.com | www.tangelo.com | | https://www.peeringdb.com/net/10453 |
-----Original Message----- From: Public <public-bounces@talk.mikrotik.com.au> On Behalf Of Malcolm Faed Sent: Friday, 6 September 2019 20:39 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: [MT-AU Public] BGP with ISP and IXP traffic variation
Hi All,
I have just completed implementing BGP with Megaport and our ISP.
I am seeing an usual pattern only with my outbound traffic. See https://malfunction.faed.name/ for a graph.
I can't easily see what 1000's of users are doing but it seems like some routes are changing between ISP and IXP, then changing back causing this pattern.
Can anyone suggest how to investigate this further? Or perhaps this is normal if anyone has experience in this.
Thanks in advance, Mal
*Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm
Office: +61 2 9939 4377 Mobile+61 424 957 053
Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au
[image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/>
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
participants (3)
-
Kyle Pharo -
Malcolm Faed -
Philip Loenneker