BGP with multiple global route tables on RouterOS
Hi all, We have previously used CCRs for BGP transit peers, but found that the delay in BGP with global routes updating the routing table was slower than we would like. This was mainly due to the fact that the BGP process only runs on a single thread so even a upgrading to a CCR1072 is not going to improve it. We built a couple of CHRs with iBGP between them. They perform better because each individual core is faster than the CCR cores, but they still really struggle when there are significant changes to the route table. One of the CPU cores sits at 100% for an extended period of time while the other core sits at almost nothing, and occasionally all of the BGP peers to drop their sessions from timeouts while it is in this state. As the peers re-establish, the route table has a lot more updates to process and so we get in a bit of a vicious circle until the it can manage to get through it all. Does anyone else have experience with global route tables (preferably multiple) on RouterOS? Does anyone have any advice on how we can increase the performance? Throwing more cores at the CHRs doesn't seem like it would have much benefit, but is something we have been considering. We are open to suggestions of alternative platforms to replace or integrate with the design. Waiting for RouterOS v7 which is meant to be more multithreaded might be an option if we had an actual ETA on it.... :) Regards, Philip Loenneker | Network Engineer | TasmaNet 40-50 Innovation Drive, Dowsing Point, Tas 7010, Australia P: 03 6165 2542 | M: 0404 097 816 philip.loenneker@tasmanet.com.au<mailto:philip.loenneker@tasmanet.com.au> www.tasmanet.com.au<http://www.tasmanet.com.au/>
Hi Philip, The protocol and route processing for all BGP peers runs within a single process. This means that no matter how many cores you throw at it, it will still run the all the BGP sessions and route processing on only 1 of them. This is not helped by the way RouterOS currently stores the routes in memory. The only way to speed this up is to use a processor with a higher performance per core, E3 Xeons with high clock speeds are the cheapest way to achieve this. In version 7, they are looking to split the individual peer sessions onto separate threads, with a separate processing thread as well as using proper RIB/FIB architecture and more efficient hashing of the stored routes. This should bring about significant improvements in route processing performance, but only time will tell... Our work around was to only take a default route from our main transit providers, with a seperate "domestic" table via another vlan. This means we only need to handle around 60,000 routes for Domestic and IX. We then steer traffic inbound from our international peers using pre-pending. This works for us as our traffic flow is mainly inbound, your situation may not be suited to this. Regards, Andrew On Thu, Jan 19, 2017 at 2:03 PM, Philip Loenneker <Philip.Loenneker@tasmanet.com.au> wrote:
Hi all,
We have previously used CCRs for BGP transit peers, but found that the delay in BGP with global routes updating the routing table was slower than we would like. This was mainly due to the fact that the BGP process only runs on a single thread so even a upgrading to a CCR1072 is not going to improve it.
We built a couple of CHRs with iBGP between them. They perform better because each individual core is faster than the CCR cores, but they still really struggle when there are significant changes to the route table. One of the CPU cores sits at 100% for an extended period of time while the other core sits at almost nothing, and occasionally all of the BGP peers to drop their sessions from timeouts while it is in this state. As the peers re-establish, the route table has a lot more updates to process and so we get in a bit of a vicious circle until the it can manage to get through it all.
Does anyone else have experience with global route tables (preferably multiple) on RouterOS? Does anyone have any advice on how we can increase the performance? Throwing more cores at the CHRs doesn't seem like it would have much benefit, but is something we have been considering. We are open to suggestions of alternative platforms to replace or integrate with the design.
Waiting for RouterOS v7 which is meant to be more multithreaded might be an option if we had an actual ETA on it.... :)
Regards, Philip Loenneker | Network Engineer | TasmaNet 40-50 Innovation Drive, Dowsing Point, Tas 7010, Australia P: 03 6165 2542 | M: 0404 097 816 philip.loenneker@tasmanet.com.au<mailto:philip.loenneker@tasmanet.com.au> www.tasmanet.com.au<http://www.tasmanet.com.au/>
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Andrew, Thanks for sharing your thoughts. Unfortunately I don't know if we can throw a powerful enough processor at it to help much. There seem to be a LOT of things we're waiting on in v7! :) We are working on reducing the number of routes we take in, but we're trying to work out how to do so effectively. We're considering options such as filtering out anything with an AS path length longer than 4 or just get rid of all /24 subnets, but we have to be careful about when we make those kind of changes due to the risk of BGP falling over. There are lots of suggestions around for using BIRD or QUAGGA or the like. I'm wondering if it's possible to have a system running one of those which peers with our upstreams and works out the best routes, then peers with our transit routers which only have to grab active routes and leave the original next-hop IP. I imagine we would either need an extra IP per provider for our BGP route server then, or each provider would have a static route to the BGP route server IP so they can communicate before the BGP session is established. But I am not confident that it will have any benefit over what we have, or over reducing the route tables, let alone the fact that it may not work at all. I just want RouterOS v7 already! :) Regards, Philip Loenneker | Network Engineer | TasmaNet -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Thrift Sent: Friday, 20 January 2017 3:03 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] BGP with multiple global route tables on RouterOS Hi Philip, The protocol and route processing for all BGP peers runs within a single process. This means that no matter how many cores you throw at it, it will still run the all the BGP sessions and route processing on only 1 of them. This is not helped by the way RouterOS currently stores the routes in memory. The only way to speed this up is to use a processor with a higher performance per core, E3 Xeons with high clock speeds are the cheapest way to achieve this. In version 7, they are looking to split the individual peer sessions onto separate threads, with a separate processing thread as well as using proper RIB/FIB architecture and more efficient hashing of the stored routes. This should bring about significant improvements in route processing performance, but only time will tell... Our work around was to only take a default route from our main transit providers, with a seperate "domestic" table via another vlan. This means we only need to handle around 60,000 routes for Domestic and IX. We then steer traffic inbound from our international peers using pre-pending. This works for us as our traffic flow is mainly inbound, your situation may not be suited to this. Regards, Andrew On Thu, Jan 19, 2017 at 2:03 PM, Philip Loenneker <Philip.Loenneker@tasmanet.com.au> wrote:
Hi all,
We have previously used CCRs for BGP transit peers, but found that the delay in BGP with global routes updating the routing table was slower than we would like. This was mainly due to the fact that the BGP process only runs on a single thread so even a upgrading to a CCR1072 is not going to improve it.
We built a couple of CHRs with iBGP between them. They perform better because each individual core is faster than the CCR cores, but they still really struggle when there are significant changes to the route table. One of the CPU cores sits at 100% for an extended period of time while the other core sits at almost nothing, and occasionally all of the BGP peers to drop their sessions from timeouts while it is in this state. As the peers re-establish, the route table has a lot more updates to process and so we get in a bit of a vicious circle until the it can manage to get through it all.
Does anyone else have experience with global route tables (preferably multiple) on RouterOS? Does anyone have any advice on how we can increase the performance? Throwing more cores at the CHRs doesn't seem like it would have much benefit, but is something we have been considering. We are open to suggestions of alternative platforms to replace or integrate with the design.
Waiting for RouterOS v7 which is meant to be more multithreaded might be an option if we had an actual ETA on it.... :)
Regards, Philip Loenneker | Network Engineer | TasmaNet 40-50 Innovation Drive, Dowsing Point, Tas 7010, Australia P: 03 6165 2542 | M: 0404 097 816 philip.loenneker@tasmanet.com.au<mailto:philip.loenneker@tasmanet.com. au> www.tasmanet.com.au<http://www.tasmanet.com.au/>
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
You could do something with bird/quagga and multi-hop BGP, e.g. Quagga peers through CCR to provider using multi-hop BGP (session is from Quagga to the Provider), quagga does route processing and then sends a subset of prefixes to the CCR with next-hop changed to the providers next-hop address. And yes, I want v7 too :) On Fri, Jan 20, 2017 at 6:09 PM, Philip Loenneker <Philip.Loenneker@tasmanet.com.au> wrote:
Hi Andrew,
Thanks for sharing your thoughts. Unfortunately I don't know if we can throw a powerful enough processor at it to help much. There seem to be a LOT of things we're waiting on in v7! :)
We are working on reducing the number of routes we take in, but we're trying to work out how to do so effectively. We're considering options such as filtering out anything with an AS path length longer than 4 or just get rid of all /24 subnets, but we have to be careful about when we make those kind of changes due to the risk of BGP falling over.
There are lots of suggestions around for using BIRD or QUAGGA or the like. I'm wondering if it's possible to have a system running one of those which peers with our upstreams and works out the best routes, then peers with our transit routers which only have to grab active routes and leave the original next-hop IP. I imagine we would either need an extra IP per provider for our BGP route server then, or each provider would have a static route to the BGP route server IP so they can communicate before the BGP session is established. But I am not confident that it will have any benefit over what we have, or over reducing the route tables, let alone the fact that it may not work at all.
I just want RouterOS v7 already! :)
Regards, Philip Loenneker | Network Engineer | TasmaNet
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Thrift Sent: Friday, 20 January 2017 3:03 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] BGP with multiple global route tables on RouterOS
Hi Philip,
The protocol and route processing for all BGP peers runs within a single process. This means that no matter how many cores you throw at it, it will still run the all the BGP sessions and route processing on only 1 of them. This is not helped by the way RouterOS currently stores the routes in memory. The only way to speed this up is to use a processor with a higher performance per core, E3 Xeons with high clock speeds are the cheapest way to achieve this.
In version 7, they are looking to split the individual peer sessions onto separate threads, with a separate processing thread as well as using proper RIB/FIB architecture and more efficient hashing of the stored routes. This should bring about significant improvements in route processing performance, but only time will tell...
Our work around was to only take a default route from our main transit providers, with a seperate "domestic" table via another vlan. This means we only need to handle around 60,000 routes for Domestic and IX. We then steer traffic inbound from our international peers using pre-pending. This works for us as our traffic flow is mainly inbound, your situation may not be suited to this.
Regards,
Andrew
On Thu, Jan 19, 2017 at 2:03 PM, Philip Loenneker <Philip.Loenneker@tasmanet.com.au> wrote:
Hi all,
We have previously used CCRs for BGP transit peers, but found that the delay in BGP with global routes updating the routing table was slower than we would like. This was mainly due to the fact that the BGP process only runs on a single thread so even a upgrading to a CCR1072 is not going to improve it.
We built a couple of CHRs with iBGP between them. They perform better because each individual core is faster than the CCR cores, but they still really struggle when there are significant changes to the route table. One of the CPU cores sits at 100% for an extended period of time while the other core sits at almost nothing, and occasionally all of the BGP peers to drop their sessions from timeouts while it is in this state. As the peers re-establish, the route table has a lot more updates to process and so we get in a bit of a vicious circle until the it can manage to get through it all.
Does anyone else have experience with global route tables (preferably multiple) on RouterOS? Does anyone have any advice on how we can increase the performance? Throwing more cores at the CHRs doesn't seem like it would have much benefit, but is something we have been considering. We are open to suggestions of alternative platforms to replace or integrate with the design.
Waiting for RouterOS v7 which is meant to be more multithreaded might be an option if we had an actual ETA on it.... :)
Regards, Philip Loenneker | Network Engineer | TasmaNet 40-50 Innovation Drive, Dowsing Point, Tas 7010, Australia P: 03 6165 2542 | M: 0404 097 816 philip.loenneker@tasmanet.com.au<mailto:philip.loenneker@tasmanet.com. au> www.tasmanet.com.au<http://www.tasmanet.com.au/>
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
That's the kind of thing I was wondering about. Has anyone done something like that? Multi-hop would still require some static routes so they could talk, which is not ideal, and may not be supported by upstream providers... -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Thrift Sent: Monday, 23 January 2017 9:46 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] BGP with multiple global route tables on RouterOS You could do something with bird/quagga and multi-hop BGP, e.g. Quagga peers through CCR to provider using multi-hop BGP (session is from Quagga to the Provider), quagga does route processing and then sends a subset of prefixes to the CCR with next-hop changed to the providers next-hop address. And yes, I want v7 too :) On Fri, Jan 20, 2017 at 6:09 PM, Philip Loenneker <Philip.Loenneker@tasmanet.com.au> wrote:
Hi Andrew,
Thanks for sharing your thoughts. Unfortunately I don't know if we can throw a powerful enough processor at it to help much. There seem to be a LOT of things we're waiting on in v7! :)
We are working on reducing the number of routes we take in, but we're trying to work out how to do so effectively. We're considering options such as filtering out anything with an AS path length longer than 4 or just get rid of all /24 subnets, but we have to be careful about when we make those kind of changes due to the risk of BGP falling over.
There are lots of suggestions around for using BIRD or QUAGGA or the like. I'm wondering if it's possible to have a system running one of those which peers with our upstreams and works out the best routes, then peers with our transit routers which only have to grab active routes and leave the original next-hop IP. I imagine we would either need an extra IP per provider for our BGP route server then, or each provider would have a static route to the BGP route server IP so they can communicate before the BGP session is established. But I am not confident that it will have any benefit over what we have, or over reducing the route tables, let alone the fact that it may not work at all.
I just want RouterOS v7 already! :)
Regards, Philip Loenneker | Network Engineer | TasmaNet
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Thrift Sent: Friday, 20 January 2017 3:03 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] BGP with multiple global route tables on RouterOS
Hi Philip,
The protocol and route processing for all BGP peers runs within a single process. This means that no matter how many cores you throw at it, it will still run the all the BGP sessions and route processing on only 1 of them. This is not helped by the way RouterOS currently stores the routes in memory. The only way to speed this up is to use a processor with a higher performance per core, E3 Xeons with high clock speeds are the cheapest way to achieve this.
In version 7, they are looking to split the individual peer sessions onto separate threads, with a separate processing thread as well as using proper RIB/FIB architecture and more efficient hashing of the stored routes. This should bring about significant improvements in route processing performance, but only time will tell...
Our work around was to only take a default route from our main transit providers, with a seperate "domestic" table via another vlan. This means we only need to handle around 60,000 routes for Domestic and IX. We then steer traffic inbound from our international peers using pre-pending. This works for us as our traffic flow is mainly inbound, your situation may not be suited to this.
Regards,
Andrew
On Thu, Jan 19, 2017 at 2:03 PM, Philip Loenneker <Philip.Loenneker@tasmanet.com.au> wrote:
Hi all,
We have previously used CCRs for BGP transit peers, but found that the delay in BGP with global routes updating the routing table was slower than we would like. This was mainly due to the fact that the BGP process only runs on a single thread so even a upgrading to a CCR1072 is not going to improve it.
We built a couple of CHRs with iBGP between them. They perform better because each individual core is faster than the CCR cores, but they still really struggle when there are significant changes to the route table. One of the CPU cores sits at 100% for an extended period of time while the other core sits at almost nothing, and occasionally all of the BGP peers to drop their sessions from timeouts while it is in this state. As the peers re-establish, the route table has a lot more updates to process and so we get in a bit of a vicious circle until the it can manage to get through it all.
Does anyone else have experience with global route tables (preferably multiple) on RouterOS? Does anyone have any advice on how we can increase the performance? Throwing more cores at the CHRs doesn't seem like it would have much benefit, but is something we have been considering. We are open to suggestions of alternative platforms to replace or integrate with the design.
Waiting for RouterOS v7 which is meant to be more multithreaded might be an option if we had an actual ETA on it.... :)
Regards, Philip Loenneker | Network Engineer | TasmaNet 40-50 Innovation Drive, Dowsing Point, Tas 7010, Australia P: 03 6165 2542 | M: 0404 097 816 philip.loenneker@tasmanet.com.au<mailto:philip.loenneker@tasmanet.com. au> www.tasmanet.com.au<http://www.tasmanet.com.au/>
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
As a thought (completely untested) - perhaps you could stick Quagga on a backend machine, and then use NAT on the CCR to make it transparent to your upstreams that you are running the BGP "off-board". Certainly not the cleanest solution though. And I'll gladly put in another vote for wanting V7! -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Philip Loenneker Sent: Monday, 23 January 2017 9:52 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] BGP with multiple global route tables on RouterOS That's the kind of thing I was wondering about. Has anyone done something like that? Multi-hop would still require some static routes so they could talk, which is not ideal, and may not be supported by upstream providers... -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Thrift Sent: Monday, 23 January 2017 9:46 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] BGP with multiple global route tables on RouterOS You could do something with bird/quagga and multi-hop BGP, e.g. Quagga peers through CCR to provider using multi-hop BGP (session is from Quagga to the Provider), quagga does route processing and then sends a subset of prefixes to the CCR with next-hop changed to the providers next-hop address. And yes, I want v7 too :) On Fri, Jan 20, 2017 at 6:09 PM, Philip Loenneker <Philip.Loenneker@tasmanet.com.au> wrote:
Hi Andrew,
Thanks for sharing your thoughts. Unfortunately I don't know if we can throw a powerful enough processor at it to help much. There seem to be a LOT of things we're waiting on in v7! :)
We are working on reducing the number of routes we take in, but we're trying to work out how to do so effectively. We're considering options such as filtering out anything with an AS path length longer than 4 or just get rid of all /24 subnets, but we have to be careful about when we make those kind of changes due to the risk of BGP falling over.
There are lots of suggestions around for using BIRD or QUAGGA or the like. I'm wondering if it's possible to have a system running one of those which peers with our upstreams and works out the best routes, then peers with our transit routers which only have to grab active routes and leave the original next-hop IP. I imagine we would either need an extra IP per provider for our BGP route server then, or each provider would have a static route to the BGP route server IP so they can communicate before the BGP session is established. But I am not confident that it will have any benefit over what we have, or over reducing the route tables, let alone the fact that it may not work at all.
I just want RouterOS v7 already! :)
Regards, Philip Loenneker | Network Engineer | TasmaNet
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Thrift Sent: Friday, 20 January 2017 3:03 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] BGP with multiple global route tables on RouterOS
Hi Philip,
The protocol and route processing for all BGP peers runs within a single process. This means that no matter how many cores you throw at it, it will still run the all the BGP sessions and route processing on only 1 of them. This is not helped by the way RouterOS currently stores the routes in memory. The only way to speed this up is to use a processor with a higher performance per core, E3 Xeons with high clock speeds are the cheapest way to achieve this.
In version 7, they are looking to split the individual peer sessions onto separate threads, with a separate processing thread as well as using proper RIB/FIB architecture and more efficient hashing of the stored routes. This should bring about significant improvements in route processing performance, but only time will tell...
Our work around was to only take a default route from our main transit providers, with a seperate "domestic" table via another vlan. This means we only need to handle around 60,000 routes for Domestic and IX. We then steer traffic inbound from our international peers using pre-pending. This works for us as our traffic flow is mainly inbound, your situation may not be suited to this.
Regards,
Andrew
On Thu, Jan 19, 2017 at 2:03 PM, Philip Loenneker <Philip.Loenneker@tasmanet.com.au> wrote:
Hi all,
We have previously used CCRs for BGP transit peers, but found that the delay in BGP with global routes updating the routing table was slower than we would like. This was mainly due to the fact that the BGP process only runs on a single thread so even a upgrading to a CCR1072 is not going to improve it.
We built a couple of CHRs with iBGP between them. They perform better because each individual core is faster than the CCR cores, but they still really struggle when there are significant changes to the route table. One of the CPU cores sits at 100% for an extended period of time while the other core sits at almost nothing, and occasionally all of the BGP peers to drop their sessions from timeouts while it is in this state. As the peers re-establish, the route table has a lot more updates to process and so we get in a bit of a vicious circle until the it can manage to get through it all.
Does anyone else have experience with global route tables (preferably multiple) on RouterOS? Does anyone have any advice on how we can increase the performance? Throwing more cores at the CHRs doesn't seem like it would have much benefit, but is something we have been considering. We are open to suggestions of alternative platforms to replace or integrate with the design.
Waiting for RouterOS v7 which is meant to be more multithreaded might be an option if we had an actual ETA on it.... :)
Regards, Philip Loenneker | Network Engineer | TasmaNet 40-50 Innovation Drive, Dowsing Point, Tas 7010, Australia P: 03 6165 2542 | M: 0404 097 816 philip.loenneker@tasmanet.com.au<mailto:philip.loenneker@tasmanet.com. au> www.tasmanet.com.au<http://www.tasmanet.com.au/>
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
I thought of NATing it too... but that could lead to a world of pain. I don't think I'd like to deal with that in a production environment. And it won't help for IPv6 peering, though they are generally smaller route tables anyway. -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Thomas Jackson Sent: Monday, 23 January 2017 9:57 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] BGP with multiple global route tables on RouterOS As a thought (completely untested) - perhaps you could stick Quagga on a backend machine, and then use NAT on the CCR to make it transparent to your upstreams that you are running the BGP "off-board". Certainly not the cleanest solution though. And I'll gladly put in another vote for wanting V7! -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Philip Loenneker Sent: Monday, 23 January 2017 9:52 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] BGP with multiple global route tables on RouterOS That's the kind of thing I was wondering about. Has anyone done something like that? Multi-hop would still require some static routes so they could talk, which is not ideal, and may not be supported by upstream providers... -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Thrift Sent: Monday, 23 January 2017 9:46 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] BGP with multiple global route tables on RouterOS You could do something with bird/quagga and multi-hop BGP, e.g. Quagga peers through CCR to provider using multi-hop BGP (session is from Quagga to the Provider), quagga does route processing and then sends a subset of prefixes to the CCR with next-hop changed to the providers next-hop address. And yes, I want v7 too :) On Fri, Jan 20, 2017 at 6:09 PM, Philip Loenneker <Philip.Loenneker@tasmanet.com.au> wrote:
Hi Andrew,
Thanks for sharing your thoughts. Unfortunately I don't know if we can throw a powerful enough processor at it to help much. There seem to be a LOT of things we're waiting on in v7! :)
We are working on reducing the number of routes we take in, but we're trying to work out how to do so effectively. We're considering options such as filtering out anything with an AS path length longer than 4 or just get rid of all /24 subnets, but we have to be careful about when we make those kind of changes due to the risk of BGP falling over.
There are lots of suggestions around for using BIRD or QUAGGA or the like. I'm wondering if it's possible to have a system running one of those which peers with our upstreams and works out the best routes, then peers with our transit routers which only have to grab active routes and leave the original next-hop IP. I imagine we would either need an extra IP per provider for our BGP route server then, or each provider would have a static route to the BGP route server IP so they can communicate before the BGP session is established. But I am not confident that it will have any benefit over what we have, or over reducing the route tables, let alone the fact that it may not work at all.
I just want RouterOS v7 already! :)
Regards, Philip Loenneker | Network Engineer | TasmaNet
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Thrift Sent: Friday, 20 January 2017 3:03 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] BGP with multiple global route tables on RouterOS
Hi Philip,
The protocol and route processing for all BGP peers runs within a single process. This means that no matter how many cores you throw at it, it will still run the all the BGP sessions and route processing on only 1 of them. This is not helped by the way RouterOS currently stores the routes in memory. The only way to speed this up is to use a processor with a higher performance per core, E3 Xeons with high clock speeds are the cheapest way to achieve this.
In version 7, they are looking to split the individual peer sessions onto separate threads, with a separate processing thread as well as using proper RIB/FIB architecture and more efficient hashing of the stored routes. This should bring about significant improvements in route processing performance, but only time will tell...
Our work around was to only take a default route from our main transit providers, with a seperate "domestic" table via another vlan. This means we only need to handle around 60,000 routes for Domestic and IX. We then steer traffic inbound from our international peers using pre-pending. This works for us as our traffic flow is mainly inbound, your situation may not be suited to this.
Regards,
Andrew
On Thu, Jan 19, 2017 at 2:03 PM, Philip Loenneker <Philip.Loenneker@tasmanet.com.au> wrote:
Hi all,
We have previously used CCRs for BGP transit peers, but found that the delay in BGP with global routes updating the routing table was slower than we would like. This was mainly due to the fact that the BGP process only runs on a single thread so even a upgrading to a CCR1072 is not going to improve it.
We built a couple of CHRs with iBGP between them. They perform better because each individual core is faster than the CCR cores, but they still really struggle when there are significant changes to the route table. One of the CPU cores sits at 100% for an extended period of time while the other core sits at almost nothing, and occasionally all of the BGP peers to drop their sessions from timeouts while it is in this state. As the peers re-establish, the route table has a lot more updates to process and so we get in a bit of a vicious circle until the it can manage to get through it all.
Does anyone else have experience with global route tables (preferably multiple) on RouterOS? Does anyone have any advice on how we can increase the performance? Throwing more cores at the CHRs doesn't seem like it would have much benefit, but is something we have been considering. We are open to suggestions of alternative platforms to replace or integrate with the design.
Waiting for RouterOS v7 which is meant to be more multithreaded might be an option if we had an actual ETA on it.... :)
Regards, Philip Loenneker | Network Engineer | TasmaNet 40-50 Innovation Drive, Dowsing Point, Tas 7010, Australia P: 03 6165 2542 | M: 0404 097 816 philip.loenneker@tasmanet.com.au<mailto:philip.loenneker@tasmanet.com. au> www.tasmanet.com.au<http://www.tasmanet.com.au/>
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
participants (3)
-
Andrew Thrift -
Philip Loenneker -
Thomas Jackson