NBNCo wireless service triggers traffic police, fiber services do not
Hi All! Earlier this week, I received a report from some guys using CCR to terminate ppp sessions over NBNCo tails and reporting that although fiber services worked perfectly, customers on wireless service continually hit traffic policer with result being download speeds about 1/2 of expected max. Again this morning, I received similar report from different operator :-( haven't confirmed what routerOS version for either of them yet.. Has anyone else seen this happen? Would setting output limit at client end ease that behaviour? Any/all comments, suggestions welcome! :-} Mike.
Hi Mike I may not be understanding this one correctly but on the WAN uplink (assuming we say have a 10/10 connection for example) try setting a queue (outgoing TX) with a Max of 10400 and a Limit of 9800 On 26 February 2015 at 11:53, Mike Everest <mike@duxtel.com> wrote:
Hi All!
Earlier this week, I received a report from some guys using CCR to terminate ppp sessions over NBNCo tails and reporting that although fiber services worked perfectly, customers on wireless service continually hit traffic policer with result being download speeds about 1/2 of expected max.
Again this morning, I received similar report from different operator :-( haven't confirmed what routerOS version for either of them yet..
Has anyone else seen this happen? Would setting output limit at client end ease that behaviour?
Any/all comments, suggestions welcome! :-}
Mike.
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
-- Best Regards Denis Hancock | Network Support | 1300 139 593 Skype denis.hancock.melbourne | http://www.samurai.com.au
Any time you hit a hard policer vs a shaping policer you'll see this sort of backoff in TCP connections; we see similar issues with Telstra policed services. Because the mikrotik queues are a bit more 'bouncy' than an ASIC controlled shaper it works best if you do the calculations to ensure that your local rate won't ever try and burst over the policed limit on the upstream. Good rule of thumb is to keep at 90% for services under 20Mbps and increase slowly up to 100Mbps (where you could conceivably increase to around 96% or 96Mbps). Remember also that if you're performing the rate limiting on the upstream side of the connection, you have more control over the max-limit of the traffic passing into the NBN 'pipe' but less control over the traffic coming back. This also plays into the mix, and may cause you to limit the inbound (upload from the user) traffic to a lower rate than you can deliver out (download to the user) from the CCR terminator. Another thing to look into is ensuring any custom queues aren't creating an artificial bufferbloat issue - http://en.wikipedia.org/wiki/Bufferbloat Food for thought. - Andrew On 26 February 2015 at 10:53, Mike Everest <mike@duxtel.com> wrote:
Hi All!
Earlier this week, I received a report from some guys using CCR to terminate ppp sessions over NBNCo tails and reporting that although fiber services worked perfectly, customers on wireless service continually hit traffic policer with result being download speeds about 1/2 of expected max.
Again this morning, I received similar report from different operator :-( haven't confirmed what routerOS version for either of them yet..
Has anyone else seen this happen? Would setting output limit at client end ease that behaviour?
Any/all comments, suggestions welcome! :-}
Mike.
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
With NBN connections you don't need to limit the CPE upload, NBN NTD does that for you, however you are correct Andrew with the limits, we have worked with about 3000 NBN connections now with PPPOE, just remember Mike that the speed NBN advertise is the L2 speed, so actual speed is less PPP overheads, Ethernet overheads etc. Andrews suggestions on figures are also almost identical to how we approach it, due to Mikrotik not controlling burst quickly enough we don't do any bursting and just set the Limit-at and Max-Limit the same with no burst or burst time, yes the customer gets a slightly slower service but it's reliable and works well. We also create a custom PFIFO queue type, the queue length should be from memory 3ms worth of data, so however many packets can pass through the connection at full speed for that period of time, then subtract about 10%, interestingly we use the same queues for MEF Ethernet services also and that theory works well. As a general "works for everything" rule I find that a queue length of 200 packets seems to work pretty well for most speed classes.
From what we have found a PPPOE Server MTU or 1492 works well too, but that could depend on how the CVC is getting delivered to you and how many vlans you have along the way.
Also as a final note, the slower services back off hard, so expect a 50% performance hit if queues aren't right, the bigger the difference between the interface speed and the service speed the harder they slow down when they are configured wrong. Regards Paul -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Cox Sent: Thursday, 26 February 2015 12:20 PM To: MikroTik Australia Public List Subject: Re: [MT-AU Public] NBNCo wireless service triggers traffic police, fiber services do not Any time you hit a hard policer vs a shaping policer you'll see this sort of backoff in TCP connections; we see similar issues with Telstra policed services. Because the mikrotik queues are a bit more 'bouncy' than an ASIC controlled shaper it works best if you do the calculations to ensure that your local rate won't ever try and burst over the policed limit on the upstream. Good rule of thumb is to keep at 90% for services under 20Mbps and increase slowly up to 100Mbps (where you could conceivably increase to around 96% or 96Mbps). Remember also that if you're performing the rate limiting on the upstream side of the connection, you have more control over the max-limit of the traffic passing into the NBN 'pipe' but less control over the traffic coming back. This also plays into the mix, and may cause you to limit the inbound (upload from the user) traffic to a lower rate than you can deliver out (download to the user) from the CCR terminator. Another thing to look into is ensuring any custom queues aren't creating an artificial bufferbloat issue - http://en.wikipedia.org/wiki/Bufferbloat Food for thought. - Andrew On 26 February 2015 at 10:53, Mike Everest <mike@duxtel.com> wrote:
Hi All!
Earlier this week, I received a report from some guys using CCR to terminate ppp sessions over NBNCo tails and reporting that although fiber services worked perfectly, customers on wireless service continually hit traffic policer with result being download speeds about 1/2 of expected max.
Again this morning, I received similar report from different operator :-( haven't confirmed what routerOS version for either of them yet..
Has anyone else seen this happen? Would setting output limit at client end ease that behaviour?
Any/all comments, suggestions welcome! :-}
Mike.
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Excellent stuff Paul :-) The only other NBN gotcha that I'm aware of is that packets re-tagged with different DSCP values other than TC4 are dropped. Source: http://forums.whirlpool.net.au/archive/1978759 (and a personal source who ran into the same problem). - Andrew On 26 February 2015 at 11:33, Paul Julian <paul@oxygennetworks.com.au> wrote:
With NBN connections you don't need to limit the CPE upload, NBN NTD does that for you, however you are correct Andrew with the limits, we have worked with about 3000 NBN connections now with PPPOE, just remember Mike that the speed NBN advertise is the L2 speed, so actual speed is less PPP overheads, Ethernet overheads etc.
Andrews suggestions on figures are also almost identical to how we approach it, due to Mikrotik not controlling burst quickly enough we don't do any bursting and just set the Limit-at and Max-Limit the same with no burst or burst time, yes the customer gets a slightly slower service but it's reliable and works well.
We also create a custom PFIFO queue type, the queue length should be from memory 3ms worth of data, so however many packets can pass through the connection at full speed for that period of time, then subtract about 10%, interestingly we use the same queues for MEF Ethernet services also and that theory works well.
As a general "works for everything" rule I find that a queue length of 200 packets seems to work pretty well for most speed classes.
From what we have found a PPPOE Server MTU or 1492 works well too, but that could depend on how the CVC is getting delivered to you and how many vlans you have along the way.
Also as a final note, the slower services back off hard, so expect a 50% performance hit if queues aren't right, the bigger the difference between the interface speed and the service speed the harder they slow down when they are configured wrong.
Regards Paul
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Cox Sent: Thursday, 26 February 2015 12:20 PM To: MikroTik Australia Public List Subject: Re: [MT-AU Public] NBNCo wireless service triggers traffic police, fiber services do not
Any time you hit a hard policer vs a shaping policer you'll see this sort of backoff in TCP connections; we see similar issues with Telstra policed services.
Because the mikrotik queues are a bit more 'bouncy' than an ASIC controlled shaper it works best if you do the calculations to ensure that your local rate won't ever try and burst over the policed limit on the upstream.
Good rule of thumb is to keep at 90% for services under 20Mbps and increase slowly up to 100Mbps (where you could conceivably increase to around 96% or 96Mbps).
Remember also that if you're performing the rate limiting on the upstream side of the connection, you have more control over the max-limit of the traffic passing into the NBN 'pipe' but less control over the traffic coming back. This also plays into the mix, and may cause you to limit the inbound (upload from the user) traffic to a lower rate than you can deliver out (download to the user) from the CCR terminator.
Another thing to look into is ensuring any custom queues aren't creating an artificial bufferbloat issue - http://en.wikipedia.org/wiki/Bufferbloat
Food for thought.
- Andrew
On 26 February 2015 at 10:53, Mike Everest <mike@duxtel.com> wrote:
Hi All!
Earlier this week, I received a report from some guys using CCR to terminate ppp sessions over NBNCo tails and reporting that although fiber services worked perfectly, customers on wireless service continually hit traffic policer with result being download speeds about 1/2 of expected max.
Again this morning, I received similar report from different operator :-( haven't confirmed what routerOS version for either of them yet..
Has anyone else seen this happen? Would setting output limit at client end ease that behaviour?
Any/all comments, suggestions welcome! :-}
Mike.
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Woah a walk down memory lane... -TiMOiD
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Cox Sent: Thursday, 26 February 2015 1:40 PM To: MikroTik Australia Public List Subject: Re: [MT-AU Public] NBNCo wireless service triggers traffic police, fiber services do not
Excellent stuff Paul :-)
The only other NBN gotcha that I'm aware of is that packets re-tagged with different DSCP values other than TC4 are dropped. Source: http://forums.whirlpool.net.au/archive/1978759 (and a personal source who ran into the same problem).
- Andrew
On 26 February 2015 at 11:33, Paul Julian <paul@oxygennetworks.com.au> wrote:
With NBN connections you don't need to limit the CPE upload, NBN NTD does that for you, however you are correct Andrew with the limits, we have worked with about 3000 NBN connections now with PPPOE, just remember Mike that the speed NBN advertise is the L2 speed, so actual speed is less PPP overheads, Ethernet overheads etc.
Andrews suggestions on figures are also almost identical to how we approach it, due to Mikrotik not controlling burst quickly enough we don't do any bursting and just set the Limit-at and Max-Limit the same with no burst or burst time, yes the customer gets a slightly slower service but it's reliable and works well.
We also create a custom PFIFO queue type, the queue length should be from memory 3ms worth of data, so however many packets can pass through the connection at full speed for that period of time, then subtract about 10%, interestingly we use the same queues for MEF Ethernet services also and that theory works well.
As a general "works for everything" rule I find that a queue length of 200 packets seems to work pretty well for most speed classes.
From what we have found a PPPOE Server MTU or 1492 works well too, but that could depend on how the CVC is getting delivered to you and how many vlans you have along the way.
Also as a final note, the slower services back off hard, so expect a 50% performance hit if queues aren't right, the bigger the difference between the interface speed and the service speed the harder they slow down when they are configured wrong.
Regards Paul
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Cox Sent: Thursday, 26 February 2015 12:20 PM To: MikroTik Australia Public List Subject: Re: [MT-AU Public] NBNCo wireless service triggers traffic police, fiber services do not
Any time you hit a hard policer vs a shaping policer you'll see this sort of backoff in TCP connections; we see similar issues with Telstra policed services.
Because the mikrotik queues are a bit more 'bouncy' than an ASIC controlled shaper it works best if you do the calculations to ensure that your local rate won't ever try and burst over the policed limit on the upstream.
Good rule of thumb is to keep at 90% for services under 20Mbps and increase slowly up to 100Mbps (where you could conceivably increase to around 96% or 96Mbps).
Remember also that if you're performing the rate limiting on the upstream side of the connection, you have more control over the max-limit of the traffic passing into the NBN 'pipe' but less control over the traffic coming back. This also plays into the mix, and may cause you to limit the inbound (upload from the user) traffic to a lower rate than you can deliver out (download to the user) from the CCR terminator.
Another thing to look into is ensuring any custom queues aren't creating an artificial bufferbloat issue - http://en.wikipedia.org/wiki/Bufferbloat
Food for thought.
- Andrew
On 26 February 2015 at 10:53, Mike Everest <mike@duxtel.com> wrote:
Hi All!
Earlier this week, I received a report from some guys using CCR to terminate ppp sessions over NBNCo tails and reporting that although fiber services worked perfectly, customers on wireless service continually hit traffic policer with result being download speeds about 1/2 of expected max.
Again this morning, I received similar report from different operator :-( haven't confirmed what routerOS version for either of them yet..
Has anyone else seen this happen? Would setting output limit at client end ease that behaviour?
Any/all comments, suggestions welcome! :-}
Mike.
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Cox Sent: Thursday, 26 February 2015 2:40 PM To: MikroTik Australia Public List Subject: Re: [MT-AU Public] NBNCo wireless service triggers traffic
fiber services do not
Excellent stuff Paul :-)
The only other NBN gotcha that I'm aware of is that packets re-tagged with different DSCP values other than TC4 are dropped. Source: http://forums.whirlpool.net.au/archive/1978759 (and a personal source who ran into the same problem).
- Andrew
On 26 February 2015 at 11:33, Paul Julian <paul@oxygennetworks.com.au> wrote:
With NBN connections you don't need to limit the CPE upload, NBN NTD does that for you, however you are correct Andrew with the limits, we have worked with about 3000 NBN connections now with PPPOE, just remember Mike that the speed NBN advertise is the L2 speed, so actual speed is less PPP overheads, Ethernet overheads etc.
Andrews suggestions on figures are also almost identical to how we approach it, due to Mikrotik not controlling burst quickly enough we don't do any bursting and just set the Limit-at and Max-Limit the same with no burst or burst time, yes the customer gets a slightly slower service but it's reliable and works well.
We also create a custom PFIFO queue type, the queue length should be from memory 3ms worth of data, so however many packets can pass through the connection at full speed for that period of time, then subtract about 10%, interestingly we use the same queues for MEF Ethernet services also and that theory works well.
As a general "works for everything" rule I find that a queue length of 200 packets seems to work pretty well for most speed classes.
From what we have found a PPPOE Server MTU or 1492 works well too, but that could depend on how the CVC is getting delivered to you and how many vlans you have along the way.
Also as a final note, the slower services back off hard, so expect a 50% performance hit if queues aren't right, the bigger the difference between the interface speed and the service speed the harder they slow down when they are configured wrong.
Regards Paul
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Cox Sent: Thursday, 26 February 2015 12:20 PM To: MikroTik Australia Public List Subject: Re: [MT-AU Public] NBNCo wireless service triggers traffic police, fiber services do not
Any time you hit a hard policer vs a shaping policer you'll see this sort of backoff in TCP connections; we see similar issues with Telstra policed services.
Because the mikrotik queues are a bit more 'bouncy' than an ASIC controlled shaper it works best if you do the calculations to ensure that your local rate won't ever try and burst over the policed limit on the upstream.
Good rule of thumb is to keep at 90% for services under 20Mbps and increase slowly up to 100Mbps (where you could conceivably increase to around 96% or 96Mbps).
Remember also that if you're performing the rate limiting on the upstream side of the connection, you have more control over the max-limit of the traffic passing into the NBN 'pipe' but less control over the traffic coming back. This also plays into the mix, and may cause you to limit the inbound (upload from the user) traffic to a lower rate than you can deliver out (download to the user) from the CCR terminator.
Another thing to look into is ensuring any custom queues aren't creating an artificial bufferbloat issue - http://en.wikipedia.org/wiki/Bufferbloat
Food for thought.
- Andrew
On 26 February 2015 at 10:53, Mike Everest <mike@duxtel.com> wrote:
Hi All!
Earlier this week, I received a report from some guys using CCR to terminate ppp sessions over NBNCo tails and reporting that although fiber services worked perfectly, customers on wireless service continually hit traffic policer with result being download speeds about 1/2 of expected max.
Again this morning, I received similar report from different operator :-( haven't confirmed what routerOS version for either of
Yoiks! I didn't know that :-o Useful to keep in mind! (I wonder if I will? ;) Cheers! police, them
yet..
Has anyone else seen this happen? Would setting output limit at client end ease that behaviour?
Any/all comments, suggestions welcome! :-}
Mike.
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
participants (5)
-
Andrew Cox -
Denis Hancock -
Mike Everest -
Paul Julian -
Tim Warnock