FW: CCR performance testing again
Try again san's pictures :) From: Alex Samad - Yieldbroker [mailto:Alex.Samad@yieldbroker.com] Sent: Friday, 22 January 2016 1:50 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again HI Thought I would put the commands I am using [root@alextest1 ~]# iperf -c 192.168.213.52 -i 30 -t 60 -u -l 12k -b 2 -> packets leave eth0 Goto ccr [root@alextest2 ~]# iperf -s -u -l 12k -> come in eth1 on here ------------------------------------------------------------ Client connecting to 192.168.213.52, UDP port 5001 Sending 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 10.172.203.51 port 59192 connected with 192.168.213.52 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-30.0 sec 7.00 GBytes 2.01 Gbits/sec [ 3] 30.0-60.0 sec 7.01 GBytes 2.01 Gbits/sec [ 3] 0.0-60.0 sec 14.0 GBytes 2.01 Gbits/sec [ 3] Sent 1224310 datagrams [ 3] Server Report: [ 3] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 3] 0.0-60.0 sec 327413 datagrams received out-of-order [root@alextest1 ~]# At 2G cpu usage Picture of low cpu usage! 3g - 99% Picture of high cpu usage ... >98% Somebody on the list was suggesting maybe a LACP issue . haven't had an opportunity to test that. ------------------------------------------------------------ Server listening on UDP port 5001 Receiving 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 52469 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-60.0 sec 6.86 GBytes 981 Mbits/sec 0.671 ms 625418/1224436 (51%) [ 3] 0.0-60.0 sec 322936 datagrams received out-of-order [ 3] WARNING: ack of last datagram failed after 10 tries. [ 4] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 59192 [ 4] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 4] 0.0-60.0 sec 327413 datagrams received out-of-order [ 4] WARNING: ack of last datagram failed after 10 tries. -----Original Message----- From: Alex Samad - Yieldbroker Sent: Friday, 22 January 2016 12:28 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again No Same vlan, different ip network. Why would it be sending redirects. Src is 192.168.213.52 dst is 10.172.213.51. Just because they are the same vlan doesn't allow them to talk directly (normally) Packet path is 192.168.213.52 -> 192.168.213.2 (ccr) -> 10.172.213.2 (ccr - same interface) - > 10.172.213.51 I have setup another test such that it doesn't do this. Instead in one interface and out another interface ... Again > 3G floods the CPU :(..... -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Damien Gardner Jnr Sent: Friday, 22 January 2016 10:38 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again In and out the same interface.. as in vlans? Or two actual subnets on the one physical network? If it's two subnets on the same network, it should be sending an ICMP redirect back for each packet that comes via the router, which would certainly be maxxing a few CPU's? On 22 January 2016 at 10:27, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Hmmm
No
Iperf at 3G udp
All 36 CPU's are up around 96-100%
It's a hairpin route in and out the same interface .... not sure if that's the problem...
Alex
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 3:30 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
I think I have found my problem.
This is a replacement CCR1036-8G-2S+ 6.33.3
I replace the config with a config from a previous ccr.
All the mac's are not the default ones. I am presuming that's causing all the packets to hit the CPU's
My other CCR sitting next to it, same config (diff ip's) same package.
Will advise when I get a chance to reset it
A
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Cox Sent: Thursday, 21 January 2016 3:18 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again
Hey Alex,
It should be fine, but can you provide an example of your test config and the routers software version? Is the test single or multi-threaded, tcp or udp?
- Andrew
On Thu, Jan 21, 2016 at 1:14 PM, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Just checked /system resource cpu
36 cpu doing ~100% (97-98%) load and irq ... feels like somethings wrong there.
A
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 1:57 PM To: public@talk.mikrotik.com.au Subject: CCR performance testing again
Hi
I'm testing switches again and doing a comparison to the CCR.
For some reason I can't seem to get past 1Gb/s without losses on my ccr1036.
Now this could be my VM's as VM to VM I can only get around 2-3Gb/s before iperf starts complaining ..
How are other people's experience with the CCR's I know about the 1Gb/s tcp stream limit, but I hand heard of any issues with UDP.
Infact I was pretty sure I had push it up to 9.8Gb/s.. but that was a while ago
A
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
-- Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Alex, List. Bonding (LAG/LACP) is NOT currently supported by FastPath on Mikrotik routers. I have been in contact with Mikrotik support about this several times now. Mikrotik managed to get FastPath RX working in RouterOS 6.x, but not TX. I have not received an ETA on being able to do FastPath TX and RX. This is almost certainly the reason for your performance issues. Hopefully they add this in v7. We almost always put our CCR's behind MC-LAG switch clusters for redundancy, but the lack of bonding + fastpath is severely limiting performance in this configuration. Regards, Andrew On Fri, Jan 22, 2016 at 5:29 PM, Alex Samad - Yieldbroker <Alex.Samad@yieldbroker.com> wrote:
Try again san's pictures :)
From: Alex Samad - Yieldbroker [mailto:Alex.Samad@yieldbroker.com] Sent: Friday, 22 January 2016 1:50 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
HI
Thought I would put the commands I am using
[root@alextest1 ~]# iperf -c 192.168.213.52 -i 30 -t 60 -u -l 12k -b 2 -> packets leave eth0
Goto ccr
[root@alextest2 ~]# iperf -s -u -l 12k -> come in eth1 on here
------------------------------------------------------------ Client connecting to 192.168.213.52, UDP port 5001 Sending 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 10.172.203.51 port 59192 connected with 192.168.213.52 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-30.0 sec 7.00 GBytes 2.01 Gbits/sec [ 3] 30.0-60.0 sec 7.01 GBytes 2.01 Gbits/sec [ 3] 0.0-60.0 sec 14.0 GBytes 2.01 Gbits/sec [ 3] Sent 1224310 datagrams [ 3] Server Report: [ 3] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 3] 0.0-60.0 sec 327413 datagrams received out-of-order [root@alextest1 ~]#
At 2G cpu usage Picture of low cpu usage!
3g - 99% Picture of high cpu usage ... >98%
Somebody on the list was suggesting maybe a LACP issue . haven't had an opportunity to test that.
------------------------------------------------------------ Server listening on UDP port 5001 Receiving 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 52469 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-60.0 sec 6.86 GBytes 981 Mbits/sec 0.671 ms 625418/1224436 (51%) [ 3] 0.0-60.0 sec 322936 datagrams received out-of-order [ 3] WARNING: ack of last datagram failed after 10 tries. [ 4] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 59192 [ 4] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 4] 0.0-60.0 sec 327413 datagrams received out-of-order [ 4] WARNING: ack of last datagram failed after 10 tries.
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Friday, 22 January 2016 12:28 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
No
Same vlan, different ip network.
Why would it be sending redirects.
Src is 192.168.213.52 dst is 10.172.213.51. Just because they are the same vlan doesn't allow them to talk directly (normally)
Packet path is 192.168.213.52 -> 192.168.213.2 (ccr) -> 10.172.213.2 (ccr - same interface) - > 10.172.213.51
I have setup another test such that it doesn't do this. Instead in one interface and out another interface ... Again > 3G floods the CPU :(.....
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Damien Gardner Jnr Sent: Friday, 22 January 2016 10:38 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again
In and out the same interface.. as in vlans? Or two actual subnets on the one physical network? If it's two subnets on the same network, it should be sending an ICMP redirect back for each packet that comes via the router, which would certainly be maxxing a few CPU's?
On 22 January 2016 at 10:27, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Hmmm
No
Iperf at 3G udp
All 36 CPU's are up around 96-100%
It's a hairpin route in and out the same interface .... not sure if that's the problem...
Alex
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 3:30 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
I think I have found my problem.
This is a replacement CCR1036-8G-2S+ 6.33.3
I replace the config with a config from a previous ccr.
All the mac's are not the default ones. I am presuming that's causing all the packets to hit the CPU's
My other CCR sitting next to it, same config (diff ip's) same package.
Will advise when I get a chance to reset it
A
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Cox Sent: Thursday, 21 January 2016 3:18 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again
Hey Alex,
It should be fine, but can you provide an example of your test config and the routers software version? Is the test single or multi-threaded, tcp or udp?
- Andrew
On Thu, Jan 21, 2016 at 1:14 PM, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Just checked /system resource cpu
36 cpu doing ~100% (97-98%) load and irq ... feels like somethings wrong there.
A
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 1:57 PM To: public@talk.mikrotik.com.au Subject: CCR performance testing again
Hi
I'm testing switches again and doing a comparison to the CCR.
For some reason I can't seem to get past 1Gb/s without losses on my ccr1036.
Now this could be my VM's as VM to VM I can only get around 2-3Gb/s before iperf starts complaining ..
How are other people's experience with the CCR's I know about the 1Gb/s tcp stream limit, but I hand heard of any issues with UDP.
Infact I was pretty sure I had push it up to 9.8Gb/s.. but that was a while ago
A
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
--
Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Thanks that's good to know. The list of things waiting on V7 is growing...... Not being able to LACP is rather limiting ??? I guess I could do something like 10G Switch A -> CCR1 -> CCR2 -> 10Gswitch B Where Switch A & B are stacked .... that would remove the LACP and make a L3 link between CCR's but L2 ?.. sigh A -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Thrift Sent: Monday, 25 January 2016 1:12 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] FW: CCR performance testing again Hi Alex, List. Bonding (LAG/LACP) is NOT currently supported by FastPath on Mikrotik routers. I have been in contact with Mikrotik support about this several times now. Mikrotik managed to get FastPath RX working in RouterOS 6.x, but not TX. I have not received an ETA on being able to do FastPath TX and RX. This is almost certainly the reason for your performance issues. Hopefully they add this in v7. We almost always put our CCR's behind MC-LAG switch clusters for redundancy, but the lack of bonding + fastpath is severely limiting performance in this configuration. Regards, Andrew On Fri, Jan 22, 2016 at 5:29 PM, Alex Samad - Yieldbroker <Alex.Samad@yieldbroker.com> wrote:
Try again san's pictures :)
From: Alex Samad - Yieldbroker [mailto:Alex.Samad@yieldbroker.com] Sent: Friday, 22 January 2016 1:50 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
HI
Thought I would put the commands I am using
[root@alextest1 ~]# iperf -c 192.168.213.52 -i 30 -t 60 -u -l 12k -b 2 -> packets leave eth0
Goto ccr
[root@alextest2 ~]# iperf -s -u -l 12k -> come in eth1 on here
------------------------------------------------------------ Client connecting to 192.168.213.52, UDP port 5001 Sending 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 10.172.203.51 port 59192 connected with 192.168.213.52 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-30.0 sec 7.00 GBytes 2.01 Gbits/sec [ 3] 30.0-60.0 sec 7.01 GBytes 2.01 Gbits/sec [ 3] 0.0-60.0 sec 14.0 GBytes 2.01 Gbits/sec [ 3] Sent 1224310 datagrams [ 3] Server Report: [ 3] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 3] 0.0-60.0 sec 327413 datagrams received out-of-order [root@alextest1 ~]#
At 2G cpu usage Picture of low cpu usage!
3g - 99% Picture of high cpu usage ... >98%
Somebody on the list was suggesting maybe a LACP issue . haven't had an opportunity to test that.
------------------------------------------------------------ Server listening on UDP port 5001 Receiving 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 52469 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-60.0 sec 6.86 GBytes 981 Mbits/sec 0.671 ms 625418/1224436 (51%) [ 3] 0.0-60.0 sec 322936 datagrams received out-of-order [ 3] WARNING: ack of last datagram failed after 10 tries. [ 4] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 59192 [ 4] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 4] 0.0-60.0 sec 327413 datagrams received out-of-order [ 4] WARNING: ack of last datagram failed after 10 tries.
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Friday, 22 January 2016 12:28 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
No
Same vlan, different ip network.
Why would it be sending redirects.
Src is 192.168.213.52 dst is 10.172.213.51. Just because they are the same vlan doesn't allow them to talk directly (normally)
Packet path is 192.168.213.52 -> 192.168.213.2 (ccr) -> 10.172.213.2 (ccr - same interface) - > 10.172.213.51
I have setup another test such that it doesn't do this. Instead in one interface and out another interface ... Again > 3G floods the CPU :(.....
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Damien Gardner Jnr Sent: Friday, 22 January 2016 10:38 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again
In and out the same interface.. as in vlans? Or two actual subnets on the one physical network? If it's two subnets on the same network, it should be sending an ICMP redirect back for each packet that comes via the router, which would certainly be maxxing a few CPU's?
On 22 January 2016 at 10:27, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Hmmm
No
Iperf at 3G udp
All 36 CPU's are up around 96-100%
It's a hairpin route in and out the same interface .... not sure if that's the problem...
Alex
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 3:30 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
I think I have found my problem.
This is a replacement CCR1036-8G-2S+ 6.33.3
I replace the config with a config from a previous ccr.
All the mac's are not the default ones. I am presuming that's causing all the packets to hit the CPU's
My other CCR sitting next to it, same config (diff ip's) same package.
Will advise when I get a chance to reset it
A
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Cox Sent: Thursday, 21 January 2016 3:18 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again
Hey Alex,
It should be fine, but can you provide an example of your test config and the routers software version? Is the test single or multi-threaded, tcp or udp?
- Andrew
On Thu, Jan 21, 2016 at 1:14 PM, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Just checked /system resource cpu
36 cpu doing ~100% (97-98%) load and irq ... feels like somethings wrong there.
A
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 1:57 PM To: public@talk.mikrotik.com.au Subject: CCR performance testing again
Hi
I'm testing switches again and doing a comparison to the CCR.
For some reason I can't seem to get past 1Gb/s without losses on my ccr1036.
Now this could be my VM's as VM to VM I can only get around 2-3Gb/s before iperf starts complaining ..
How are other people's experience with the CCR's I know about the 1Gb/s tcp stream limit, but I hand heard of any issues with UDP.
Infact I was pretty sure I had push it up to 9.8Gb/s.. but that was a while ago
A
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
--
Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Oh FarQ. I just did a big roll out on 1072's with LACP everywhere. The customer required 10gig's and redundancy it's not finished yet luckily but they are going to ask serious questions when i turn this off. Not happy Jan. I sure wish there was a warning box that alerted you when fast path was not being used when you use a feature. To reiterate. FarQ Matt. On 25/01/2016 2:11 PM, Alex Samad - Yieldbroker wrote:
Hi
Thanks that's good to know.
The list of things waiting on V7 is growing......
Not being able to LACP is rather limiting ???
I guess I could do something like
10G Switch A -> CCR1 -> CCR2 -> 10Gswitch B
Where Switch A & B are stacked .... that would remove the LACP and make a L3 link between CCR's but L2 ?.. sigh
A
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Thrift Sent: Monday, 25 January 2016 1:12 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] FW: CCR performance testing again
Hi Alex, List.
Bonding (LAG/LACP) is NOT currently supported by FastPath on Mikrotik routers.
I have been in contact with Mikrotik support about this several times now. Mikrotik managed to get FastPath RX working in RouterOS 6.x, but not TX. I have not received an ETA on being able to do FastPath TX and RX.
This is almost certainly the reason for your performance issues.
Hopefully they add this in v7. We almost always put our CCR's behind MC-LAG switch clusters for redundancy, but the lack of bonding + fastpath is severely limiting performance in this configuration.
Regards,
Andrew
On Fri, Jan 22, 2016 at 5:29 PM, Alex Samad - Yieldbroker <Alex.Samad@yieldbroker.com> wrote:
Try again san's pictures :)
From: Alex Samad - Yieldbroker [mailto:Alex.Samad@yieldbroker.com] Sent: Friday, 22 January 2016 1:50 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
HI
Thought I would put the commands I am using
[root@alextest1 ~]# iperf -c 192.168.213.52 -i 30 -t 60 -u -l 12k -b 2 -> packets leave eth0
Goto ccr
[root@alextest2 ~]# iperf -s -u -l 12k -> come in eth1 on here
------------------------------------------------------------ Client connecting to 192.168.213.52, UDP port 5001 Sending 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 10.172.203.51 port 59192 connected with 192.168.213.52 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-30.0 sec 7.00 GBytes 2.01 Gbits/sec [ 3] 30.0-60.0 sec 7.01 GBytes 2.01 Gbits/sec [ 3] 0.0-60.0 sec 14.0 GBytes 2.01 Gbits/sec [ 3] Sent 1224310 datagrams [ 3] Server Report: [ 3] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 3] 0.0-60.0 sec 327413 datagrams received out-of-order [root@alextest1 ~]#
At 2G cpu usage Picture of low cpu usage!
3g - 99% Picture of high cpu usage ... >98%
Somebody on the list was suggesting maybe a LACP issue . haven't had an opportunity to test that.
------------------------------------------------------------ Server listening on UDP port 5001 Receiving 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 52469 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-60.0 sec 6.86 GBytes 981 Mbits/sec 0.671 ms 625418/1224436 (51%) [ 3] 0.0-60.0 sec 322936 datagrams received out-of-order [ 3] WARNING: ack of last datagram failed after 10 tries. [ 4] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 59192 [ 4] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 4] 0.0-60.0 sec 327413 datagrams received out-of-order [ 4] WARNING: ack of last datagram failed after 10 tries.
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Friday, 22 January 2016 12:28 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
No
Same vlan, different ip network.
Why would it be sending redirects.
Src is 192.168.213.52 dst is 10.172.213.51. Just because they are the same vlan doesn't allow them to talk directly (normally)
Packet path is 192.168.213.52 -> 192.168.213.2 (ccr) -> 10.172.213.2 (ccr - same interface) - > 10.172.213.51
I have setup another test such that it doesn't do this. Instead in one interface and out another interface ... Again > 3G floods the CPU :(.....
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Damien Gardner Jnr Sent: Friday, 22 January 2016 10:38 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again
In and out the same interface.. as in vlans? Or two actual subnets on the one physical network? If it's two subnets on the same network, it should be sending an ICMP redirect back for each packet that comes via the router, which would certainly be maxxing a few CPU's?
On 22 January 2016 at 10:27, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Hmmm
No
Iperf at 3G udp
All 36 CPU's are up around 96-100%
It's a hairpin route in and out the same interface .... not sure if that's the problem...
Alex
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 3:30 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
I think I have found my problem.
This is a replacement CCR1036-8G-2S+ 6.33.3
I replace the config with a config from a previous ccr.
All the mac's are not the default ones. I am presuming that's causing all the packets to hit the CPU's
My other CCR sitting next to it, same config (diff ip's) same package.
Will advise when I get a chance to reset it
A
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Cox Sent: Thursday, 21 January 2016 3:18 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again
Hey Alex,
It should be fine, but can you provide an example of your test config and the routers software version? Is the test single or multi-threaded, tcp or udp?
- Andrew
On Thu, Jan 21, 2016 at 1:14 PM, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Just checked /system resource cpu
36 cpu doing ~100% (97-98%) load and irq ... feels like somethings wrong there.
A
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 1:57 PM To: public@talk.mikrotik.com.au Subject: CCR performance testing again
Hi
I'm testing switches again and doing a comparison to the CCR.
For some reason I can't seem to get past 1Gb/s without losses on my ccr1036.
Now this could be my VM's as VM to VM I can only get around 2-3Gb/s before iperf starts complaining ..
How are other people's experience with the CCR's I know about the 1Gb/s tcp stream limit, but I hand heard of any issues with UDP.
Infact I was pretty sure I had push it up to 9.8Gb/s.. but that was a while ago
A
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
--
Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
-- /* Matt Perkins Direct 1300 137 379 Spectrum Networks Ptd. Ltd. Office 1300 133 299 matt@spectrum.com.au Level 6, 350 George Street Sydney 2000 Spectrum Networks is a member of the Communications Alliance & TIO */
You do know they (well the 1036) can't route more than ~1Gb/s on a single tcp stream . another feature that's coming in V7 (actually mentioned in the Melbourne MUM slides under beta V7 I believe) A -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Matt Perkins Sent: Monday, 25 January 2016 2:24 PM To: public@talk.mikrotik.com.au Subject: Re: [MT-AU Public] FW: CCR performance testing again Oh FarQ. I just did a big roll out on 1072's with LACP everywhere. The customer required 10gig's and redundancy it's not finished yet luckily but they are going to ask serious questions when i turn this off. Not happy Jan. I sure wish there was a warning box that alerted you when fast path was not being used when you use a feature. To reiterate. FarQ Matt. On 25/01/2016 2:11 PM, Alex Samad - Yieldbroker wrote:
Hi
Thanks that's good to know.
The list of things waiting on V7 is growing......
Not being able to LACP is rather limiting ???
I guess I could do something like
10G Switch A -> CCR1 -> CCR2 -> 10Gswitch B
Where Switch A & B are stacked .... that would remove the LACP and make a L3 link between CCR's but L2 ?.. sigh
A
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Thrift Sent: Monday, 25 January 2016 1:12 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] FW: CCR performance testing again
Hi Alex, List.
Bonding (LAG/LACP) is NOT currently supported by FastPath on Mikrotik routers.
I have been in contact with Mikrotik support about this several times now. Mikrotik managed to get FastPath RX working in RouterOS 6.x, but not TX. I have not received an ETA on being able to do FastPath TX and RX.
This is almost certainly the reason for your performance issues.
Hopefully they add this in v7. We almost always put our CCR's behind MC-LAG switch clusters for redundancy, but the lack of bonding + fastpath is severely limiting performance in this configuration.
Regards,
Andrew
On Fri, Jan 22, 2016 at 5:29 PM, Alex Samad - Yieldbroker <Alex.Samad@yieldbroker.com> wrote:
Try again san's pictures :)
From: Alex Samad - Yieldbroker [mailto:Alex.Samad@yieldbroker.com] Sent: Friday, 22 January 2016 1:50 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
HI
Thought I would put the commands I am using
[root@alextest1 ~]# iperf -c 192.168.213.52 -i 30 -t 60 -u -l 12k -b 2 -> packets leave eth0
Goto ccr
[root@alextest2 ~]# iperf -s -u -l 12k -> come in eth1 on here
------------------------------------------------------------ Client connecting to 192.168.213.52, UDP port 5001 Sending 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 10.172.203.51 port 59192 connected with 192.168.213.52 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-30.0 sec 7.00 GBytes 2.01 Gbits/sec [ 3] 30.0-60.0 sec 7.01 GBytes 2.01 Gbits/sec [ 3] 0.0-60.0 sec 14.0 GBytes 2.01 Gbits/sec [ 3] Sent 1224310 datagrams [ 3] Server Report: [ 3] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 3] 0.0-60.0 sec 327413 datagrams received out-of-order [root@alextest1 ~]#
At 2G cpu usage Picture of low cpu usage!
3g - 99% Picture of high cpu usage ... >98%
Somebody on the list was suggesting maybe a LACP issue . haven't had an opportunity to test that.
------------------------------------------------------------ Server listening on UDP port 5001 Receiving 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 52469 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-60.0 sec 6.86 GBytes 981 Mbits/sec 0.671 ms 625418/1224436 (51%) [ 3] 0.0-60.0 sec 322936 datagrams received out-of-order [ 3] WARNING: ack of last datagram failed after 10 tries. [ 4] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 59192 [ 4] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 4] 0.0-60.0 sec 327413 datagrams received out-of-order [ 4] WARNING: ack of last datagram failed after 10 tries.
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Friday, 22 January 2016 12:28 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
No
Same vlan, different ip network.
Why would it be sending redirects.
Src is 192.168.213.52 dst is 10.172.213.51. Just because they are the same vlan doesn't allow them to talk directly (normally)
Packet path is 192.168.213.52 -> 192.168.213.2 (ccr) -> 10.172.213.2 (ccr - same interface) - > 10.172.213.51
I have setup another test such that it doesn't do this. Instead in one interface and out another interface ... Again > 3G floods the CPU :(.....
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Damien Gardner Jnr Sent: Friday, 22 January 2016 10:38 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again
In and out the same interface.. as in vlans? Or two actual subnets on the one physical network? If it's two subnets on the same network, it should be sending an ICMP redirect back for each packet that comes via the router, which would certainly be maxxing a few CPU's?
On 22 January 2016 at 10:27, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Hmmm
No
Iperf at 3G udp
All 36 CPU's are up around 96-100%
It's a hairpin route in and out the same interface .... not sure if that's the problem...
Alex
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 3:30 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
I think I have found my problem.
This is a replacement CCR1036-8G-2S+ 6.33.3
I replace the config with a config from a previous ccr.
All the mac's are not the default ones. I am presuming that's causing all the packets to hit the CPU's
My other CCR sitting next to it, same config (diff ip's) same package.
Will advise when I get a chance to reset it
A
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Cox Sent: Thursday, 21 January 2016 3:18 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again
Hey Alex,
It should be fine, but can you provide an example of your test config and the routers software version? Is the test single or multi-threaded, tcp or udp?
- Andrew
On Thu, Jan 21, 2016 at 1:14 PM, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Just checked /system resource cpu
36 cpu doing ~100% (97-98%) load and irq ... feels like somethings wrong there.
A
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 1:57 PM To: public@talk.mikrotik.com.au Subject: CCR performance testing again
Hi
I'm testing switches again and doing a comparison to the CCR.
For some reason I can't seem to get past 1Gb/s without losses on my ccr1036.
Now this could be my VM's as VM to VM I can only get around 2-3Gb/s before iperf starts complaining ..
How are other people's experience with the CCR's I know about the 1Gb/s tcp stream limit, but I hand heard of any issues with UDP.
Infact I was pretty sure I had push it up to 9.8Gb/s.. but that was a while ago
A
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
--
Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
-- /* Matt Perkins Direct 1300 137 379 Spectrum Networks Ptd. Ltd. Office 1300 133 299 matt@spectrum.com.au Level 6, 350 George Street Sydney 2000 Spectrum Networks is a member of the Communications Alliance & TIO */ _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Yes. that wont cause us any problems. Very few tcp applications run faster then 1G/bit because of latency in any case. Matt On 25/01/2016 2:44 PM, Alex Samad - Yieldbroker wrote:
You do know they (well the 1036) can't route more than ~1Gb/s on a single tcp stream . another feature that's coming in V7 (actually mentioned in the Melbourne MUM slides under beta V7 I believe)
A
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Matt Perkins Sent: Monday, 25 January 2016 2:24 PM To: public@talk.mikrotik.com.au Subject: Re: [MT-AU Public] FW: CCR performance testing again
Oh FarQ. I just did a big roll out on 1072's with LACP everywhere. The customer required 10gig's and redundancy it's not finished yet luckily but they are going to ask serious questions when i turn this off. Not happy Jan.
I sure wish there was a warning box that alerted you when fast path was not being used when you use a feature. To reiterate.
FarQ
Matt.
On 25/01/2016 2:11 PM, Alex Samad - Yieldbroker wrote:
Hi
Thanks that's good to know.
The list of things waiting on V7 is growing......
Not being able to LACP is rather limiting ???
I guess I could do something like
10G Switch A -> CCR1 -> CCR2 -> 10Gswitch B
Where Switch A & B are stacked .... that would remove the LACP and make a L3 link between CCR's but L2 ?.. sigh
A
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Thrift Sent: Monday, 25 January 2016 1:12 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] FW: CCR performance testing again
Hi Alex, List.
Bonding (LAG/LACP) is NOT currently supported by FastPath on Mikrotik routers.
I have been in contact with Mikrotik support about this several times now. Mikrotik managed to get FastPath RX working in RouterOS 6.x, but not TX. I have not received an ETA on being able to do FastPath TX and RX.
This is almost certainly the reason for your performance issues.
Hopefully they add this in v7. We almost always put our CCR's behind MC-LAG switch clusters for redundancy, but the lack of bonding + fastpath is severely limiting performance in this configuration.
Regards,
Andrew
On Fri, Jan 22, 2016 at 5:29 PM, Alex Samad - Yieldbroker <Alex.Samad@yieldbroker.com> wrote:
Try again san's pictures :)
From: Alex Samad - Yieldbroker [mailto:Alex.Samad@yieldbroker.com] Sent: Friday, 22 January 2016 1:50 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
HI
Thought I would put the commands I am using
[root@alextest1 ~]# iperf -c 192.168.213.52 -i 30 -t 60 -u -l 12k -b 2 -> packets leave eth0
Goto ccr
[root@alextest2 ~]# iperf -s -u -l 12k -> come in eth1 on here
------------------------------------------------------------ Client connecting to 192.168.213.52, UDP port 5001 Sending 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 10.172.203.51 port 59192 connected with 192.168.213.52 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-30.0 sec 7.00 GBytes 2.01 Gbits/sec [ 3] 30.0-60.0 sec 7.01 GBytes 2.01 Gbits/sec [ 3] 0.0-60.0 sec 14.0 GBytes 2.01 Gbits/sec [ 3] Sent 1224310 datagrams [ 3] Server Report: [ 3] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 3] 0.0-60.0 sec 327413 datagrams received out-of-order [root@alextest1 ~]#
At 2G cpu usage Picture of low cpu usage!
3g - 99% Picture of high cpu usage ... >98%
Somebody on the list was suggesting maybe a LACP issue . haven't had an opportunity to test that.
------------------------------------------------------------ Server listening on UDP port 5001 Receiving 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 52469 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-60.0 sec 6.86 GBytes 981 Mbits/sec 0.671 ms 625418/1224436 (51%) [ 3] 0.0-60.0 sec 322936 datagrams received out-of-order [ 3] WARNING: ack of last datagram failed after 10 tries. [ 4] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 59192 [ 4] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 4] 0.0-60.0 sec 327413 datagrams received out-of-order [ 4] WARNING: ack of last datagram failed after 10 tries.
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Friday, 22 January 2016 12:28 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
No
Same vlan, different ip network.
Why would it be sending redirects.
Src is 192.168.213.52 dst is 10.172.213.51. Just because they are the same vlan doesn't allow them to talk directly (normally)
Packet path is 192.168.213.52 -> 192.168.213.2 (ccr) -> 10.172.213.2 (ccr - same interface) - > 10.172.213.51
I have setup another test such that it doesn't do this. Instead in one interface and out another interface ... Again > 3G floods the CPU :(.....
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Damien Gardner Jnr Sent: Friday, 22 January 2016 10:38 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again
In and out the same interface.. as in vlans? Or two actual subnets on the one physical network? If it's two subnets on the same network, it should be sending an ICMP redirect back for each packet that comes via the router, which would certainly be maxxing a few CPU's?
On 22 January 2016 at 10:27, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Hmmm
No
Iperf at 3G udp
All 36 CPU's are up around 96-100%
It's a hairpin route in and out the same interface .... not sure if that's the problem...
Alex
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 3:30 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
I think I have found my problem.
This is a replacement CCR1036-8G-2S+ 6.33.3
I replace the config with a config from a previous ccr.
All the mac's are not the default ones. I am presuming that's causing all the packets to hit the CPU's
My other CCR sitting next to it, same config (diff ip's) same package.
Will advise when I get a chance to reset it
A
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Cox Sent: Thursday, 21 January 2016 3:18 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again
Hey Alex,
It should be fine, but can you provide an example of your test config and the routers software version? Is the test single or multi-threaded, tcp or udp?
- Andrew
On Thu, Jan 21, 2016 at 1:14 PM, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Just checked /system resource cpu
36 cpu doing ~100% (97-98%) load and irq ... feels like somethings wrong there.
A
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 1:57 PM To: public@talk.mikrotik.com.au Subject: CCR performance testing again
Hi
I'm testing switches again and doing a comparison to the CCR.
For some reason I can't seem to get past 1Gb/s without losses on my ccr1036.
Now this could be my VM's as VM to VM I can only get around 2-3Gb/s before iperf starts complaining ..
How are other people's experience with the CCR's I know about the 1Gb/s tcp stream limit, but I hand heard of any issues with UDP.
Infact I was pretty sure I had push it up to 9.8Gb/s.. but that was a while ago
A
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
--
Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
-- /* Matt Perkins Direct 1300 137 379 Spectrum Networks Ptd. Ltd. Office 1300 133 299 matt@spectrum.com.au Level 6, 350 George Street Sydney 2000 Spectrum Networks is a member of the Communications Alliance & TIO */
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
-- /* Matt Perkins Direct 1300 137 379 Spectrum Networks Ptd. Ltd. Office 1300 133 299 matt@spectrum.com.au Level 6, 350 George Street Sydney 2000 Spectrum Networks is a member of the Communications Alliance & TIO */
Reminds me of this old one. https://www.mikrotik-routeros.com/2014/04/routeros-v7-0-released/ On 25/01/2016 5:07 PM, Matt Perkins wrote:
Yes. that wont cause us any problems. Very few tcp applications run faster then 1G/bit because of latency in any case.
Matt
On 25/01/2016 2:44 PM, Alex Samad - Yieldbroker wrote:
You do know they (well the 1036) can't route more than ~1Gb/s on a single tcp stream . another feature that's coming in V7 (actually mentioned in the Melbourne MUM slides under beta V7 I believe)
A
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Matt Perkins Sent: Monday, 25 January 2016 2:24 PM To: public@talk.mikrotik.com.au Subject: Re: [MT-AU Public] FW: CCR performance testing again
Oh FarQ. I just did a big roll out on 1072's with LACP everywhere. The customer required 10gig's and redundancy it's not finished yet luckily but they are going to ask serious questions when i turn this off. Not happy Jan.
I sure wish there was a warning box that alerted you when fast path was not being used when you use a feature. To reiterate.
FarQ
Matt.
On 25/01/2016 2:11 PM, Alex Samad - Yieldbroker wrote:
Hi
Thanks that's good to know.
The list of things waiting on V7 is growing......
Not being able to LACP is rather limiting ???
I guess I could do something like
10G Switch A -> CCR1 -> CCR2 -> 10Gswitch B
Where Switch A & B are stacked .... that would remove the LACP and make a L3 link between CCR's but L2 ?.. sigh
A
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Thrift Sent: Monday, 25 January 2016 1:12 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] FW: CCR performance testing again
Hi Alex, List.
Bonding (LAG/LACP) is NOT currently supported by FastPath on Mikrotik routers.
I have been in contact with Mikrotik support about this several times now. Mikrotik managed to get FastPath RX working in RouterOS 6.x, but not TX. I have not received an ETA on being able to do FastPath TX and RX.
This is almost certainly the reason for your performance issues.
Hopefully they add this in v7. We almost always put our CCR's behind MC-LAG switch clusters for redundancy, but the lack of bonding + fastpath is severely limiting performance in this configuration.
Regards,
Andrew
On Fri, Jan 22, 2016 at 5:29 PM, Alex Samad - Yieldbroker <Alex.Samad@yieldbroker.com> wrote:
Try again san's pictures :)
From: Alex Samad - Yieldbroker [mailto:Alex.Samad@yieldbroker.com] Sent: Friday, 22 January 2016 1:50 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
HI
Thought I would put the commands I am using
[root@alextest1 ~]# iperf -c 192.168.213.52 -i 30 -t 60 -u -l 12k -b 2 -> packets leave eth0
Goto ccr
[root@alextest2 ~]# iperf -s -u -l 12k -> come in eth1 on here
------------------------------------------------------------ Client connecting to 192.168.213.52, UDP port 5001 Sending 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 10.172.203.51 port 59192 connected with 192.168.213.52 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-30.0 sec 7.00 GBytes 2.01 Gbits/sec [ 3] 30.0-60.0 sec 7.01 GBytes 2.01 Gbits/sec [ 3] 0.0-60.0 sec 14.0 GBytes 2.01 Gbits/sec [ 3] Sent 1224310 datagrams [ 3] Server Report: [ 3] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 3] 0.0-60.0 sec 327413 datagrams received out-of-order [root@alextest1 ~]#
At 2G cpu usage Picture of low cpu usage!
3g - 99% Picture of high cpu usage ... >98%
Somebody on the list was suggesting maybe a LACP issue . haven't had an opportunity to test that.
------------------------------------------------------------ Server listening on UDP port 5001 Receiving 12288 byte datagrams UDP buffer size: 32.0 MByte (default) ------------------------------------------------------------ [ 3] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 52469 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-60.0 sec 6.86 GBytes 981 Mbits/sec 0.671 ms 625418/1224436 (51%) [ 3] 0.0-60.0 sec 322936 datagrams received out-of-order [ 3] WARNING: ack of last datagram failed after 10 tries. [ 4] local 192.168.213.52 port 5001 connected with 10.172.203.51 port 59192 [ 4] 0.0-60.0 sec 6.93 GBytes 992 Mbits/sec 0.576 ms 618577/1224308 (51%) [ 4] 0.0-60.0 sec 327413 datagrams received out-of-order [ 4] WARNING: ack of last datagram failed after 10 tries.
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Friday, 22 January 2016 12:28 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
No
Same vlan, different ip network.
Why would it be sending redirects.
Src is 192.168.213.52 dst is 10.172.213.51. Just because they are the same vlan doesn't allow them to talk directly (normally)
Packet path is 192.168.213.52 -> 192.168.213.2 (ccr) -> 10.172.213.2 (ccr - same interface) - > 10.172.213.51
I have setup another test such that it doesn't do this. Instead in one interface and out another interface ... Again > 3G floods the CPU :(.....
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Damien Gardner Jnr Sent: Friday, 22 January 2016 10:38 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again
In and out the same interface.. as in vlans? Or two actual subnets on the one physical network? If it's two subnets on the same network, it should be sending an ICMP redirect back for each packet that comes via the router, which would certainly be maxxing a few CPU's?
On 22 January 2016 at 10:27, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Hmmm
No
Iperf at 3G udp
All 36 CPU's are up around 96-100%
It's a hairpin route in and out the same interface .... not sure if that's the problem...
Alex
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 3:30 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: RE: [MT-AU Public] CCR performance testing again
I think I have found my problem.
This is a replacement CCR1036-8G-2S+ 6.33.3
I replace the config with a config from a previous ccr.
All the mac's are not the default ones. I am presuming that's causing all the packets to hit the CPU's
My other CCR sitting next to it, same config (diff ip's) same package.
Will advise when I get a chance to reset it
A
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Andrew Cox Sent: Thursday, 21 January 2016 3:18 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] CCR performance testing again
Hey Alex,
It should be fine, but can you provide an example of your test config and the routers software version? Is the test single or multi-threaded, tcp or udp?
- Andrew
On Thu, Jan 21, 2016 at 1:14 PM, Alex Samad - Yieldbroker < Alex.Samad@yieldbroker.com> wrote:
Just checked /system resource cpu
36 cpu doing ~100% (97-98%) load and irq ... feels like somethings wrong there.
A
-----Original Message----- From: Alex Samad - Yieldbroker Sent: Thursday, 21 January 2016 1:57 PM To: public@talk.mikrotik.com.au Subject: CCR performance testing again
Hi
I'm testing switches again and doing a comparison to the CCR.
For some reason I can't seem to get past 1Gb/s without losses on my ccr1036.
Now this could be my VM's as VM to VM I can only get around 2-3Gb/s before iperf starts complaining ..
How are other people's experience with the CCR's I know about the 1Gb/s tcp stream limit, but I hand heard of any issues with UDP.
Infact I was pretty sure I had push it up to 9.8Gb/s.. but that was a while ago
A
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.
au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.
au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.
au
--
Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au
-- /* Matt Perkins Direct 1300 137 379 Spectrum Networks Ptd. Ltd. Office 1300 133 299 matt@spectrum.com.au Level 6, 350 George Street Sydney 2000 Spectrum Networks is a member of the Communications Alliance & TIO */
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
-- /* Matt Perkins Direct 1300 137 379 Spectrum Networks Ptd. Ltd. Office 1300 133 299 matt@spectrum.com.au Level 6, 350 George Street Sydney 2000 Spectrum Networks is a member of the Communications Alliance & TIO */
participants (3)
-
Alex Samad - Yieldbroker -
Andrew Thrift -
Matt Perkins