I have gotten close to 1G, the limit is the Nic.. ROS produced CHR which has a vmxnet nic, which people have said they gotten close to 10G. Alex On 15 March 2017 at 20:48, Damien Gardner Jnr <rendrag@rendrag.net> wrote:

We have a RouterOS VM on ESXI at work running as our L2TP VPN server. I can happily pull 800mbps through it on my VPN session (routerOS VM in XenServer on my end as the VPN client)

On 15 March 2017 at 19:42, Dave Browning <dave@sentrian.com.au> wrote:

...

Hi,

What sort of speeds are people able to push (routed) through x86 routers before seeing TX/RX Errors/Drops? Underlying being ESX 6.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

--

Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

Hi 100Mbps sounds very low. e1000 nic, I don't have DVswitch and all my hosts had 10G nics to 10G switching A On 16 March 2017 at 10:16, Dave Browning <dave@sentrian.com.au> wrote:

Having drama’s pushing 100Mbp/s, not sure what’s to blame be it the Mikro, DVSwitch or physical switching. Next thing to test is CHR.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Alex Samad [mailto:alex@samad.com.au] Sent: March 16, 2017 at 8:40:38 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

I have gotten close to 1G, the limit is the Nic.. ROS produced CHR which has a vmxnet nic, which people have said they gotten close to 10G.

Alex

Dollars to donuts, that'll be it.. Switch to E1000 or E1000E and see if you get gigabit :) Also make sure you're running the latest patches, as ESXI 6.5 out of the box has a broken NIC driver, and L2TP packets don't make it through to the Tik's On 16 March 2017 at 10:22, Dave Browning <dave@sentrian.com.au> wrote:

The NIC is ‘dynamic’ not E1000 or VMXNET3 - Could be part of the problem? Got like 80-90 of the x86 RoS’s running too..

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Alex Samad [mailto:alex@samad.com.au] Sent: March 16, 2017 at 9:20:25 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Hi

100Mbps sounds very low. e1000 nic, I don't have DVswitch and all my hosts had 10G nics to 10G switching

A

On 16 March 2017 at 10:16, Dave Browning <dave@sentrian.com.au> wrote:

...

Having drama’s pushing 100Mbp/s, not sure what’s to blame be it the Mikro, DVSwitch or physical switching. Next thing to test is CHR.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Alex Samad [mailto:alex@samad.com.au] Sent: March 16, 2017 at 8:40:38 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

I have gotten close to 1G, the limit is the Nic.. ROS produced CHR which has a vmxnet nic, which people have said they gotten close to 10G.

Alex

_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

-- Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder

Hi Dave! Have you checked that the CHR is properly licensed? Throughput is (very) limited when license is invalid or expired :-} Cheers! Mike.

-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Dave Browning Sent: Thursday, 16 March 2017 10:17 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Having drama’s pushing 100Mbp/s, not sure what’s to blame be it the Mikro, DVSwitch or physical switching. Next thing to test is CHR.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Alex Samad [mailto:alex@samad.com.au] Sent: March 16, 2017 at 8:40:38 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

I have gotten close to 1G, the limit is the Nic.. ROS produced CHR which has a vmxnet nic, which people have said they gotten close to 10G.

Alex

On 15 March 2017 at 20:48, Damien Gardner Jnr <rendrag@rendrag.net> wrote:

...

We have a RouterOS VM on ESXI at work running as our L2TP VPN server. I can happily pull 800mbps through it on my VPN session (routerOS VM in XenServer on my end as the VPN client)

On 15 March 2017 at 19:42, Dave Browning <dave@sentrian.com.au> wrote:

...

Hi,

What sort of speeds are people able to push (routed) through x86 routers before seeing TX/RX Errors/Drops? Underlying being ESX 6.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com .au

--

Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au

_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

Replaced 2 test routers with CHR’s, same config as x86 (actually just restored .rsc) - Boom full throughput we’d expect. 8Gbp/s when on same host, 1Gbp/s host to host and our full 800Mbp/s on intercap. Weird. Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Dave Browning [mailto:dave@sentrian.com.au] Sent: March 16, 2017 at 9:31:25 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86 [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing] Not CHR at this stage, old school x86. Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Mike Everest [mailto:mike@duxtel.com] Sent: March 16, 2017 at 9:28:14 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86 Hi Dave! Have you checked that the CHR is properly licensed? Throughput is (very) limited when license is invalid or expired :-} Cheers! Mike.

-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Dave Browning Sent: Thursday, 16 March 2017 10:17 AM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Having drama’s pushing 100Mbp/s, not sure what’s to blame be it the Mikro, DVSwitch or physical switching. Next thing to test is CHR.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Alex Samad [mailto:alex@samad.com.au] Sent: March 16, 2017 at 8:40:38 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

I have gotten close to 1G, the limit is the Nic.. ROS produced CHR which has a vmxnet nic, which people have said they gotten close to 10G.

Alex

On 15 March 2017 at 20:48, Damien Gardner Jnr <rendrag@rendrag.net> wrote:

...

We have a RouterOS VM on ESXI at work running as our L2TP VPN server. I can happily pull 800mbps through it on my VPN session (routerOS VM in XenServer on my end as the VPN client)

On 15 March 2017 at 19:42, Dave Browning <dave@sentrian.com.au> wrote:

...

Hi,

What sort of speeds are people able to push (routed) through x86 routers before seeing TX/RX Errors/Drops? Underlying being ESX 6.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com .au

--

Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au

_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

Yep, currently on demo and things are looking good. Maybe they are passively forcing CHR’s - x86’s have been ‘free’ for a while now ;) Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Alex Samad [mailto:alex@samad.com.au] Sent: March 16, 2017 at 4:05:57 PM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86 From memory you can get a demo licence and I think licence also determines who much throughput... A On 16 March 2017 at 16:58, Dave Browning <dave@sentrian.com.au> wrote:

Replaced 2 test routers with CHR’s, same config as x86 (actually just restored .rsc) - Boom full throughput we’d expect. 8Gbp/s when on same host, 1Gbp/s host to host and our full 800Mbp/s on intercap. Weird.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Dave Browning [mailto:dave@sentrian.com.au] Sent: March 16, 2017 at 9:31:25 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]

Not CHR at this stage, old school x86.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Mike Everest [mailto:mike@duxtel.com] Sent: March 16, 2017 at 9:28:14 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Hi Dave!

Have you checked that the CHR is properly licensed? Throughput is (very) limited when license is invalid or expired :-}

Cheers!

Mike.

_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning.

Yep, changed from ‘flexible’ to ‘e1000’ - No change to throughput. Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Damien Gardner Jnr [mailto:rendrag@rendrag.net] Sent: March 16, 2017 at 4:19:58 PM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86 Did you try changing the RouterOS VM to e1000! On Thu, 16 Mar 2017 at 5:10 pm, Dave Browning <dave@sentrian.com.au> wrote:

Yep, currently on demo and things are looking good. Maybe they are passively forcing CHR’s - x86’s have been ‘free’ for a while now ;)

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Alex Samad [mailto:alex@samad.com.au] Sent: March 16, 2017 at 4:05:57 PM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

From memory

you can get a demo licence and I think licence also determines who much throughput...

A

On 16 March 2017 at 16:58, Dave Browning <dave@sentrian.com.au> wrote:

...

Replaced 2 test routers with CHR’s, same config as x86 (actually just restored .rsc) - Boom full throughput we’d expect. 8Gbp/s when on same host, 1Gbp/s host to host and our full 800Mbp/s on intercap. Weird.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Dave Browning [mailto:dave@sentrian.com.au] Sent: March 16, 2017 at 9:31:25 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]

Not CHR at this stage, old school x86.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Mike Everest [mailto:mike@duxtel.com] Sent: March 16, 2017 at 9:28:14 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Hi Dave!

Have you checked that the CHR is properly licensed? Throughput is (very) limited when license is invalid or expired :-}

Cheers!

Mike.

_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

-- Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning.

Driving and emailing? At least you weren't driving and TXTing! :-D

-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Damien Gardner Jnr Sent: Thursday, 16 March 2017 5:59 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Sorry, meant that as a ? not a !, was driving :)

That's really bizarre! Our RouterOS VM's are latest version, and they're happily routing gigabit! I'm stumped :(

On 16 March 2017 at 17:25, Dave Browning <dave@sentrian.com.au> wrote:

...

Yep, changed from ‘flexible’ to ‘e1000’ - No change to throughput.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Damien Gardner Jnr [mailto:rendrag@rendrag.net] Sent: March 16, 2017 at 4:19:58 PM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Did you try changing the RouterOS VM to e1000!

On Thu, 16 Mar 2017 at 5:10 pm, Dave Browning <dave@sentrian.com.au> wrote:

...

Yep, currently on demo and things are looking good. Maybe they are passively forcing CHR’s - x86’s have been ‘free’ for a while now ;)

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Alex Samad [mailto:alex@samad.com.au] Sent: March 16, 2017 at 4:05:57 PM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

From memory

you can get a demo licence and I think licence also determines who much throughput...

A

On 16 March 2017 at 16:58, Dave Browning <dave@sentrian.com.au> wrote:

...

Replaced 2 test routers with CHR’s, same config as x86 (actually just restored .rsc) - Boom full throughput we’d expect. 8Gbp/s when on same host, 1Gbp/s host to host and our full 800Mbp/s on intercap. Weird.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Dave Browning [mailto:dave@sentrian.com.au] Sent: March 16, 2017 at 9:31:25 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]

Not CHR at this stage, old school x86.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Mike Everest [mailto:mike@duxtel.com] Sent: March 16, 2017 at 9:28:14 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Hi Dave!

Have you checked that the CHR is properly licensed? Throughput is (very) limited when license is invalid or expired :-}

Cheers!

Mike.

_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.co m.au

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.co m.au

--

Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au

--

Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

guessed as much ;) it's like that in lots of places this time of day! Cheers!

-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Damien Gardner Jnr Sent: Thursday, 16 March 2017 6:07 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Haha! If you've seen the traffic in Tuggerah business park in a Thursday arvo, you'll know technically I was standing and emailing lol :-)

On Thu, 16 Mar 2017 at 6:03 pm, Mike Everest <mike@duxtel.com> wrote:

...

Driving and emailing? At least you weren't driving and TXTing! :-D

...

-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Damien Gardner Jnr Sent: Thursday, 16 March 2017 5:59 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Sorry, meant that as a ? not a !, was driving :)

That's really bizarre! Our RouterOS VM's are latest version, and they're happily routing gigabit! I'm stumped :(

On 16 March 2017 at 17:25, Dave Browning <dave@sentrian.com.au> wrote:

...

Yep, changed from ‘flexible’ to ‘e1000’ - No change to throughput.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Damien Gardner Jnr [mailto:rendrag@rendrag.net] Sent: March 16, 2017 at 4:19:58 PM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Did you try changing the RouterOS VM to e1000!

On Thu, 16 Mar 2017 at 5:10 pm, Dave Browning <dave@sentrian.com.au> wrote:

...

Yep, currently on demo and things are looking good. Maybe they are passively forcing CHR’s - x86’s have been ‘free’ for a while now ;)

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Alex Samad [mailto:alex@samad.com.au] Sent: March 16, 2017 at 4:05:57 PM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

From memory

you can get a demo licence and I think licence also determines who much throughput...

A

On 16 March 2017 at 16:58, Dave Browning <dave@sentrian.com.au> wrote:

...

Replaced 2 test routers with CHR’s, same config as x86 (actually just restored .rsc) - Boom full throughput we’d expect. 8Gbp/s when on same host, 1Gbp/s host to host and our full 800Mbp/s on intercap. Weird.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Dave Browning [mailto:dave@sentrian.com.au] Sent: March 16, 2017 at 9:31:25 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]

Not CHR at this stage, old school x86.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Mike Everest [mailto:mike@duxtel.com] Sent: March 16, 2017 at 9:28:14 AM GMT+10 To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Hi Dave!

Have you checked that the CHR is properly licensed? Throughput is (very) limited when license is invalid or expired :-}

Cheers!

Mike.

_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikroti k.co m.au

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikroti k.co m.au

--

Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au

--

Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.co m.au

_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com. au

--

Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

So I tried e1000 again just to be sure and that seems to improve things, not sure what's changed though because we've been using the flexible adapters for Mikro's for years now without issue. Did some side by side comparison of CHR w/ VMXNET3 vs x86 w/ e1000 and came up with these results. https://www.dropbox.com/s/u8smkddhtl8886h/Mikro%20x86%20vs%20CHR.docx?dl=0 Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 On 16 Mar. 2017, at 5:40 pm, Dave Browning <mailto:dave@sentrian.com.au> wrote: [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing] It's between 2 virtuals if that makes a difference. So, Virtual Machine - Virtual Mikro - Virtual Mikro - Virtual Machine Not just, Host - Virtual Mikro - Host In the second scenario can push big speeds no worries. Sorry want clear on that. Interested to see what other come up with in that scenario. No NAT'ing, just routing. Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 On 16 Mar. 2017, at 5:00 pm, Damien Gardner Jnr <mailto:rendrag@rendrag.net> wrote: Sorry, meant that as a ? not a !, was driving :) That's really bizarre! Our RouterOS VM's are latest version, and they're happily routing gigabit! I'm stumped :( On 16 March 2017 at 17:25, Dave Browning <mailto:dave@sentrian.com.au> wrote: Yep, changed from ‘flexible’ to ‘e1000’ - No change to throughput. Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Damien Gardner Jnr [mailto:rendrag@rendrag.net] Sent: March 16, 2017 at 4:19:58 PM GMT+10 To: MikroTik Australia Public List <mailto:public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86 Did you try changing the RouterOS VM to e1000! On Thu, 16 Mar 2017 at 5:10 pm, Dave Browning <mailto:dave@sentrian.com.au> wrote: Yep, currently on demo and things are looking good. Maybe they are passively forcing CHR’s - x86’s have been ‘free’ for a while now ;) Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Alex Samad [mailto:alex@samad.com.au] Sent: March 16, 2017 at 4:05:57 PM GMT+10 To: MikroTik Australia Public List <mailto:public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86 From memory you can get a demo licence and I think licence also determines who much throughput... A On 16 March 2017 at 16:58, Dave Browning <mailto:dave@sentrian.com.au> wrote: Replaced 2 test routers with CHR’s, same config as x86 (actually just restored .rsc) - Boom full throughput we’d expect. 8Gbp/s when on same host, 1Gbp/s host to host and our full 800Mbp/s on intercap. Weird. Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Dave Browning [mailto:dave@sentrian.com.au] Sent: March 16, 2017 at 9:31:25 AM GMT+10 To: MikroTik Australia Public List <mailto:public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86 [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing] Not CHR at this stage, old school x86. Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Mike Everest [mailto:mike@duxtel.com] Sent: March 16, 2017 at 9:28:14 AM GMT+10 To: MikroTik Australia Public List <mailto:public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86 Hi Dave! Have you checked that the CHR is properly licensed? Throughput is (very) limited when license is invalid or expired :-} Cheers! Mike. _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au -- Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust mailto:rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au -- Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust mailto:rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

The cluster is heavily utilised, all hefty ESX boxes. CPU usage across the cluster is pretty consistent around the 75% mark. Hard to say if usage between tests were different as we run > 300 VMs Our way forward is going to be CHR RoS instances for services higher then 100Mbp/s and just stick with x86 for anything less using e1000 It's worth noting too that when pushing big throughout through the CHR, the CPU on the VM is a lot happier then on x86. x86 seems to spike like crazy and we have to through multiple coers at bigger services and split out VLAN'd interfaces to seperate NICs to spread the load otherwise TX drops are plentiful. Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064

On 18 Mar. 2017, at 8:21 am, Damien Gardner Jnr <rendrag@rendrag.net> wrote:

Hmm, was the ESXi server under less CPU load while you were doing the testing this time?

It occurs to me that if you're doing VM->RouterOS->RouterOS->VM at 1gbit, there is 3gbps passing through the CPU on the hypervisor. Where as if only using one RouterOS VM, there's only 2gbps passing through the CPU on the hypervisor, for the same 1gbps stream between the Ubuntu VM's.

I've noticed on my big XenServer, that if I have 8 1080->720 transcodes going on my Plex VM (so it's using all 12 virtual CPU cores (16 physical cores/32 HT in the box) just for Plex, plus a couple for the media server behind it which is also a VM, that my available bandwidth to other VM's on the box drops from full gigabit to around 700mbps.

...

On 18 March 2017 at 09:11, Dave Browning <dave@sentrian.com.au> wrote:

So I tried e1000 again just to be sure and that seems to improve things, not sure what's changed though because we've been using the flexible adapters for Mikro's for years now without issue. Did some side by side comparison of CHR w/ VMXNET3 vs x86 w/ e1000 and came up with these results. https://www.dropbox.com/s/u8smkddhtl8886h/Mikro%20x86%20vs%20CHR.docx?dl=0

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 On 16 Mar. 2017, at 5:40 pm, Dave Browning <mailto:dave@sentrian.com.au> wrote: [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]

It's between 2 virtuals if that makes a difference.

So, Virtual Machine - Virtual Mikro - Virtual Mikro - Virtual Machine

Not just, Host - Virtual Mikro - Host

In the second scenario can push big speeds no worries. Sorry want clear on that.

Interested to see what other come up with in that scenario. No NAT'ing, just routing.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064

On 16 Mar. 2017, at 5:00 pm, Damien Gardner Jnr <mailto: rendrag@rendrag.net> wrote:

Sorry, meant that as a ? not a !, was driving :)

That's really bizarre! Our RouterOS VM's are latest version, and they're happily routing gigabit! I'm stumped :(

On 16 March 2017 at 17:25, Dave Browning <mailto:dave@sentrian.com.au> wrote:

Yep, changed from ‘flexible’ to ‘e1000’ - No change to throughput.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Damien Gardner Jnr [mailto:rendrag@rendrag.net] Sent: March 16, 2017 at 4:19:58 PM GMT+10 To: MikroTik Australia Public List <mailto:public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Did you try changing the RouterOS VM to e1000!

On Thu, 16 Mar 2017 at 5:10 pm, Dave Browning <mailto:dave@sentrian.com.au

...

wrote:

Yep, currently on demo and things are looking good. Maybe they are passively forcing CHR’s - x86’s have been ‘free’ for a while now ;)

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Alex Samad [mailto:alex@samad.com.au] Sent: March 16, 2017 at 4:05:57 PM GMT+10 To: MikroTik Australia Public List <mailto:public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

From memory

you can get a demo licence and I think licence also determines who much throughput...

A

On 16 March 2017 at 16:58, Dave Browning <mailto:dave@sentrian.com.au> wrote:

Replaced 2 test routers with CHR’s, same config as x86 (actually just restored .rsc) - Boom full throughput we’d expect. 8Gbp/s when on same host, 1Gbp/s host to host and our full 800Mbp/s on intercap. Weird.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Dave Browning [mailto:dave@sentrian.com.au] Sent: March 16, 2017 at 9:31:25 AM GMT+10 To: MikroTik Australia Public List <mailto:public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]

Not CHR at this stage, old school x86.

Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Mike Everest [mailto:mike@duxtel.com] Sent: March 16, 2017 at 9:28:14 AM GMT+10 To: MikroTik Australia Public List <mailto:public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86

Hi Dave!

Have you checked that the CHR is properly licensed? Throughput is (very) limited when license is invalid or expired :-}

Cheers!

Mike.

_______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

--

Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust mailto:rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

--

Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust mailto:rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

--

Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au

This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning.