The cluster is heavily utilised, all hefty ESX boxes. CPU usage across the cluster is pretty consistent around the 75% mark. Hard to say if usage between tests were different as we run > 300 VMs Our way forward is going to be CHR RoS instances for services higher then 100Mbp/s and just stick with x86 for anything less using e1000 It's worth noting too that when pushing big throughout through the CHR, the CPU on the VM is a lot happier then on x86. x86 seems to spike like crazy and we have to through multiple coers at bigger services and split out VLAN'd interfaces to seperate NICs to spread the load otherwise TX drops are plentiful. Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064
On 18 Mar. 2017, at 8:21 am, Damien Gardner Jnr <rendrag@rendrag.net> wrote:
Hmm, was the ESXi server under less CPU load while you were doing the testing this time?
It occurs to me that if you're doing VM->RouterOS->RouterOS->VM at 1gbit, there is 3gbps passing through the CPU on the hypervisor. Where as if only using one RouterOS VM, there's only 2gbps passing through the CPU on the hypervisor, for the same 1gbps stream between the Ubuntu VM's.
I've noticed on my big XenServer, that if I have 8 1080->720 transcodes going on my Plex VM (so it's using all 12 virtual CPU cores (16 physical cores/32 HT in the box) just for Plex, plus a couple for the media server behind it which is also a VM, that my available bandwidth to other VM's on the box drops from full gigabit to around 700mbps.
On 18 March 2017 at 09:11, Dave Browning <dave@sentrian.com.au> wrote:
So I tried e1000 again just to be sure and that seems to improve things, not sure what's changed though because we've been using the flexible adapters for Mikro's for years now without issue. Did some side by side comparison of CHR w/ VMXNET3 vs x86 w/ e1000 and came up with these results. https://www.dropbox.com/s/u8smkddhtl8886h/Mikro%20x86%20vs%20CHR.docx?dl=0
Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 On 16 Mar. 2017, at 5:40 pm, Dave Browning <mailto:dave@sentrian.com.au> wrote: [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
It's between 2 virtuals if that makes a difference.
So, Virtual Machine - Virtual Mikro - Virtual Mikro - Virtual Machine
Not just, Host - Virtual Mikro - Host
In the second scenario can push big speeds no worries. Sorry want clear on that.
Interested to see what other come up with in that scenario. No NAT'ing, just routing.
Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064
On 16 Mar. 2017, at 5:00 pm, Damien Gardner Jnr <mailto: rendrag@rendrag.net> wrote:
Sorry, meant that as a ? not a !, was driving :)
That's really bizarre! Our RouterOS VM's are latest version, and they're happily routing gigabit! I'm stumped :(
On 16 March 2017 at 17:25, Dave Browning <mailto:dave@sentrian.com.au> wrote:
Yep, changed from ‘flexible’ to ‘e1000’ - No change to throughput.
Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Damien Gardner Jnr [mailto:rendrag@rendrag.net] Sent: March 16, 2017 at 4:19:58 PM GMT+10 To: MikroTik Australia Public List <mailto:public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86
Did you try changing the RouterOS VM to e1000!
On Thu, 16 Mar 2017 at 5:10 pm, Dave Browning <mailto:dave@sentrian.com.au
wrote:
Yep, currently on demo and things are looking good. Maybe they are passively forcing CHR’s - x86’s have been ‘free’ for a while now ;)
Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Alex Samad [mailto:alex@samad.com.au] Sent: March 16, 2017 at 4:05:57 PM GMT+10 To: MikroTik Australia Public List <mailto:public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86
From memory
you can get a demo licence and I think licence also determines who much throughput...
A
On 16 March 2017 at 16:58, Dave Browning <mailto:dave@sentrian.com.au> wrote:
Replaced 2 test routers with CHR’s, same config as x86 (actually just restored .rsc) - Boom full throughput we’d expect. 8Gbp/s when on same host, 1Gbp/s host to host and our full 800Mbp/s on intercap. Weird.
Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Dave Browning [mailto:dave@sentrian.com.au] Sent: March 16, 2017 at 9:31:25 AM GMT+10 To: MikroTik Australia Public List <mailto:public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86
[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
Not CHR at this stage, old school x86.
Dave Browning | Network Engineer P 07 3369 7666 Level 1, 12 Railway Tce, Milton QLD 4064 From: Mike Everest [mailto:mike@duxtel.com] Sent: March 16, 2017 at 9:28:14 AM GMT+10 To: MikroTik Australia Public List <mailto:public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Max Routed Throughout x86
Hi Dave!
Have you checked that the CHR is properly licensed? Throughput is (very) limited when license is invalid or expired :-}
Cheers!
Mike.
_______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
--
Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust mailto:rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
--
Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust mailto:rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning. _______________________________________________ Public mailing list mailto:Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
--
Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rendrag@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
This email and any files transmitted with it contain confidential information for the exclusive use of the intended recipient. If you received this email in error please notify Sentrian Pty Ltd immediately by return email to the sender and delete the original email. We do not guarantee that this email or any attached files are free of viruses. All recipients should undertake their own virus scanning.