Fortech I.T. Solutions - SANDBOX

iSCSI backed storage - benchmark results:

Test Name	This Computer

CPU - Integer Math	59.9
CPU - Floating Point Math	335.9
CPU - Find Prime Numbers	145.0
CPU - SSE/3DNow!	1303.8
CPU - Compression	1714.7
CPU - Encryption	12.9
CPU - Image Rotation	195.2
CPU - String Sorting	1003.5
Graphics 2D - Lines	27.7
Graphics 2D - Rectangles	32.8
Graphics 2D - Shapes	0.3
Graphics 2D - Fonts and Text	44.3
Graphics 2D - GUI	244.4
Memory - Allocate Small Block	1730.4
Memory - Read Cached	952.8
Memory - Read Uncached	664.5
Memory - Write	915.6
Memory - Large RAM	48.4
Disk - Sequential Read	30.0
Disk - Sequential Write	25.5
Disk - Random Seek + RW	1.5
CPU Mark	387.2
2D Graphics Mark	171.6
Memory Mark	318.8
Disk Mark	206.5

PassMark Rating	227.2

NFS backed storage - benchmark results:

Test Name	This Computer

CPU - Integer Math	58.9
CPU - Floating Point Math	329.1
CPU - Find Prime Numbers	137.4
CPU - SSE/3DNow!	1296.7
CPU - Compression	1667.6
CPU - Encryption	12.2
CPU - Image Rotation	237.8
CPU - String Sorting	1126.5
Graphics 2D - Lines	27.8
Graphics 2D - Rectangles	40.9
Graphics 2D - Shapes	0.2
Graphics 2D - Fonts and Text	43.4
Graphics 2D - GUI	245.5
Memory - Allocate Small Block	1681.1
Memory - Read Cached	901.4
Memory - Read Uncached	651.0
Memory - Write	907.8
Memory - Large RAM	61.6
Disk - Sequential Read	30.2
Disk - Sequential Write	33.8
Disk - Random Seek + RW	24.0
CPU Mark	394.4
2D Graphics Mark	180.1
Memory Mark	310.7
Disk Mark	318.0
Pa
ssMark Rating	251.2

• Major Vendor (Free "embedded" version, Aug-08 release (*) (Results added Nov-08) )

This will be integrated to table above shortly. But for now:

Some new bench results from Nov-1-08, running

• "Major vendor" "Embedded Server eXtension Infrastructure" platform; on same 2ghz athlon /2gig ram system. Hypervisor is run from 2gig USB memory stick. No local storage. NFS/iSCSI via Intel Gig-Ether nic. Public/Management NIC interface is also via another intel GigEther Nic.
• 2 different bench results below, the first is using iSCSI backed storage on openfiler 2.3 server; the second is NFS backed storage on openfiler 2.3 server. Storage network is dedicated gig-ether.

@]

ROUGH RESULTS ADDENDUM:

Bench results from Nov-1-08, running

• esxi on same 2ghz athlon /2gig ram system

• Xen Express Version 5.0 (results added Sept-08)

Note that XEN EXPRESS results were added Sept-19-08, since that new available release. It was tested against a SATA disk/local storage; same hardware otherwise as everything else.

1784.3 ||

248.3 || || 488.9 || 693.4

(*) Unclear if (1) really should be 'that much slower' than (2), and possibly even (3) as well;

 I'm ultimately not qualified to really understand the precise inner works and nitty grit detail 

involved.

1 - Major Vendor Free Edition: full hardware abstraction / emulation; no requirement for VT assist hardware; anticipated to have more abstraction / performance penalty. (*)

(*) Unclear if (1) really should be 'that much slower' than (2), and possibly even (3) as well; I'm ultimately not qualified to really understand the precise inner works and nitty grit detail involved.

In general - for this page and this work - usual benchmark disclaimers apply of course :-)

• Note that the products here break out into 4 'bins' of virtualization, as far as I'm concerned, and as a result, we should anticipate 'inherently' some performance differences between these approaches to virtualization: (At least, according to 'dogma' ?)

VERY Skeptical that SATA disk really made so much performance difference to M.Vendr-Server, but maybe possible? (more likely cache or some other feature(s) obscure 'real' performance?)

Or something else ?

VERY surprised with performance of VZHOST/Virtuozzo - had been under impression it should be among the top of the performance curve, based on 'dogma and documentation'. Unclear if this result is utterly bogus, or if this is just an unexpected surprise. Already have re-tested this twice now on this hardware with similar results. Maybe need to test on different hardware?

VERY surprised with performance of VZHOST/Virtuozzo - had been under impression it should be among the top of the performance curve, based on 'dogma and documentation'. Unclear if this result is utterly bogus, or if this is just an unexpected surprise. Already have re-tested this twice now on this hardware with similar results. Maybe need to test on different hardware?

This was my own testing work, for my own use. It is not intended as a perfect test, nor ultimately for anyone else for that matter. Material here is released purely for the possible interest of others interested in virtualization. As-is / Where-Is / no implied nor express anything. Your results likely will vary, I'm sure mine will (at least somewhat) if I did this again tomorrow using the same gear. So take this all with a grain of salt and don't worry to much about the specific details. Please do not reproduce this work in any significant part anywhere else in any way. This web site is public so link to this page if you feel you need to 'distribute this data as it stands / in its entirety'.

DISCLAIMER:

This was my own testing work, for my own use. It is not intended as a perfect test, nor ultimately for anyone else for that matter. Material here is released purely for the possible interest of others interested in virtualization. As-is / Where-Is / no implied nor express anything. Your results likely will vary, I'm sure mine will (at least somewhat) if I did this again tomorrow using the same gear. So take this all with a grain of salt and don't worry to much about the specific details.

• add notes here / update & tidy disclaimers.

VERY Skeptical that SATA disk really made so much performance difference to M.Vendr-Server, but maybe possible? (more likely cache or some other feature(s) obscure 'real' performance?)

something is fishy with KVM disk? And also with Xen disk ? Humm. More difference here than expected. VERY Skeptical that SATA disk really made so much performance difference to M.Vendr-Server, but maybe possible?

• Major Vendor (Free edition) version XXXX (*)

• Major Vendor (Free edition) version XXXX build XXXXX (*)

(*) Note that no less than one major vendor does not like benchmarks being published on their product(s),  so their name(s) cannot be given here

1 - Major Vendor Free Edition: full hardware abstraction / emulation; no requirement for VT assist hardware; anticipated to have more abstraction / performance penalty.

• add M.Vendr commercial non-free (30 day trial) bench results

Skeptical that SATA disk really made so much performance difference to M.Vendr-Server, but maybe possible?

Maj.Vendr ver XXXX cannot be explicitly given here (recent download)

TODO:

• add VMWare ESX 3.5.0 (30 day trial) bench results
• tidy up order, sort them by PassMark rating from left to right (declining scores)
• add notes here / disclaimers.

CONFIGURATION NOTES:

Bare Metal = Athlon 64bit socket AM2 2ghz / 1gig Ram
KVM ProxMox = As above, running ProxMox ver 0.9 / 512mb Virtual Machine
Xen Express = as above, running Xen Express v4.10 / 512mb Virtual Machine
P4 ref - bare metal older ref box, Classic socket 478 P4 @ 2.4ghz / 2gig ram / intel onboard video

Proxmox has ParaVirt NIC but not disk
Xen has Paravirt drivers for Nic and Disk
no special drivers on bare metal install (windows defaults) - appeared to autodetect fine for disk and NIC in particular.

all cases running Win2003 Std
except P4 box was Standard Winxp 32bit

something is fishy with KVM disk? And also with Xen disk ? Humm.
Skeptical that SATA disk really made so much performance difference to VMWare-Server, but maybe possible?

VI_SSE is Virtual Iron ver 4.2 / Single Server edition, Win2003, fully patched.

vmware is ver 1.0.5 build 80187

| BenchMark - Measure | Bare Metal | KVM Prox | VI_EE | VMWARE | VI_SSE | VZHOST | Xen Xprss | | P4 REF | | VMWare - sata disk | | CPU - Integer Math | 62.5 | 59.2 | 59.1 | 29.1 | 60.4 | 58.8 | 61.6 | | 43.4 | | 59.3 | | CPU - Floating Point Math | 359.7 | 298 | 339.4 | 167.7 | 315.8 | 344.9 | 335.9 | | 235.2 | | 335.1 | | CPU - Find Prime Numbers | 147.4 | 136 | 142.2 | 74.1 | 137.2 | 135.1 | 144.5 | | 156.2 | | 135.9 | | CPU - SSE/3DNow! | 1307.5 | 1142.6 | 1254.6 | 630.1 | 1154.1 | 1211.9 | 1269.3 | | 1079.7 | | 1187.6 | | CPU - Compression | 1779 | 1712.5 | 1659.8 | 812.5 | 1692.3 | 1652.2 | 1691.2 | | 1659 | | 1730 | | CPU - Encryption | 12.9 | 12.3 | 12.7 | 6.2 | 12.5 | 12.4 | 12.6 | | 11 | | 12.5 | | CPU - Image Rotation | 286.5 | 270.6 | 282.4 | 132.5 | 263.2 | 272.9 | 284.3 | | 171.6 | | 272.8 | | CPU - String Sorting | 1156.8 | 1071.6 | 1153.9 | 538.7 | 1059.3 | 1089.8 | 1105.4 | | 572.4 | | 1092.8 |

| Graphics 2D - Lines | 649 | 15.6 | 26 | 156.8 | 26.5 | 42.8 | 27.3 | | 22.9 | | 28.4 | | Graphics 2D - Rectangles | 513.4 | 17.2 | 36.8 | 142.8 | 38.7 | 41.7 | 37.8 | | 27.6 | | 43.5 | | Graphics 2D - Shapes | 358.8 | 0.3 | 0.3 | 94.2 | 0.3 | 1.2 | 0.2 | | 10.8 | | 1 | | Graphics 2D - Fonts and Text | 1319.5 | 18.2 | 47.6 | 413.9 | 54 | 60.8 | 44.9 | | 165 | | 53.9 | | Graphics 2D - GUI | 1295.4 | 81.9 | 227 | 366.9 | 196.4 | 243.2 | 278.9 | | 132.1 | | 246.8 |

| Memory - Allocate Small Block | 1845.2 | 1739.1 | 1762.6 | 804.2 | 1637.6 | 1780.5 | 1762 | | 841.2 | | 1689.8 | | Memory - Read Cached | 973.4 | 801.9 | 957.6 | 463.5 | 858.3 | 886.4 | 955.8 | | 1477.8 | | 924.7 | | Memory - Read Uncached | 884.1 | 740.3 | 852.1 | 375.7 | 840.1 | 837.6 | 855.7 | | 1268.6 | | 795.9 | | Memory - Write | 861.1 | 719.8 | 938.1 | 420.5 | 885.8 | 845.6 | 811.4 | | 533.2 | | 857 | | Memory - Large RAM | 198.8 | 7.7 | 7.1 | 19.6 | 32.5 | 50.7 | 54.9 | | 206.5 | | 39 |

| Disk - Sequential Read | 44 | 44.9 | 24.5 | 46.8 | 49.8 | 43.8 | 2.4 | | 52.1 | | 152.7 | | Disk - Sequential Write | 44.6 | 40.2 | 26.3 | 25.7 | 50.1 | 43 | 2.5 | | 48.3 | | 25.4 | | Disk - Random Seek + RW | 2.7 | 12.8 | 8.5 | 8.8 | 2.9 | 2.7 | 1.4 | | 3.4 | | 183.6 |

| CD - Read | --- | 9.3 | 4.3 | --- | 1.6 | --- | --- | | --- | | 3.6 | | CPU Mark | 414.5 | 373.8 | 400.6 | 197.1 | 375.7 | 388.1 | 401.7 | | 317.8 | | 385.5 | | 2D Graphics Mark | 2884.6 | 73.2 | 170.6 | 803.2 | 167.5 | 208.4 | 187.7 | | 213.4 | | 191.3 | | Memory Mark | 352.1 | 296.4 | 334 | 154 | 314.5 | 325.4 | 328.2 | | 319.9 | | 318.4 | | Disk Mark | 330.2 | 354.1 | 214.3 | 294.2 | 371.8 | 323.8 | 22.8 | | 375.3 | | 1307.7 | | CD Mark | --- | 1139.1 | 527.2 | --- | 201.2 | --- | --- | | --- | | 435.6 |

| PassMark Rating | 645.2 | 337.9 | 282.5 | 252.2 | 274 | 257.3 | 198.4 | | 248.3 | | 488.9 |

• Note that the products here break out into 4 'bins' of virtualization, as far as I'm concerned, and as a result, we should anticipate 'inherently' some performance differences between these approaches to virtualization:

  1 - VMWare Server: full hardware abstraction / emulation; no requirement for VT assist hardware; anticipated to have more abstraction / performance penalty.

  2 - XEN hypervisor: paravirtualization and full virtualization with VT assist. Xen and Virtual Iron are both in this category. Anticipated to have some performance hit.

  3 - KVM - Linux Kernel Virtualization Module: requires VT assist in this config; hypothetically anticipated to have less performance hit than Xen style hypervisor (?)

  4 - Container based: (Parallels/Virtuozzo) is not abstracting hardware, so is expected to have performance closest to bare metal.

• Note that the products here break out into 4 'bins' of virtualization, as far as I'm concerned, and as a result, we should anticipate 'inherently' some performance differenes between these approaches to virtualization:

  1 - VMWare Server - full hardware abstraction / emulation; no requirement for VT assist hardware; anticipated to have more abstraction / performance penalty.

  2 - XEN hypervisor - paravirtualization and full virtualization with VT assist. Xen and Virtual Iron are both in this category. Anticipated to have some performance hit.

  3 - KVM - Linux Kernel Virtualization Module - requires VT assist in this config; hypothetically anticipated to have less performance hit than Xen style hypervisor (?)

  4 - Container based: (Parallels/Virtuozzo) - is not abstracting hardware, so is expected to have performance closest to bare metal.

• Parallels Virtuozzo 4.0

• Note that the products here break out into 4 'bins' of virtualization, as far as I'm concerned:

1. VMWare Server - full hardware abstraction / emulation; no requirement for VT assist hardware; anticipated to have more abstraction / performance penalty.
2. XEN hypervisor - paravirtualization and full virtualization with VT assist. Xen and Virtual Iron are both in this category. Anticipated to have some performance hit.
3. KVM - Linux Kernel Virtualization Module - requires VT assist in this config; hypothetically anticipated to have less performance hit than Xen style hypervisor (?)
4. Container based: (Parallels/Virtuozzo) - is not abstracting hardware, so is expected to have performance closest to bare metal.

• I am running only the single virtualized system on the virtualization layer.

• Clearly I'm not using a "big" / production class physical host - and the intent of these 'basic tests' is simply to establish the "general overhead penalty" of the virtualization layer itself.

Take Note:

• I am running only the single virtualized OS on the virtualization layer.
• Typical production Virtualization deployments would be expected to have multiple virtual hosts on one physical host, which may impact performance metrics, etc.
• Clearly I'm not using a "big" physical host - and the intent of these 'basic tests' is simply to establish the "general overhead penalty" of the virtualization layer itself.

• Benchmark app used was "PassMark Performance Test" (ver.6.1), from URL, http://www.passmark.com/products/pt.htm

Virtualization platforms tested were:

Hardware used for tests was definitely "not" high end:

Test setup:

- It wasn't a huge download, and was easy to install

- seemed fairly broad based, covering a variety of CPU and Disk I/O performance metrics (which were of most interest to me)

- note that graphics/video performance is inherently of NO interest to me (sessions are in RDP) - I'm concerned only with "server performance" (CPU/IO) here.

Declared inconsistencies:

• Xen Express version 4.10

• VMWare Server (Free edition) version 1.0.5 build 80187
• ProxMox VE ver 0.9

Hardware used for tests was definitely "not" high end:

• MSI K9MM-V mainboard (socket AM2 / AMD)
• AMD Athlon 64 "3200+" 2.0ghz cpu
• 2 gigs DDR2 553 ram
• Local HDD, 320gig Seagate ATA

Test setup:

• Install virtualization layer
• Install Win2003 Server Std Trial edition in a virtual machine, with 512mb RAM available
• Patch Windows to latest state via Windows Update / Offline Update
• Run basic benchmark via RDP session to Win2003 host.
• Benchmark app used was XXXXX
• Rationale for this benchmark app:

  It wasn't a huge download, and was easy to install

  seemed fairly broad based, covering a variety of CPU and Disk I/O performance metrics (which were of most interest to me)

  note that graphics/video performance is inherently of NO interest to me (sessions are in RDP) - I'm concerned only with "server performance" (CPU/IO) here.

Declared inconsistencies:

• "Bare Metal" install has 1 gig RAM available, rather than 512mb - smallest part I had
• Virtual Iron Enterprise Edition uses iSCSI as the backing store for data, not a local disk - and thus, IMHO, it is not appropriate to compare the IO performance of VI_EE with the other IO results here. In particular, I was using a test-grade OpenFiler iSCSI target on old-grade hardware which doesn't offer same performance as local storage.
• usual benchmark disclaimers apply of course :-)

CONTEXT

In June-2008, in preparation for a number of client projects I was developing, it seemed prudent to perform some base installs of a number of 'standard' virtualization solutions, with the intent of confirming feature set; ease of installation; and performance of the various virtualization platforms on consistent hardware.

Clearly, benchmarks are imperfect and ultimately tell you only one thing (at best) - how well a given environment will run the chosen benchmark. This may or may not reflect on how well the environment will perform in other roles / for other benchmarks. However, such information may be of use / interest to compare and contrast features / environments / etc.

Virtualization platforms tested were,

• Xen Express version XXX
• Virtual Iron (Single Server Edition v 4.2, and Enterprise Edition, v 4.3)
• VMWare Server (Free edition) version XXX
• ProxMox VE ver 0.9