Home, Cost-Based Optimizer, Benchmarks, Server Systems, System Architecture, Processors, Storage, TPC-H Studies

Server Systems 2011 Q3 (Updated 2011-07)

Additional related material:
 Server Hardware 2010 Q3, Server Hardware 2009 Q3,
 NEC Express5800/A1080a (2010-06),
 Server Sizing (Interim) (2010-08),
 Big Iron Revival (2009-05), Big Iron Revival II (2009-09), Big Iron Revival III (2010-09).
 Intel Xeon 5600 and 7500 series (2010-04, this material has been updated in the new links above)

Since 2010, Intel has launched a refresh of the Xeon 5600 processors based on Westmere-EP, updated the high-end with Westmere-EX replacing Nehalem-EX, and introduced Sandy Bridge architecture processors on the low-end EN line.

The Xeon 5600 products introduced introduced in 2010 feature 6-cores only in the X models (95 & 130W), keeping the 80W E models at 4 cores. The 2011 refresh for the Xeon 5600 processsors include a frequency bump at the top from 3.33 to 3.46GHz, and a gap filler a 3.2GHz and the new E5649 and E5645 6-core 80W models.

The new Westmere-EX line replacing the Xeon 7500 are now the E7 series. The E7-8800 line are for system supporting 8 or more sockets. The E7-4800 line are for systems with upto 4 sockets. The E7-2800 line replaces the Xeon 6500 line for 2-way systems with very high memory capacity.

At the low-end, are the Sandy Bridge based E3 quad-core processors for single socket systems. From 1995 to the early 2000s, most servers started at 2-way. But over the last several years, especially with multi-core processors, 1-way systems are perfectly viable, even for database applications. The recommended system examples are given below for the current generation processors.

Recommended Systems 2011 Q3

The recommended systems range from a single socket E3 servers at the low-end to 4 and 8-way at the high-end. The mid-range options are 2-way systems mostly based on either the Intel Xeon 5600 6-core or the AMD Opteron 12-core processors. The aggregate server application through-put performance between a Xeon system with 12 cores totals seems to be in the same range as an Opteron system with 24 cores total.

At the single core level, the Intel Xeon is more powerful than the Opteron, as so is a better fit for tasks that are not multi-threaded. One might think that server applications are multi-threaded, but this is frequently not the case for certain side tasks. One other option is a 2-way Xeon 6500 or E7-2800 system for applications that benefit from very high memory capacity.

At the high end, the options are Xeon E7 systems with 4 or 8 sockets or 4-way Opteron systems. At the 4-way system level, the Xeon systems have better throughput performance than Opteron, but the Opteron has lower cost, so there could be a fit for 4-way Opteron in situations sensitive to system cost, presumably in environments on CALS licensing.

Entry systems

Below are Dell PowerEdge T110 II single socket system examples. The processor options for the T110 II include the dual-core Pentium G620/850, the Core I3-2100 and Xeon E3-1220 to 1270. The upper-end E3-1290 and 1280 are not options. The first item is priced with the E3-1220 processor, 8GB memory, and 2 x 250GB 7.2K SATA HDDs ($99) using the onboard SATA controller (software RAID no-cost option).

SystemProcessorGHzCoresL3MemoryNotesPrice
T110 IIE3-12203.1048M8GB 2x4G2x7.2K HD$921
T110 IIE3-12703.4048M16GB 4x4G2x15K HD$2,167
T7101x E56452.40612M24GB 6x4G2x15K HD$3,131

The second system steps up to E3-1270 procesor, 16GB memory, H200 SAS/SATA RAID controller, and 2 x 146GB 15K 3.5in SAS hard drives for a total price differential of $1,246 between the two systems.

 upgrade to E3-1270$230
 upgrade to 16GB memory (4x4GB):$257
 H200 RAID controller:$199
 250GB 7.2K SATA HDD: $99
 146GB 15K 3.5in SAS HDD:$379

Further stepping up to 32GB memory (4x8GB) is a cost increment of $3,086, in which case it actually makes more sense to buy the 2-socket Xeon 5600 system with either 1 or 2 sockets populated. The third system is the T710 two-socket system with 1 socket populated, and 24GB memory. The lower price of memory on the T710 compared with the 32GB option on the T110 is why that configuration is not viable.

The T110 II has 2 x8 and 1x4 PCI-E G2 slots, along with 1 x1 G1 slot. The fact is that a single quad-processor today is far more powerful than the 4-socket systems of 10 years ago, and many applications should run fine on 16GB memory.

The only issue is disk IO. External storage could be connected, but that is really a better match for mid-range and high-end systems considering the expense of external storage. The best fit in the entry server, aside from low-activity databases, is to configure this system with one or two consumer grade PCI-E SSDs ($300-500 each) or SATA SSDs (60GB at $135 and 120GB at $220). Frequent backup to local hard disk should be sufficient to provide protection against data loss.

I have always strongly advocated the enterprise class 10K and 15K SAS hard disks for database storage. There was just too much risk in the 7.2K SATA drives, even for backups. However, for the very low-end, in the interest of getting cost as low as possible, I would consider an entry system (1 processor socket) with 2 SATA disks in RAID 1 for the OS, executables and database backup, and having the main database and tempdb on a single PCI-E SSD without RAID, or perhaps 2 SATA SSDs in RAID.
I would also suggest employing the 2.5in notebook HDD instead of 3.5in desktop HDD.

Mid-range systems

Below are 2-socket system examples built around the Xeon 5600 and E7 processor lines. The baseline system is a 2-way E5645, 24GB -6x4GB memory and 2x146GB 15K 2.5in HDDs, with 16 x 2.5in 6Gbps HDD bays. Stepping upto the Xeon X5690 is $1390 per processor, with no really good choices in between. The 48GB-6x8GB configuration is plus $900. Another 6x8GB to 96GB-12x8GB is $1,914. The next step to 192GB-12x16GB adds $6036. The practical price range spanned by the 2-socket Xeon 5600 systems is from $4-15K.

SystemProcessorGHzCoresL3QPIMemoryNotesPrice
T7102x E56452.402x612M5.8624GB 6x4G2x15K HD$3,900
T7102x X56903.402x612M6.4 96GB 12x8G2x15K HD$9,494
T7102x X56903.402x612M6.4 192GB 12x16G2x15K HD$15,531

Some component pricing below:

 upgrade to Xeon X5690$1390 each
 Memory 4, 8 and 16GB DIMMS :$169, $319, $822 each
 RAID controller H200, H800:$199, $649-799
 146GB 15K 2.5in SAS HDD:$359 each
 300GB 10K 2.5in SAS HDD:$344 each
 2 x 10GbE, Intel X520 DA:$849

Below are configuration examples with 2-sockets populated built around the Dell R810 with the Xeon E7 series processors. Both E7-28xx and E7-48xx options are possible. If only 2-sockets are planned, the 28xx series are slightly lower in price than comparable 48xx. The base system is with 2 x E7-2850 2.0GHz 10-core processors, and 128GB memory.

SystemProcessorGHzCoresL3QPIMemoryNotesPrice
R8102x E7-28502.002x1024M6.4128GB 32x4G2x15K HD$13,980
R8102x E7-28702.402x1030M6.4256GB 32x8G2x15K HD$23,192

Component pricing below:

 upgrade to 2x E7-2870 2.4GHz$3,920
 Memory from 128G-32x4 to 256G-32x8:$5,292
 Memory from 128G-32x4 to 512G-32x16:$21,237

I recommend the R910 over of the R810 even with 2-socket populated. The R810 has adequate PCI-E slots for many database environments but the 5 extra x4 in the R910 are important for extra the PCI-E SSDs to achieve exceptional IO performance. Otherwise equivalent R910 configuration to the above are $15,765 and $25,177, or about $2K higher. Any database application important enough to need this kind of capability is also important enough to justify the 5U chassis.

In general, the T710 with 2 Xeon 5600 series processor is probably the best choice for most database applications. The R810 with E7 series processors has machine-check architecture (MCA) for higher reliability, and huge memory capacity. The nominal compute cycles, based on the number of core times the core GHz is about the same, but many application that still have non-parallel components would work better with the higher single core GHz of the Xeon 5600.

High-end 4-way systems

Below are 4-way system configuration examples for the Dell PowerEdge R910 (4 power supplies?).

SystemProcessorGHzCoresL3QPIMemoryNotesPrice
R9104x E7-48502.002x1024M6.4256GB 64x4G2x15K HD$30,075
R9104x E7-48702.402x1030M6.4512GB 64x8G2x15K HD$48,898
R9104x E7-48702.402x1030M6.41TB 64x16G2x15K HD$80,316

Component pricing below for the above Dell systems. The brand new super high capacity 32GB does not make sense right now, but should become a viable option in about 2 years from now.

 ComponentSetPer Unit
 Upgrade E7-4850 to E7-4870$4,120 per pair 
 Memory 128G-32x4:$5,715$178
 Memory 256G-32x8:$10,208$319
 Memory 512G-32x16:$25,681$802
 Memory 1T-32x32:$177,748$5,554

Below are 4-way system configuration examples for the HP ProLiant DL580 G7, with 4 power supplies.

SystemProcessorGHzCoresL3QPIMemoryNotesPrice
DL580 G74x E7-48502.002x1024M6.4256GB 64x4G2x15K HD$35,031
DL580 G74x E7-48702.402x1030M6.4512GB 64x8G2x15K HD$52,871
DL580 G74x E7-48702.402x1030M6.41TB 64x16G 4R2x15K HD$89,991
DL580 G74x E7-48702.402x1030M6.41TB 64x16G 2R2x15K HD$118,791

Component pricing for the HP ProLiant systems:

 ComponentSetPer Unit
 E7-4850 $3,449
 E7-4870 $5,509
 Memory 128G-32x4 LV$3,896$121
 Memory 256G-32x8 LV$10,208 $319
 Memory 512G-32x16 4 Rank$28,768$899
 Memory 512G-32x16 2 Rank LV$43,168$1349
 Memory 1T-32x32:$143,968$4,499
 146GB 15K 2.5in SAS HDD: $369
 300GB 10K 2.5in SAS HDD: $349

High-end 8-way systems

Below are 8-way system configuration examples for the HP ProLiant DL980 G7, with 8 power supplies.

SystemProcessorGHzCoresL3QPIMemoryNotesPrice
DL980 G78x E7-48702.402x1030M6.41TB 128x8G2x15K HD$137,761
DL980 G78x E7-48702.402x1030M6.42TB 128x16G 4R2x15K HD$212,001
DL980 G78x E7-48702.402x1030M6.42TB 128x16G 2R2x15K HD$269,601

Storage - Brief

Before SSD, there is no practical way of accommodating sufficient disks internally to support a serious database. For external storage, on the Dell side, the PowerVault MD1220 should be able to deliver 2GB/s with 24 disks. Configured with 24 146GB 15K or 300GB 10K disks at a price of $11.5K, or $9K with 146GB 10K disks. On the HP side, HP has no updated the direct-attach MSA70 to 6Gbps SAS, so the only viable option is the entry SAN, the P2000G3. I would prefer the SAS interface, but the SFF 2.5in disks is required for density, so the FC dual-controller may be the only option(?). With 24 x 146GB 15K disks, this is appears to carry a price of $20K.

As an approximate guideline, one external storage unit with 24 disks and adequate IO channels per processor socket is a reasonable configuration for situations where the active portion of the database exceeds the memory buffer. To a degree, a system with huge memory can reduce the need for disk IO, but this leads to an unbalanced configuration that runs fine for the main tasks, but is bottlenecked on other tasks. The HP ML/DL 370 can support some databases with internal disks.

With Solid-State devices becoming practical and sufficiently mature, it is actually now practical to configure sufficient IO performance internally. The more practical SSD solution today is the PCI-E interface. There is not anything fundamentally wrong with SSD on the SATA and SAS interfaces, but rather that the storage enclosure products today does not match bandwidth to the number of bays for SSD. With SSDs, we might want 4-5 bays per x4 SAS channel instead of 24 suitable for hard disks. The key is having the right mix of IO slots to match the available SSD products. Most PCI-E SSDs today have bandwidth in the 1-2GB/s range, which is a good match for a x4 PCI-E Gen 2 slot. A x8 Gen 2 slot can support 3.2GB/s net bandwidth. Most server systems have a mix of x8 and x4 slots, so we do not have the best server - storage match.

For PCI-E SSD, we currently have the following range of products. Fusio-IO has PCI-E gen 2 SLC products at 320 and 640GB and MLC products at 640 and 1280GB. There are PCI-E gen 1 SLC at 160 and 320GB, and MLC at 320 and 640GB. Pricing for Fusion seems to be about $15K per 320GB on SLC, and same for 640GB MLC.

OCZ has consumer an enterprise PCI-E SSD products. On the consumer side, the RevoDrive X2 is currently available, with the X3 coming soon? The RevoDrive X2 ranges from $450 at 160G, $980 at 360GB to $1350 at 480GB and $3159 at 960GB. For the enterprise class MLC, the VeloDrive is $2249 at 300GB.

Summary

For the high-end 2-way system, the Xeon 5600 series is the better choice on the strength of the single core performance. The multi-core performance of the 2 x 6-core Xeon 5680 and the 2 x 12 core Opteron 6176 are close in TPC-H, the Xeon 5680 has better OLTP performance most probably because of Hyper-Threading, and the single core performance is far superior.

Appendix

System Memory, PCI-E and Internal Storage Details

Below are the technical detail highlights for the Dell PowerEdge Servers most suitable for database servers. The key system features, aside from the processor sockets, are the number DIMM sockets and PCI-E slots. The number DIMM sockets, and the economical maximum memory used to be extremely important. Today, the memory capacity of most systems is beyond reasonable database requirement, and this aspect is nolonger the principal metric. In the days of disk storage, the preferred PCI-E slot width was x8, with 1 x4 slot for additional network I/O. Storage adapters, both RAID controllers and HBAs, were generally matched to the bandwidth of a x8 slot, for both the Gen 1 and Gen2 time frames. Some system have one or more x16 slots. Because no server adapter matched the x16 bandwidth, this was really a waste of PCI-E bandwidth. Today, with SSD storage on the rise, most PCI-E SSDs seem to be matched to the bandwidth of a x4 PCI-E Gen 2 slot (2.0GB/s nominal, 1.6GB/s net).

SystemSocketsDIMMsPCI-E x16/x8/x4Int DrivesNotes
T110 II140 / 2 / 14/6?4U
R210 II141 / 0 / 02/41U
T710218(12)1 / 4+1 / 18/16+25U
R710218(12)0 / 2 / 2+16/82U
R8104320 / 5 / 1+1-/62U
R9104640 / 4+1 / 6-/164U

The T710 and R710 have 18 DIMM sockets, but only supports 18x8GB = 144GB or 12x16GB = 192GB. One PCI-E slots, +1 indicated slot dedicated to the internal storage controller. Where there are PCI-E slot configuration options, the most suitable for database environments is selected. Internal drives bays numbers are for 3.5in or 2.5in, with + for optional media bay.

For the above Dell servers, when there is a choice between the T and R models, my preference is for the T models, mostly on the basis of the number of PCI-E slots. Multiple PCI-E SSDs are better than fewer, regardless of whatever fantasies a sales rep might have about SSD performance.

Below are systems detail for mid-range and high-end HP ProLiant servers. HP servers are higher priced than otherwise comparable Dell servers, but the database oriented models have more PCI-E slots and bandwidth.

SystemSocketsDIMMsPCI-E x16/x8/x4Int DrivesNotes
ML370G6218(12)2 / 1+1 / 58/245U
DL380G7218(12)0 / 2 / 4-/8+82U
DL580G74640 / 6 / 4-/84U
DL980G781280 / 10 / 5-/87U?

The HP 370/380 supports 12x32GB memory. One x8 slot on the 370 is for embedded NIC. There is also one PCI-E G1 x4 slot in the HP systems above.

The number of internal storage bays should match the market segment. In a low-end system, it desired to have sufficient internal storage bays for the complete environment. The avoids the need for external storage, for which the bare disk array enclosure could cost more than the server, and has the benefit of simplicity. On the other hand, a high-end system will need external storage for the main database, and the internal disk drives only need to support the OS. It would seem 2 internal drive bays is sufficient. This was expanded to 4 because a number of people insist on RAID 5 even for the OS boot drive, without really understanding why. Because people seem to also want to buy very expensive SAN storage, which was usually under the control of a different group than the database group, getting extra storage space allocation for auxilary functions was exceedingly difficult. So people decided it was better to have extra local storage in high-end servers, if only to reduce the need to communicate with the storage group.

Intel Xeon 5600 (Westmere-EP) Refresh and E7 series (Westmere-EX)

Lets take a look at the Xeon 5600, 7500 and 6500 SKUs. The low-voltage, low power SKUs are omitted. These are fine products for high-density environments, web servers, and utility database. The Line-of-business and DW databases should be on the X models.

New Xeon 5600, 2011 Jul 10
ModelCoresThreadsGHzTurboL3QPI GT/sMemoryPowerPrice*
X56906123.463.612M6.41333130$1,663
X56756123.063.3312M6.4133395$1,440
E56496122.533.0612M5.86133380$774
E56456122.402.9312M5.86106680$551
X5687483.603.7312M6.41333130$1,693
X5672483.203.4612M6.4133395$1,440

Original Xeon 5600, 2010 Q2

ModelCoresThreadsGHzTurboL3QPI GT/sMemoryPowerPrice*
X56806123.333.612M6.41333130$1,663
X56706122.933.3312M6.4133395$1,440
X56606122.803.212M6.4133395$1,219
X56506122.663.0612M6.4133395$996
E5640482.662.9312M5.86106680$774
E5630482.532.812M5.86106680$551
E5620482.402.6612M5.86106680$387
X5677483.463.7312M6.41333130$1,693
X5667483.063.4612M6.4133395$1,440

* Intel 1k pricing, lower SKU's omitted.

Xeon E7-x800 (Westmere-EX), 2011 Jul 10
ModelCoresThreadsGHzTurboL3QPI GT/sMemoryPowerPrice*
E7-8870 10202.402.8 30M6.4?130$4,616
E7-8860 10202.262.6624M6.4?130$4,061
E7-8850 10202.002.4 24M6.4?130$3,059
E7-8830 8162.132.4 24M6.4?105$2,280
E7-8867L10202.132.5330M6.4?105$4,172
E7-8837L 8 82.672.8 24M6.4?130$2,280

ModelCoresThreadsGHzTurboL3QPI GT/sMemoryPowerPrice*
E7-487010202.402.8 30M6.4?130$4,394
E7-486010202.262.6624M6.4?130$3,838
E7-485010202.002.4 24M6.4?130$2,837
E7-4830 8162.132.4 24M6.4?105$2,059
E7-4820 8162.002.2618M5.86?105$1,446
E7-4807 6121.861.8618M4.8?95$890

The E7-28xx series are otherwise the same as the E7-4800 with slightly lower price ($4,227, 3,670, 2,558, 1,779, 1,334, 774 respectively).

Xeon 7500 (Nehalem-EX), 2010 Q3
ModelCoresThreadsGHzTurboL3QPI GT/sMemoryPowerPrice*
X75608162.262.6624M6.41066?130$3,692
X75508162.002.418M6.4?130$2,729
E75406122.002.2618M6.4?105$1,980
E75306121.862.1318M5.86?105$1,391
E7520481.861.8618M4.8?95$856
X7542662.662.818M5.86?130$1,980

Xeon 6500 SKUs

ModelCoresThreadsGHzTurboL3QPI GT/sMemoryPowerPrice*
X65508162.002.418M6.4?130$2,461
E65406122.002.2618M6.4?105$1,712
E6510481.731.7312M4.8?105$744

Xeon E3 (Sandy Bridge-EN) SKUs
ModelCoresThreadsGHzTurboL3QPI GT/sMemoryPowerPrice*
E3-1290483.60?8Mn/a?95$885
E3-1280483.50?8Mn/a?95$612
E3-1275483.40?8Mn/a?80$339
E3-1270483.40?8Mn/a?80$328
E3-1240483.30?8Mn/a?80$250
E3-1230483.20?8Mn/a?80$215

Before commenting, recall the main differences between the Xeon 5600 and Xeon 7500/6500 series. The Xeon 5600 series (32nm process) has 2 QPI links and 3 memory channels. The Xeon 7500 series (45nm process) has 4 QPI links, 4 memory channel, larger cache per core (for the 24M version, 3M vs 2M) plus extensive reliability features. The 2 QPI links on the 5600 series allows a 2-way (socket) system. The 4 QPI links on the 7500 series allows glueless 4-way and 8-way. My understanding is the 6500 series is the 7500 with only 2 QPI links enable for 2-way systems with 16-cores and 8 memory channels total, at lower frequency than the 5600 with 12-cores and 6 memory channels total, plus the 7500 RAS features.