Additional related material:
Server Hardware 2010 Q3, Server Hardware 2009 Q3,
NEC Express5800/A1080a (2010-06),
Server Sizing (Interim) (2010-08),
Big Iron Revival (2009-05), Big Iron Revival II (2009-09), Big Iron Revival III (2010-09).
Intel Xeon 5600 and 7500 series (2010-04, this material has been updated in the new links above)
Since 2010, Intel has launched a refresh of the Xeon 5600 processors based on Westmere-EP, updated the high-end with Westmere-EX replacing Nehalem-EX, and introduced Sandy Bridge architecture processors on the low-end EN line.
The Xeon 5600 products introduced introduced in 2010 feature 6-cores only in the X models (95 & 130W), keeping the 80W E models at 4 cores. The 2011 refresh for the Xeon 5600 processsors include a frequency bump at the top from 3.33 to 3.46GHz, and a gap filler a 3.2GHz and the new E5649 and E5645 6-core 80W models.
The new Westmere-EX line replacing the Xeon 7500 are now the E7 series. The E7-8800 line are for system supporting 8 or more sockets. The E7-4800 line are for systems with upto 4 sockets. The E7-2800 line replaces the Xeon 6500 line for 2-way systems with very high memory capacity.
At the low-end, are the Sandy Bridge based E3 quad-core processors for single socket systems. From 1995 to the early 2000s, most servers started at 2-way. But over the last several years, especially with multi-core processors, 1-way systems are perfectly viable, even for database applications. The recommended system examples are given below for the current generation processors.
The recommended systems range from a single socket E3 servers at the low-end to 4 and 8-way at the high-end. The mid-range options are 2-way systems mostly based on either the Intel Xeon 5600 6-core or the AMD Opteron 12-core processors. The aggregate server application through-put performance between a Xeon system with 12 cores totals seems to be in the same range as an Opteron system with 24 cores total.
At the single core level, the Intel Xeon is more powerful than the Opteron, as so is a better fit for tasks that are not multi-threaded. One might think that server applications are multi-threaded, but this is frequently not the case for certain side tasks. One other option is a 2-way Xeon 6500 or E7-2800 system for applications that benefit from very high memory capacity.
At the high end, the options are Xeon E7 systems with 4 or 8 sockets or 4-way Opteron systems. At the 4-way system level, the Xeon systems have better throughput performance than Opteron, but the Opteron has lower cost, so there could be a fit for 4-way Opteron in situations sensitive to system cost, presumably in environments on CALS licensing.Entry systems
Below are Dell PowerEdge T110 II single socket system examples. The processor options for the T110 II include the dual-core Pentium G620/850, the Core I3-2100 and Xeon E3-1220 to 1270. The upper-end E3-1290 and 1280 are not options. The first item is priced with the E3-1220 processor, 8GB memory, and 2 x 250GB 7.2K SATA HDDs ($99) using the onboard SATA controller (software RAID no-cost option).
|T110 II||E3-1220||3.10||4||8M||8GB 2x4G||2x7.2K HD||$921|
|T110 II||E3-1270||3.40||4||8M||16GB 4x4G||2x15K HD||$2,167|
|T710||1x E5645||2.40||6||12M||24GB 6x4G||2x15K HD||$3,131|
The second system steps up to E3-1270 procesor, 16GB memory, H200 SAS/SATA RAID controller, and 2 x 146GB 15K 3.5in SAS hard drives for a total price differential of $1,246 between the two systems.
|upgrade to E3-1270||$230|
|upgrade to 16GB memory (4x4GB):||$257|
|H200 RAID controller:||$199|
|250GB 7.2K SATA HDD:||$99|
|146GB 15K 3.5in SAS HDD:||$379|
Further stepping up to 32GB memory (4x8GB) is a cost increment of $3,086, in which case it actually makes more sense to buy the 2-socket Xeon 5600 system with either 1 or 2 sockets populated. The third system is the T710 two-socket system with 1 socket populated, and 24GB memory. The lower price of memory on the T710 compared with the 32GB option on the T110 is why that configuration is not viable.
The T110 II has 2 x8 and 1x4 PCI-E G2 slots, along with 1 x1 G1 slot. The fact is that a single quad-processor today is far more powerful than the 4-socket systems of 10 years ago, and many applications should run fine on 16GB memory.
The only issue is disk IO. External storage could be connected, but that is really a better match for mid-range and high-end systems considering the expense of external storage. The best fit in the entry server, aside from low-activity databases, is to configure this system with one or two consumer grade PCI-E SSDs ($300-500 each) or SATA SSDs (60GB at $135 and 120GB at $220). Frequent backup to local hard disk should be sufficient to provide protection against data loss.
I have always strongly advocated the enterprise class 10K and 15K SAS hard disks for database storage.
There was just too much risk in the 7.2K SATA drives, even for backups.
However, for the very low-end, in the interest of getting cost as low as possible,
I would consider an entry system (1 processor socket) with 2 SATA disks in RAID 1
for the OS, executables and database backup, and having the main database and tempdb on
a single PCI-E SSD without RAID, or perhaps 2 SATA SSDs in RAID.
I would also suggest employing the 2.5in notebook HDD instead of 3.5in desktop HDD.
Below are 2-socket system examples built around the Xeon 5600 and E7 processor lines. The baseline system is a 2-way E5645, 24GB -6x4GB memory and 2x146GB 15K 2.5in HDDs, with 16 x 2.5in 6Gbps HDD bays. Stepping upto the Xeon X5690 is $1390 per processor, with no really good choices in between. The 48GB-6x8GB configuration is plus $900. Another 6x8GB to 96GB-12x8GB is $1,914. The next step to 192GB-12x16GB adds $6036. The practical price range spanned by the 2-socket Xeon 5600 systems is from $4-15K.
|T710||2x E5645||2.40||2x6||12M||5.86||24GB 6x4G||2x15K HD||$3,900|
|T710||2x X5690||3.40||2x6||12M||6.4||96GB 12x8G||2x15K HD||$9,494|
|T710||2x X5690||3.40||2x6||12M||6.4||192GB 12x16G||2x15K HD||$15,531|
Some component pricing below:
|upgrade to Xeon X5690||$1390 each|
|Memory 4, 8 and 16GB DIMMS :||$169, $319, $822 each|
|RAID controller H200, H800:||$199, $649-799|
|146GB 15K 2.5in SAS HDD:||$359 each|
|300GB 10K 2.5in SAS HDD:||$344 each|
|2 x 10GbE, Intel X520 DA:||$849|
Below are configuration examples with 2-sockets populated built around the Dell R810 with the Xeon E7 series processors. Both E7-28xx and E7-48xx options are possible. If only 2-sockets are planned, the 28xx series are slightly lower in price than comparable 48xx. The base system is with 2 x E7-2850 2.0GHz 10-core processors, and 128GB memory.
|R810||2x E7-2850||2.00||2x10||24M||6.4||128GB 32x4G||2x15K HD||$13,980|
|R810||2x E7-2870||2.40||2x10||30M||6.4||256GB 32x8G||2x15K HD||$23,192|
Component pricing below:
|upgrade to 2x E7-2870 2.4GHz||$3,920|
|Memory from 128G-32x4 to 256G-32x8:||$5,292|
|Memory from 128G-32x4 to 512G-32x16:||$21,237|
I recommend the R910 over of the R810 even with 2-socket populated. The R810 has adequate PCI-E slots for many database environments but the 5 extra x4 in the R910 are important for extra the PCI-E SSDs to achieve exceptional IO performance. Otherwise equivalent R910 configuration to the above are $15,765 and $25,177, or about $2K higher. Any database application important enough to need this kind of capability is also important enough to justify the 5U chassis.
In general, the T710 with 2 Xeon 5600 series processor is probably the best choice for most database applications. The R810 with E7 series processors has machine-check architecture (MCA) for higher reliability, and huge memory capacity. The nominal compute cycles, based on the number of core times the core GHz is about the same, but many application that still have non-parallel components would work better with the higher single core GHz of the Xeon 5600.
High-end 4-way systems
Below are 4-way system configuration examples for the Dell PowerEdge R910 (4 power supplies?).
|R910||4x E7-4850||2.00||2x10||24M||6.4||256GB 64x4G||2x15K HD||$30,075|
|R910||4x E7-4870||2.40||2x10||30M||6.4||512GB 64x8G||2x15K HD||$48,898|
|R910||4x E7-4870||2.40||2x10||30M||6.4||1TB 64x16G||2x15K HD||$80,316|
Component pricing below for the above Dell systems. The brand new super high capacity 32GB does not make sense right now, but should become a viable option in about 2 years from now.
|Upgrade E7-4850 to E7-4870||$4,120 per pair|
Below are 4-way system configuration examples for the HP ProLiant DL580 G7, with 4 power supplies.
|DL580 G7||4x E7-4850||2.00||2x10||24M||6.4||256GB 64x4G||2x15K HD||$35,031|
|DL580 G7||4x E7-4870||2.40||2x10||30M||6.4||512GB 64x8G||2x15K HD||$52,871|
|DL580 G7||4x E7-4870||2.40||2x10||30M||6.4||1TB 64x16G 4R||2x15K HD||$89,991|
|DL580 G7||4x E7-4870||2.40||2x10||30M||6.4||1TB 64x16G 2R||2x15K HD||$118,791|
Component pricing for the HP ProLiant systems:
|Memory 128G-32x4 LV||$3,896||$121|
|Memory 256G-32x8 LV||$10,208||$319|
|Memory 512G-32x16 4 Rank||$28,768||$899|
|Memory 512G-32x16 2 Rank LV||$43,168||$1349|
|146GB 15K 2.5in SAS HDD:||$369|
|300GB 10K 2.5in SAS HDD:||$349|
Below are 8-way system configuration examples for the HP ProLiant DL980 G7, with 8 power supplies.
|DL980 G7||8x E7-4870||2.40||2x10||30M||6.4||1TB 128x8G||2x15K HD||$137,761|
|DL980 G7||8x E7-4870||2.40||2x10||30M||6.4||2TB 128x16G 4R||2x15K HD||$212,001|
|DL980 G7||8x E7-4870||2.40||2x10||30M||6.4||2TB 128x16G 2R||2x15K HD||$269,601|
Before SSD, there is no practical way of accommodating sufficient disks internally to support a serious database. For external storage, on the Dell side, the PowerVault MD1220 should be able to deliver 2GB/s with 24 disks. Configured with 24 146GB 15K or 300GB 10K disks at a price of $11.5K, or $9K with 146GB 10K disks. On the HP side, HP has no updated the direct-attach MSA70 to 6Gbps SAS, so the only viable option is the entry SAN, the P2000G3. I would prefer the SAS interface, but the SFF 2.5in disks is required for density, so the FC dual-controller may be the only option(?). With 24 x 146GB 15K disks, this is appears to carry a price of $20K.
As an approximate guideline, one external storage unit with 24 disks and adequate IO channels per processor socket is a reasonable configuration for situations where the active portion of the database exceeds the memory buffer. To a degree, a system with huge memory can reduce the need for disk IO, but this leads to an unbalanced configuration that runs fine for the main tasks, but is bottlenecked on other tasks. The HP ML/DL 370 can support some databases with internal disks.
With Solid-State devices becoming practical and sufficiently mature, it is actually now practical to configure sufficient IO performance internally. The more practical SSD solution today is the PCI-E interface. There is not anything fundamentally wrong with SSD on the SATA and SAS interfaces, but rather that the storage enclosure products today does not match bandwidth to the number of bays for SSD. With SSDs, we might want 4-5 bays per x4 SAS channel instead of 24 suitable for hard disks. The key is having the right mix of IO slots to match the available SSD products. Most PCI-E SSDs today have bandwidth in the 1-2GB/s range, which is a good match for a x4 PCI-E Gen 2 slot. A x8 Gen 2 slot can support 3.2GB/s net bandwidth. Most server systems have a mix of x8 and x4 slots, so we do not have the best server - storage match.
For PCI-E SSD, we currently have the following range of products. Fusio-IO has PCI-E gen 2 SLC products at 320 and 640GB and MLC products at 640 and 1280GB. There are PCI-E gen 1 SLC at 160 and 320GB, and MLC at 320 and 640GB. Pricing for Fusion seems to be about $15K per 320GB on SLC, and same for 640GB MLC.
OCZ has consumer an enterprise PCI-E SSD products. On the consumer side, the RevoDrive X2 is currently available, with the X3 coming soon? The RevoDrive X2 ranges from $450 at 160G, $980 at 360GB to $1350 at 480GB and $3159 at 960GB. For the enterprise class MLC, the VeloDrive is $2249 at 300GB.
For the high-end 2-way system, the Xeon 5600 series is the better choice on the strength of the single core performance. The multi-core performance of the 2 x 6-core Xeon 5680 and the 2 x 12 core Opteron 6176 are close in TPC-H, the Xeon 5680 has better OLTP performance most probably because of Hyper-Threading, and the single core performance is far superior.
Below are the technical detail highlights for the Dell PowerEdge Servers most suitable for database servers. The key system features, aside from the processor sockets, are the number DIMM sockets and PCI-E slots. The number DIMM sockets, and the economical maximum memory used to be extremely important. Today, the memory capacity of most systems is beyond reasonable database requirement, and this aspect is nolonger the principal metric. In the days of disk storage, the preferred PCI-E slot width was x8, with 1 x4 slot for additional network I/O. Storage adapters, both RAID controllers and HBAs, were generally matched to the bandwidth of a x8 slot, for both the Gen 1 and Gen2 time frames. Some system have one or more x16 slots. Because no server adapter matched the x16 bandwidth, this was really a waste of PCI-E bandwidth. Today, with SSD storage on the rise, most PCI-E SSDs seem to be matched to the bandwidth of a x4 PCI-E Gen 2 slot (2.0GB/s nominal, 1.6GB/s net).
|System||Sockets||DIMMs||PCI-E x16/x8/x4||Int Drives||Notes|
|T110 II||1||4||0 / 2 / 1||4/6?||4U|
|R210 II||1||4||1 / 0 / 0||2/4||1U|
|T710||2||18(12)||1 / 4+1 / 1||8/16+2||5U|
|R710||2||18(12)||0 / 2 / 2+1||6/8||2U|
|R810||4||32||0 / 5 / 1+1||-/6||2U|
|R910||4||64||0 / 4+1 / 6||-/16||4U|
The T710 and R710 have 18 DIMM sockets, but only supports 18x8GB = 144GB or 12x16GB = 192GB. One PCI-E slots, +1 indicated slot dedicated to the internal storage controller. Where there are PCI-E slot configuration options, the most suitable for database environments is selected. Internal drives bays numbers are for 3.5in or 2.5in, with + for optional media bay.
For the above Dell servers, when there is a choice between the T and R models, my preference is for the T models, mostly on the basis of the number of PCI-E slots. Multiple PCI-E SSDs are better than fewer, regardless of whatever fantasies a sales rep might have about SSD performance.
Below are systems detail for mid-range and high-end HP ProLiant servers. HP servers are higher priced than otherwise comparable Dell servers, but the database oriented models have more PCI-E slots and bandwidth.
|System||Sockets||DIMMs||PCI-E x16/x8/x4||Int Drives||Notes|
|ML370G6||2||18(12)||2 / 1+1 / 5||8/24||5U|
|DL380G7||2||18(12)||0 / 2 / 4||-/8+8||2U|
|DL580G7||4||64||0 / 6 / 4||-/8||4U|
|DL980G7||8||128||0 / 10 / 5||-/8||7U?|
The HP 370/380 supports 12x32GB memory. One x8 slot on the 370 is for embedded NIC. There is also one PCI-E G1 x4 slot in the HP systems above.
The number of internal storage bays should match the market segment. In a low-end system, it desired to have sufficient internal storage bays for the complete environment. The avoids the need for external storage, for which the bare disk array enclosure could cost more than the server, and has the benefit of simplicity. On the other hand, a high-end system will need external storage for the main database, and the internal disk drives only need to support the OS. It would seem 2 internal drive bays is sufficient. This was expanded to 4 because a number of people insist on RAID 5 even for the OS boot drive, without really understanding why. Because people seem to also want to buy very expensive SAN storage, which was usually under the control of a different group than the database group, getting extra storage space allocation for auxilary functions was exceedingly difficult. So people decided it was better to have extra local storage in high-end servers, if only to reduce the need to communicate with the storage group.
Lets take a look at the Xeon 5600, 7500 and 6500 SKUs. The low-voltage, low power SKUs are omitted. These are fine products for high-density environments, web servers, and utility database. The Line-of-business and DW databases should be on the X models.New Xeon 5600, 2011 Jul 10
Original Xeon 5600, 2010 Q2
* Intel 1k pricing, lower SKU's omitted.Xeon E7-x800 (Westmere-EX), 2011 Jul 10
The E7-28xx series are otherwise the same as the E7-4800 with slightly lower price ($4,227, 3,670, 2,558, 1,779, 1,334, 774 respectively).Xeon 7500 (Nehalem-EX), 2010 Q3
Xeon 6500 SKUs
Before commenting, recall the main differences between the Xeon 5600 and Xeon 7500/6500 series. The Xeon 5600 series (32nm process) has 2 QPI links and 3 memory channels. The Xeon 7500 series (45nm process) has 4 QPI links, 4 memory channel, larger cache per core (for the 24M version, 3M vs 2M) plus extensive reliability features. The 2 QPI links on the 5600 series allows a 2-way (socket) system. The 4 QPI links on the 7500 series allows glueless 4-way and 8-way. My understanding is the 6500 series is the 7500 with only 2 QPI links enable for 2-way systems with 16-cores and 8 memory channels total, at lower frequency than the 5600 with 12-cores and 6 memory channels total, plus the 7500 RAS features.