Home, Cost-Based Optimizer, Benchmarks, Server Systems, System Architecture, Processors, Storage, TPC-H Studies
Additional related material:
Server Hardware 2009 Q3,
NEC Express5800/A1080a (2010-06),
Server Sizing (Interim) (2010-08),
Big Iron Revival (2009-05),
Big Iron Revival II (2009-09),
Big Iron Revival III (2010-09).
Intel Xeon 5600 and 7500 series (2010-04, this material has been updated in the new links above)
| System | Processor | GHz | Cores | L3 | Memory | Price |
|---|---|---|---|---|---|---|
| A | 1x Xeon E3440 | 2.53 | 4 | 8M | 12GB 2x4G | $1,750 |
| B | 2x Xeon E5620 | 2.40 | 4 | 12M | 24GB 6x4G | $4,644 |
| C | 2x Xeon X5680 | 3.33 | 6 | 12M | 48GB 6x8G | $9,000 |
| D-1 | 4x Opteron 6174 | 2.2 | 12 | 12M | 128GB 32x4G | $14,100 |
| D-2 | 4x Xeon X7560 | 2.26 | 8 | 24M | 128GB 16x8G | $30,300 |
| E | 8x Xeon X7560 | 2.26 | 8 | 24M | 256GB 32x8G | $100K? |
| Price includes 2 x 73GB 15K HDD, but not the main storage system. | ||||||
The table above spans a broad price range. The choice represent my opinion of the best system in the price range. For the high-end 2-way system, the Xeon 5600 series is the better choice on the strength of the single core performance. The multi-core performance of the 2 x 6-core Xeon 5680 and the 2 x 12 core Opteron 6176 are close in TPC-H, the Xeon 5680 has better OLTP performance most probably because of Hyper-Threading, and the single core performance is far superior.
The table below shows selected system specifications.
| Processors | Cores | DIMM | PCI-E G2 | Max Memory | Total cores |
|---|---|---|---|---|---|
| 2x Xeon X56x0 | 6 | 18 | 5 x8, 1 x4 | 192GB | 12 |
| 4x Opteron 6100 | 12 | 32 | 5 x8, 1 x4 | 512GB | 48 |
| 4x Xeon X7560 | 8 | 64 | 4 x8, 6 x4 | 1TB | 32 |
| 8x Xeon X7560 | 8 | 128 | 9 x8, 5 x4 | 2TB | 64 |
Last year, Intel announced that there 8 system vendors preparing 15 8-way and 16-way designs for the Xeon 7500 processor. At the time I speculated on which vendors these might be. Since product launch earlier this year, here is the following situation:
| 4-way | 8-way | 16-way | |
|---|---|---|---|
| Dell | R910 | n/a | n/a |
| HP | DL580G7 | DL980G7 (soon?) | - |
| IBM | x3850 X5 | x3850 X5? | - |
| Sun | - | - | - |
| Unisys | ES7000 Model 7600R G2 | 7600R G2 | - |
| Fujitsu | PRIMERGY RX900 S5 | PRIMEQUEST 1800E | - |
| NEC | A1080a-E | Express 5800/A1080a-E | - |
| Cisco | - | - | - |
| SGI | - | - | - |
| ? | - | - | - |
At the time of the Intel Xeon 7500 launch, Dell, IBM and NEC had systems. Dell has stated that they do not intend to play in the 8-way space, but Dell does have a pair of Xeon 7500 4-way systems. HP just announced the 8-way DL980G7 and 4-way DL580G7 plus blade systems, with published benchmarks on both systems. IBM has 4-way system convertible to 8-way, but only published TPC benchmarks on the 4-way. Sun has no Xeon 7500 based systems. Sun does not even have systems for Xeon 5600 processors, which is socket compatible with Xeon 5500. Unisys has not announced systems, but is expected to. Fujitsu has a 4-way system. NEC has an 8-way system with TPC-E benchmark publish at the Xeon 7500 product launch.
The ordering of the above is as follows: The three major US-base server vendors in alphabetical order, followed by Sun (which has a full system lineup), followed US-based high-end system specialist Unisys, then followed by non-US based system vendors. Other than that, nothing should be inferred from the order of listing. In various places, I have discussed the performance benchmark results, and I have cited system configuration exampled with Dell and HP servers. IBM is also a major player in the US server market, but I have not cited examples with IBM servers because I do not have recent direct experience on their systems.
As of late June: HP, IBM and NEC have announced 8-way Xeon 7500 systems, of which the NEC system is shipping. It is expected that Unisys will have an 8-way and/or 16-way system. It would have been previously expected that Sun intended to launch an 8-way Xeon system. However, now that Oracle has acquired Sun, this is unclear. Sun has announced that they will shift future system development away from AMD Opteron to just Intel Xeon on the non-SPARC side. The Oracle strategy seems revolves around RAC and Oracle Database Machine, with the emphasis on really inexpensive 2-way and maybe 4-way systems and really expensive Oracle licenses. The other candidates for fielding 8-way servers include: Fujitsu, Hitachi and Bull
There has been no word of a 16-way system, but any vendor with commitments to large Itanium systems are candidates.
Dell is major player in the server, but it could be that the anticipated volume of the 8-way is too small.
Lets take a look at the Xeon 5600, 7500 and 6500 SKUs. The low-voltage, low power SKUs are omitted. These are fine products for high-density environments, web servers, and utility database. The Line-of-business and DW databases should be on the X models.
Xeon 5600 SKUs| Model | Cores | Threads | GHz | Turbo | L3 | QPI GT/s | Memory | Power | Price* |
|---|---|---|---|---|---|---|---|---|---|
| X5680 | 6 | 12 | 3.33 | 3.6 | 12M | 6.4 | 1333 | 130 | $1,663 |
| X5670 | 6 | 12 | 2.93 | 3.33 | 12M | 6.4 | 1333 | 95 | $1,440 |
| X5660 | 6 | 12 | 2.80 | 3.2 | 12M | 6.4 | 1333 | 95 | $1,219 |
| X5650 | 6 | 12 | 2.66 | 3.06 | 12M | 6.4 | 1333 | 95 | $996 |
| E5640 | 4 | 8 | 2.66 | 2.93 | 12M | 5.86 | 1066 | 80 | $774 |
| E5630 | 4 | 8 | 2.53 | 2.8 | 12M | 5.86 | 1066 | 80 | $551 |
| E5620 | 4 | 8 | 2.40 | 2.66 | 12M | 5.86 | 1066 | 80 | $387 |
| X5677 | 4 | 8 | 3.46 | 3.73 | 12M | 6.4 | 1333 | 130 | $1,693 |
| X5667 | 4 | 8 | 3.06 | 3.46 | 12M | 6.4 | 1333 | 95 | $1,440 |
* Intel 1k pricing
Xeon 7500 SKUs| Model | Cores | Threads | GHz | Turbo | L3 | QPI GT/s | Memory | Power | Price* |
|---|---|---|---|---|---|---|---|---|---|
| X7560 | 8 | 16 | 2.26 | 2.66 | 24M | 6.4 | 1066? | 130 | $3,692 |
| X7550 | 8 | 16 | 2.00 | 2.4 | 18M | 6.4 | ? | 130 | $2,729 |
| E7540 | 6 | 12 | 2.00 | 2.26 | 18M | 6.4 | ? | 105 | $1,980 |
| E7530 | 6 | 12 | 1.86 | 2.13 | 18M | 5.86 | ? | 105 | $1,391 |
| E7520 | 4 | 8 | 1.86 | 1.86 | 18M | 4.8 | ? | 95 | $856 |
| X7542 | 6 | 6 | 2.66 | 2.8 | 18M | 5.86 | ? | 130 | $1,980 |
| Model | Cores | Threads | GHz | Turbo | L3 | QPI GT/s | Memory | Power | Price* |
|---|---|---|---|---|---|---|---|---|---|
| X6550 | 8 | 16 | 2.00 | 2.4 | 18M | 6.4 | ? | 130 | $2,461 |
| E6540 | 6 | 12 | 2.00 | 2.26 | 18M | 6.4 | ? | 105 | $1,712 |
| E6510 | 4 | 8 | 1.73 | 1.73 | 12M | 4.8 | ? | 105 | $744 |
Before commenting, recall the main differences between the Xeon 5600 and Xeon 7500/6500 series. The Xeon 5600 series (32nm process) has 2 QPI links and 3 memory channels. The Xeon 7500 series (45nm process) has 4 QPI links, 4 memory channel, larger cache per core (for the 24M version, 3M vs 2M) plus extensive reliability features. The 2 QPI links on the 5600 series allows a 2-way (socket) system. The 4 QPI links on the 7500 series allows glueless 4-way and 8-way. My understanding is the 6500 series is the 7500 with only 2 QPI links enable for 2-way systems with 16-cores and 8 memory channels total, at lower frequency than the 5600 with 12-cores and 6 memory channels total, plus the 7500 RAS features.
Now lets looks at system pricing for the 2-way Dell PowerEdge T710 (Xeon 5600), R810 (either 7500 or 6500) and the 4-way R910 (7500). All systems with redundant power supplies, 2x73GB 15K 2.5in drives, 6Gb/s SAS. 4 power supplies in the 4-way
Dell PowerEdge T710 Systems with 2 Xeon 5600 processors (Repriced 2010-08)| System | Processor | GHz | Cores | L3 | QPI | - | Memory | Price |
|---|---|---|---|---|---|---|---|---|
| T710 | X5680 | 3.33 | 6 | 12M | 6.4 | 1333 | 48GB 6x8G | $8,995 |
| T710 | X5670 | 2.93 | 6 | 12M | 6.4 | 1333 | 48GB 6x8G | $8,415 |
| T710 | X5660 | 2.80 | 6 | 12M | 6.4 | 1333 | 48GB 6x8G | $7,855 |
| T710 | X5650 | 2.66 | 6 | 12M | 6.4 | 1333 | 48GB 6x8G | $7,395 |
| T710 | E5640 | 2.66 | 4 | 12M | 5.86 | 1066 | 48GB 6x8G | $6,735 |
| T710 | E5630 | 2.53 | 4 | 12M | 5.86 | 1066 | 48GB 6x8G | $6,215 |
| T710 | E5620 | 2.53 | 4 | 12M | 5.86 | 1066 | 48GB 6x8G | $5,835 |
Memory: 6x8GB set for $2,382
Dell PowerEdge R910 Systems with 4 Xeon 7500 processors (Repriced 2010-08)| System | Processor | GHz | Cores | L3 | QPI | - | Memory | Price |
|---|---|---|---|---|---|---|---|---|
| R910 | X7560 | 2.26 | 8 | 24M | 6.4 | 1066 | 128GB 16x8G | $30,318 |
| R910 | X7550 | 2.00 | 8 | 18M | 6.4 | 1066 | 128GB 16x8G | $26,118 |
| R910 | E7540 | 2.00 | 6 | 18M | 6.4 | 1066 | 128GB 16x8G | $21,718 |
| R910 | E7530 | 1.86 | 6 | 12M | 5.86 | 980 | 128GB 16x8G | $19,518 |
| R910 | E7520 | 1.86 | 4 | 18M | 4.80 | 800 | 128GB 16x8G | $17,218 |
Memory: 16x8GB set for $6,122
Dell PowerEdge T710 Systems with 2 Xeon 5600 processors (2010-06?)| System | Processor | GHz | Cores | L3 | QPI | - | Memory | Price |
|---|---|---|---|---|---|---|---|---|
| T710 | X5680 | 3.33 | 6 | 12M | 6.4 | 1333 | 72GB 18x4G | $9,974 |
| T710 | X5660 | 2.80 | 6 | 12M | 6.4 | 1333 | 72GB 18x4G | $8,634 |
| T710 | X5650 | 2.66 | 6 | 12M | 6.4 | 1333 | 72GB 18x4G | $8,154 |
| T710 | E5640 | 2.66 | 4 | 12M | 5.86 | 1066 | 72GB 18x4G | $7,474 |
| T710 | E5630 | 2.53 | 4 | 12M | 5.86 | 1066 | 72GB 18x4G | $6,934 |
For some reason, Dell does not offer the T710 with the second from top X5670 2.93GHz.
Dell PowerEdge R810 Systems with 2 Xeon 7500 or 6500 processors| System | Processor | GHz | Cores | L3 | QPI | - | Memory | Price |
|---|---|---|---|---|---|---|---|---|
| R810 | X7560 | 2.26 | 8 | 24M | 6.4 | 1066 | 64GB 16x4G | $17,866 |
| R810 | X7542 | 2.66 | 6 | 12M | 5.86 | ? | 64GB 16x4G | $13,366 |
| R810 | X6550 | 2.00 | 8 | 18M | 6.4 | 1066 | 64GB 16x4G | $13,066 |
| R810 | E7540 | 2.00 | 6 | 18M | 6.4 | 1066 | 64GB 16x4G | $12,166 |
| R810 | E6540 | 2.00 | 6 | 18M | 6.4 | 1066 | 64GB 16x4G | $11,496 |
| System | Processor | GHz | Cores | L3 | QPI | - | Memory | Price |
|---|---|---|---|---|---|---|---|---|
| R910 | X7560 | 2.26 | 8 | 24M | 6.4 | 1066 | 64GB 16x4G | $19,246 |
| R910 | X7550 | 2.00 | 8 | 18M | 6.4 | 1066 | 64GB 16x4G | $16,446 |
| R910 | E7540 | 2.00 | 6 | 18M | 6.4 | 1066 | 64GB 16x4G | $13,546 |
| R910 | E7530 | 1.86 | 6 | 18M | 5.86 | 980 | 64GB 16x4G | $12,446 |
| System | Processor | GHz | Cores | L3 | QPI | - | Memory | Price |
|---|---|---|---|---|---|---|---|---|
| R910 | X7560 | 2.26 | 8 | 24M | 6.4 | 1066 | 128GB 32x4G | $34,040 |
| R910 | X7550 | 2.00 | 8 | 18M | 6.4 | 1066 | 128GB 32x4G | $28,440 |
| R910 | E7540 | 2.00 | 6 | 18M | 6.4 | 1066 | 128GB 32x4G | $22,640 |
| R910 | E7530 | 1.86 | 6 | 18M | 5.86 | 980 | 128GB 32x4G | $20,440 |
Previously, I had argued that processors and systems today were so powerful that the standard practice of buying 4-way systems for critical database server by default be changed to 2-way. What I mean by default is in lieu of proper system sizing analysis.
It may seem strange that I suggest not doing a proper sizing analysis (one of my services as a consultant). But from the sizing analysis I have seen done by other people, the quality of the work was poor and the effort cost more than a pair 4-way systems.
What this means is that the practical solution used to be to buy a 4-way system. Try it out. If it not sufficient, then hire someone (there are many people who can do this) to make it work on a 4-way. If that does not work, consider pruning features until it does work.
So why not just move up to an 8-way or larger system? Because 8-way and larger are mostly NUMA systems. Technically, all Opteron 2-way and up are NUMA. But by NUMA, I really mean systems where there is a large discrepancy between local and remote node memory access. There are very very few people who can do performance analysis on a NUMA system (not those who claim to be able to). Do a search on SQL NUMA to see who has published meaningful material on this matter.
Anyways, the default choice today should be a 2-way system. However, since this is critical system, perhaps there are features from the high-end that we want. I believe this is the rational for the Xeon 6500 from Intel, and the PowerEdge R810 from Dell.
In looking over the T710, R810 and R910, I am inclined to say the effort was not entirely successful, as with many first iterations. The effort definitely deserves merit, and is the proper direction for the future. But it just needs further refinement. Of course, the true measure whether people actually buy the R810 in volume, not just one persons opinion.
The R810 with either X7560 or X6550 just gives up too much frequency for the extra 2 cores per socket, and fourth memory channel. Some environments might want the X7500/6500 RAS features despite this. And there is only a $1400 price difference between the R810 and R910 with 2 sockets populated.
The amount of $1,400 is very small for having two extra sockets available, even though most people never populate sockets after system purchase. It would be nice if could buy the R910 with 4-sockets populated, but not have to pay the per-socket software licensing until they are turned-on, like in RISC world.
True, the R810 is a 2U form factor compared with 4U for the R910, allowing much higher density. But the assumption was this is a critical database server, for which an extra 2U is not a show stopper. (There are people who get hung up on the latest industry jargon/fads, and forget the job one is making sure your business in running.)
AMD Opteron 6176 (Magny-Cours) 2-way 12-core results have been just published, with the HP ProLiant DL385G7. I will add more detail later. The 2-way TPC-E result is 887.38 and the TPC-C result is 705,652. Interestingly, both the HP ProLiant DL370G6 with the Xeon W5580 and the DL385G7 Opteron TPC-C results are on SQL Server 2005. Perhaps the Microsoft mandate to use TPC-E is for SQL Server 2008, hence the C on 2005 was allowed? Also of interest is that the Opteron 6176 TPC-C result uses 125 SSDs instead of hard disks (1300 HDs in the Xeon W5580 result).
Before comparing the Opteron 12-core with Xeon 5500, let us first compare against the previous generation Xeon 5400 quad-core. The 2-way 12-core Opteron 6176 achieved OLTP results higher than the Xeon 5460 by 2.5X on TPC-C and 2.8X on TPC-E. These are very good results for a 3X increase in the number of cores. Now in comparing against the quad-core Xeon 5500 series, the 12-core Opteron is just marginally higher. I am inclined to think much of this is due to the Hyper-Threading capability in the Xeon 5500 series. HT was much maligned in the NetBurst architecture generation. Some people today still blindly regurgitate the advice to disable HT, not realizing this advice was applicable to the old NetBurst and not the new Nehalem architecture processors. At some point AMD may have to admit that implementing HT will be a necessity.
The price for the DL385G7 with 2x6176 processors from the TPC-H report is $1,511 for the system chassis, $1,799 for each processor, $990 for each 8GB kit, and perhaps another $1K for comparable configuration as above. This is very reasonable, except for the memory which seems high. Each 8GB kit should be around $500.
Dell PowerEdge R715 Systems with 2 Opteron 6100 processors| System | Processor | GHz | Cores | L3 | - | - | Memory | Price |
|---|---|---|---|---|---|---|---|---|
| R715 | 6176 | 2.3 | 12 | 2x6M | - | 1333 | 64GB 8x8G | $8,475 |
| R715 | 6174 | 2.2 | 12 | 2x6M | - | 1333 | 64GB 8x8G | $8,075 |
| R715 | 6172 | 2.1 | 12 | 2x6M | - | 1333 | 64GB 8x8G | $7,775 |
| R715 | 6168 | 1.9 | 12 | 2x6M | - | 1333 | 64GB 8x8G | $7,135 |
| R715 | 6136 | 2.4 | 8? | 2x6M | - | 1333 | 64GB 8x8G | $7,135 |
| System | Processor | GHz | Cores | L3 | - | - | Memory | Price |
|---|---|---|---|---|---|---|---|---|
| R815 | 6174 | 2.2 | 12 | 2x6M | - | 1333 | 128GB 16x8G | $14,120 |
| R815 | 6172 | 2.1 | 12 | 2x6M | - | 1333 | 128GB 16x8G | $13,520 |
| R815 | 6168 | 1.9 | 12 | 2x6M | - | 1333 | 128GB 16x8G | $12,240 |
The T815 is a 2U system and does not support the top frequency 6176 with all four sockets populated.