Home, Optimizer, Benchmarks, Server Systems, Systems Architecture, Processors, Storage,

PCI-ESASFCHDDSSD Technology RAID ControllersDirect-Attach,
  SAN,  Dell MD3200,  CLARiiON AX4, CX4VNXV-Max,  HP P2000EVAP9000/VSP,  Hitachi AMS
  SSD products: SATA/SAS SSDsPCI-E SSDsFusion iOother SSD

PCI-E Nominal and Realizable Bandwidth (bi-directional)

PCI-E (or PCIe) bandwidth can be tricky to understand. It is perfectly legitamate to cite PCIe 1.0 as 2.5GT/s (giga-transfers/sec), 2.0 as 5GT/s and 3.0 as 8GT/s. PCIe 2.0 and 3.0 can also be referred to as gen 2 and gen 3. Notice the term GT/s as opposed to GHz. Engineers like to associated the term Hertz in GHz with the clock frequency. And it is popular to transfer data at a higher multiple of the clock. PCIe in fact has a reference clock at 100MHz. So we use the term giga-transfers per sec instead of GHz.

In high-speed data transmission, it is necessary to avoid having the transmitter and receiver drift out of synch due to runs of 0's or 1's. To do this without a separate clock signal, PCIe 1.0 and 2.0 encodes every 8 bits to 10b. The PCIe 3.0 encoding scheme is 128b/130b.

PCIe

Next, we cannot simply send raw data. There must be information on the send, the recipient, and additional information. The PCIe protocol has 3 layers: physical, data link and transaction. The physical layer has a Frame, the data link layer has a Packet Sequence Number at the front and a CRC at the end, and the transaction layer has a header.

PCIe
PCIe Protocol layers

PCIe
PCIe Packet

PCIe
Packet Header

PCIe

PCIe
PCIe Packet

By the time all is said and done, the table below shows the approximate net bandwidth for data in x4 and x8 slots.

PCIe GenRaw bit rateEncodingInterconnect bandwidthBW per lane per directionBW per direction x8Net Data BW x4Net Data BW x8
1.02.5GT/s8b/10b2Gbps250MB/s2GB/s0.8GB/s1.6GB/s
2.05.0GT/s8b/10b4Gbps500MB/s4GB/s1.6GB/s3.2GB/s
3.08.0GT/s128b/130b8Gbps1GB/s8GB/s3.2GB/s?6.4GB/s?

PCI-E 2.0 signals at 5.0Gbit/s, also with 8b/10b encoding for a bandwidth of 400MB/s per lane per direction. PCI-E 3.0 signals at 8.0Gbit/s, but with 128b/130b encoding for essentially double the 2.0 bandwidth.

Finally, note that there is difference between the maximum GT/s for each PCIe version and the PCIe version. A product can implement 2.5 or 5GT/s, but still be 3.0 compliant.

Systems of the Intel Core 2 processor architecture generation (Xeon 5100-5400 and Xeon 7300-7400 series) are almost exclusively PCI-E gen 1, as are the accompanying chipsets: the 5000P and 7300. The Intel 5400 MCH did support PCI-E gen 2, but no tier-1 system vendor produced a server with this chipset. (Supermicro, popular with white-box builders, did have 5400-based motherboards.) Systems of the Intel Nehalem generation and later have PCI-E gen 2. If someone could advise on when AMD Opteron transitioned from PCI-E gen 1 to gen 2, I would appreciate it.

The diagram below from the Intel IOH specification shows Intel 5520 IOH with 36 PCI-E gen 2 lanes, which are organized in 2 sets of 4 x4 and one set of 2 x2. Several combinations of x16, x8, x4 and even x2 slots can be configured. A server system should employ a proper combination of x8 and x4 slots. The x16 configuration is mostly for workstation graphics or special coprocessor, as there are no server disk or network adapters that can make effective use of x16 slots. Of course, there is the possible exception of Fusion-IO octal cards.

Xeon5500_1IOH

The diagram below shows more detail for the Intel IOH.

Xeon7500_IOH