Home, Cost-Based Optimizer, Benchmarks, Server Systems, Systems Architecture, Processors, Storage,
  Storage Overview, System View of Storage, SQL Server View of Storage, File Layout,

PCI-ESASFCHDDSSD Technology RAID ControllersDirect-Attach
  SAN,  Dell MD3200,  EMC AX4CX4VNXV-Max,  HP P2000EVAP9000/VSP,  Hitachi AMS 
  SSD products: SATA/SAS SSDsPCI-E SSDsFusion iOother SSD 

PCI-E SSD Products Today (Fusion iO covered separately)

Below is a quick survey of SSD either currently available or expected in the near-term.

Intel DC P3(5,6,7)00 P320h PCI-E gen 3 SSDsUpdate 2014-06

Tom's itpro 2014 Jun 02 has this article Intel Intros New Data Center PCIe SSD Family and AnandTech 2014 Jun 03 Intel SSD DC P3700 Review: The PCIe SSD Transition Begins with NVMe. The DC P3500, 3600 and 3700 PCI-E SSDs are gen 3 with x4 width, and are based on the new NVMe standard.

Model DC P3500DC P3600DC P3700
Interface PCI-E g3 x4PCI-E g3 x4PCI-E g3 x4
Form Factors HHHL PCI-E &
2.5in SFF-8639
HHHL PCI-E &
2.5in SFF-8639
HHHL PCI-E &
2.5in SFF-8639
Capacities 400GB, 1.2TB,
& 2.0TB
400GB, 800GB,
& 1.2TB
400GB, 800GB,
1.6TB & 2.0TB
Raw Capacity???
Write Enduranceup to 1 PBW
0.3 DWPD
up to 10.95 PBW
3 DWPD
up to 36.5 PBW
10 DWPD
64K Read/Write (MB/s) up to 2500/1700 up to 2600/1700up to 2800/1900
4K Read/Writeup to 450K/35Kup to 450K/70K up to 460K/160K
4K 70/30 Mixed R/Wup to 85Kup to 170Kup to 250K
Pricing $1.5/GB
$599+
$2/GB
$783+
$3/GB
$1207+

 

Intel DC P3700 400GB800GB1.6TB2.0TB
Sequential Read 2,700 2,800 2,800 2,800
Sequential Write 1,200 1,900 1,900 1,900
Random Read 4KB 450,000450,000450,000450,000
Random Write 4KB 75,000 90,000 150,000175,000
4KB Read/Write 70/30150,000200,000240,000250,000
Random Read 8KB 260,000280,000280,000280,000
Random Write 8KB 32,000 45,000 75,000 90,000

All performance specifications are "up to"

Intel DC P3600 400GB800GB1.2TB1.6TB2.0TB
Sequential Read 2,100 2,600 2,600 2,600 2,600
Sequential Write 550 1,000 1,250 1,600 1,600
Random Read 4KB 320,000430,000450,000450,000450,000
Random Write 4KB 30,000 50,000 50,000 56,000 56,000
4KB Read/Write 70/3080,000 110,000130,000160,000160,000
Random Read 8KB 180,000240,000260,000260,000260,000
Random Write 8KB 19,000 26,000 27,000 33,000 35,000

All performance specifications are "up to"

The AnandTech article cites 20µs average read/write latency. The P3700 uses HET MLC NAND. Presumably the P3600 and P3500 use ordinary MLC, with the difference in write endurance due to the degree of over-provisioning? The P3500, P3600 and P3700 over-provisioning is cited as 25%, 12 % and 7% respectively. AnandTech cites pricing as $3/GB, $1.95/GB and 1.495/GB respectively. Since we are not used car salesmen, I will cite this a $3, 2 and 1.5/GB. DRAM capacities range from 512M to 2.5GB.

As bandwidth tops out at 2.8GB/s, I am thinking that there are 8 or 9 NAND channels at 333MB/s on the back-end?

NVM Express (NVMe)

"NVM Express (NVMe) is register level interface that allows host software to communicate with a non-volatile memory subsystem. This Interface ... typically attached to the PCI Express interface."

Some objectives of NVMe are 1) allow OS boot from a PCI-E device (as opposed to SATA) and 2) completely re-architect the driver for operation at extremely high IOPS, far beyond anything considered for HDD storage arrays with emphasis on code-path efficiency and parallelism. Other capabilities are commands to match the nature of NAND, as previous interfaces were designed around hard disk storage.

NVMe and the other new storage initiatives do not take a hard position on the NAND controller interface, with plans to accommodate co-existence of both SATA/SAS and direct PCIE.

 

Micron P320h Update 2012-10

See the AnandTech Micron P320h PCIe SSD (700GB) Review review for details. Also see the ssd review micron-p320h-hhhl-700gb-pcie-enterprise-ssd-review. The Micron P320h is SLC. The pricing from CDW is about $15K per TB. This is definitely an interesting product.

  P320hP320h
Capacities (GB) 350 700
Raw Capacity (GB)512 1024
Write Endurance (PB)25 50
Max Read 3,200MB/s3,200MB/s
Max Write 1,900MB/s1,900MB/s
Random Read 4K 785K IOPS785K IOPS
Random Write 4K 205K IOPS205K IOPS
Pricing $5,578 $11,155

It is too bad Micron does not have an MLC product. For a transaction processing database, we might isolate a hot table to fit well with SLC pricing. However with MLC prices approaching $5K per TB for enterprise (or just better than consumer) grade products, it is feasible to put the entire database (except for archival tables) on SSD. In this case, the SLC write endurance is not necessary.

Below is the Micron controller, interfacing PCI-E directly to the NAND interface. There are 32 channels on the NAND side. Of course, a SATA SSD can have 8 NAND channels. So 4 SATA SSDs would have 32 NAND channels. The difference is that ONFI 2.0 allows up to 133MB/s and 2.1 allows 166 or 200MB/s per NAND channel, so 8 channels is more than the bandwidth of a single 6Gbps SAS lane (600MB/s net). The real benefit of interfacing directly from PCI-E to NAND is that this avoids the extra bandwidth matching step.

Micron p320h

Using the figure below and the 350GB net capacity of the smaller model, the 0.875 factor for RAIN (RAI on NAND instead of disks) implies 400GB capacity prior to RAIN. The 0.78 factor for over-provisioning implies a raw capacity of 512GB. This would imply that the 350GB net capacity is binary, not decimal?

Micron p320h

Micron also has a PCI-E SDD in 2.5in HDD form factor at 175 and 350GB.

Capacities (GB) 175 350
Write Endurance (PB)25 50
Max Read 1,750MB/s1,750MB/s
Max Write 1,100MB/s1,100MB/s
Random Read 4K 415K IOPS415K IOPS
Random Write 4K 145K IOPS145K IOPS
Pricing ? ?

Intel 910 Series

The Intel 910 Series finally came out earlier this year. The 910 uses an LSI PCI-E to SAS controller followed by SAS to NAND controllers.

 910910
Capacities (GB)400800
Raw Capacity (GB)768?1536?
Write Endurance (PB)5-710-14
Controllers??
Max Read 1,000MB/s2,000MB/s
Max Write750MB/s1,000MB/s
Random Read 4K90K IOPS180K IOPS
Random Write 4K38K IOPS75K IOPS
Pricing$2,000$4,000

Some technology enthusiasts like to talk about the theoretical "advantages" of not being encumbered by the extra interface transition through SAS. I would point out that the LSI SAS controller is a very mature server product. Too many product vendors cite impressive performance numbers for their PCI-E SSD on the basis of a single adapter in the system.

In the server environment, performance from a single card is only one of several metrics. It is also critically important that the device have very low overhead. The objective is for the application (the SQL Server database engine in our case) to run with as little disruption as possible. The complete server system will have several PCI-E SSD devices. If the IO performance does not scale over multiple devices, then it is of limited value in servers. LSI has deep experience on server side as LSI SAS controllers are used in almost server systems. They are using the controller of choice for TPC benchmarks with proven ability to scale.

The NAND is 25nm with High Endurance Technology (HET). Write endurance is 2.5PB per 200GB module at 4KB and 3.5PB at 8KB write IO. This is 30X over the consumer SSDs?

Read and write latency is quoted as < 65us. The 800GB model can support 1500MB/s large block write if higher power is available. The architecture of the 800GB is shown below (from Intel Solid-State Drive 910 Series Product Specification).

Intel 910

The PCI-E bridge chip has an SAS interface on the backend. There are 4 SAS channels to a SAS/NAND ASIC. Each NAND module is 200GB. So the 400GB module has 2 SAS channels? It would seem that the Intel 910 uses half of the x4 SAS at 400GB and a full x4 SAS at 800GB?

Tom's hardware Intel SSD 910 Review: PCI Express-Based Enterprise Storage and Intel SSD 710 Tested: MLC NAND Flash Hits The Enterprise

LSI Nytro WarpDrive

There are several sub-brands of the LSI Nytro WarpDrive.
The MegaRAID Application Acceleration Cards has NAND flash plus software to cache hard drives on the x4 SAS port. The Application Acceleration Card is the straight PCI-E SSD. There are 200 and 400GB SLC and 400, 800 and 1600GB MLC versions. LSI also has Nytro XD Application Acceleration Storage Solution, a software product for caching SAN or direct-attach storage on SSD.

 SLCMLC
Capacities (GB)200/400400/800/1.6TB
Raw Capacity (GB)??
Write Endurance (PB)??
Controllers??
Max Read 2,000MB/s2,000MB/s
Max Write1,700MB/s1,000MB/s
Random Read 4K 238K IOPS218K IOPS
Random Write 4K133K IOPS75K IOPS
Random Read 8K 189K IOPS183K IOPS
Random Write 8K137K IOPS118K IOPS
Pricing$6K/12K?$6K/10K/$20K?

I think the LSI came out in 2011, but I was not paying attention. The SLP-300 is rated at 1,400MB/s read, 1,200MB/s write for 64K sequential. 4K random is 150K read and 190K write. Interface is x8 PCI-E gen 2. Latency is < 50usec.

OCZ Z-Drive R4 R and C

OCZ PCI-E SSDs moved OCZ PCI-E SSDs

Micron Updated 2011-05

Anandtech details the Micron P320h PCIe SLC SSD with 350 & 700GB capacities. Specifications are 3GB/s read, 2GB/s write, 750K read & 298K write IOPS (140K for 350GB and 200K for 700GB models?). The IOPS numbers vary by source, so it is probably a complicated matter that is not being documented fully. Endurance is 25 & 50 petabytes respectively.

A feature of the P320h is Redundant Array of Independent NAND (RAIN). I believe this to be the correct evolutionary shift from HDD RAID, as the SSD is not fundamentally a single device, but rather multiple devices. So the right place for redundancy is inside the SSD. Computerworld reports RAIN is implemented as 7 channels of data plus 1 channel for parity. Read and Write latency is 50 μsec.

x

x

The table below is from the Micron p320 datasheet. I am not sure how the numbers work out. I am thinking there are currently SLC NAND 8-Gbit chips in production. It is possible to put 4 or 8 chips in one package. Then 8 chips of 8Gbit each is 64Gb or 8GB (so presumably NAND density is at the package level?). It would seem reasonable that an enterprise class SSD be over-provisioned, plus ECC and RAIN, so 64 x 8GB (= 512GB) packages in a 320GB product. Then the 700GB part would have to be 128 x 8GB or 64 x 16GB?

CapacityNAND ProcessNAND DensityPackage CountDie per Package
350GB34nm64Gb644
700GB34nm64Gb648

The Micron website says the P320h is sampling as of June 2011 with production in Q3. Computerworld reports that Micron intends to price around $16 per GB or $16K per TB. There is no mention of an MLC product.

Intel SSD

Various websites, AnandTech etc, discuss specifications for the upcoming Intel 710 and 720 series SSD, but the Intel website does not mention these on their SSD products page yet. The 720 is PCIe 2.0 SLC product with 2.2GB/s read, 1.8GB/s write, 180K IOPS read, 56K IOPS write.

Original 2010-10

HP

The HP specification sheet for their PCIe IO Accelerators also provides information on the Fusion-IO driver memory usage.

The amount of free RAM required by the driver depends on the size of the blocks used when writing to the drive. The smaller the blocks, the more RAM required. Here are the guidelines for each 80GB of storage:

Average
Block Size (bytes)
RAM usage
(Megabytes)
8,192400
4,096800
2,0481,500
1,0242,900
5125,600

Give credit to the HP technical people really know what key information that should be documented is. This is too important to leave to the simpletons in !@#$%^&*. (IBM also puts out great Redbooks, give credit to IBM for spending money on important material, not just marketing rubbish).