Home, Cost-Based Optimizer, Benchmarks, Server Systems, Systems Architecture, Processors, Storage,
  Storage Overview, System View of Storage, SQL Server View of Storage, File Layout,

PCI-ESASFCHDDSSD Technology RAID ControllersDirect-Attach
  SAN,  Dell MD3200,  EMC AX4CX4VNXV-Max,  HP P2000EVAP9000/VSP,  Hitachi AMS
  SSD products: SATA/SAS SSDsPCI-E SSDsFusion iOother SSD

Fusion-iO 2011 Oct

Fusion-iO just announced the new ioDrive2 and ioDrive2 Duo on Oct 2011. The MLC models will be available late November and the SLC models afterwards. See the Fusion-iO press release for more info.

Below are the Fusion-IO ioDrive2 and ioDrive2 Duo specifications. The general idea seems to be for the ioDrive2 to match the realizable bandwidth of a PCI-E gen2 x4 slot (1.6GB/s) and for the ioDrive2 Duo to match the bandwidth of a PCI-E gen2 x8 slot (3.2GB/s). I assume that there is a good explanation why most models have specifications slightly below the corresponding PCI-E limits.

The exception is that 365GB model at about 50% of the PCI-E g2 x4 limit. Suppose that the 785GB model implement parallelism with 16 channels and 4 die per channel. Rather than building the 365GB model with the same 16 channels, but a different NAND package with 2 die each, they just implemented 8 channels using the same 4 die per package. Lets see if Fusion explains this detail.

Fusion-IO ioDrive2

ioDrive2 Capacity 400GB600GB365GB785GB1.2TB
NAND TypeSLC (Single Level Cell)MLC (Multi Level Cell)
Read Bandwidth (1MB) 1.4 GB/s1.5 GB/s910 MB/s1.5 GB/s1.5 GB/s
Write Bandwidth (1MB)1.3 GB/s1.3 GB/s590 MB/s1.1 GB/s1.3 GB/s
Read IOPS (512 Byte) 190,000193,000137,000141,000143,000
Write IOPS (512 Byte) 535,000535,000535,000535,000535,000
Read Access Latency 47 Ás47 Ás68 Ás68 Ás68 Ás
Write Access Latency15 Ás15 Ás15 Ás15 Ás15 Ás
Bus InterfacePCI-E Gen 2 x4
Price$11K$16K$6K?$10K?$14K?

Fusion-IO ioDrive2 Duo

ioDrive2 Capacity1.2TB2.4TB
NAND TypeSLC (Single Level Cell)MLC (Multi Level Cell)
Read Bandwidth (1MB) 3.0 GB/s3.0 GB/s
Write Bandwidth (1MB)2.5 GB/s2.5 GB/s
Read IOPS (512 Byte) 285,000385,000
Write IOPS (512 Byte) 725,000725,000
Read Access Latency 47 Ás68 Ás
Write Access Latency15 Ás15 Ás
Bus InterfacePCI-E Gen 2 x8
Price$??$25K?

Dell lists the ioDrive2 Duo 2.4TB MLC card for $25K. The Octal 5.12TB is $94K?

SLC verus MLC NAND
Between the SLC and MLC models, the SLC models have much better 512-byte reads IOPS than the MLC models, with only moderately better bandwidth and read latency. Not mentioned, but common knowledge is that SLC NAND has much greater write-cycle endure than MLC NAND.

It is my opinion that most database, transaction processing and DW, can accommodate MLC NAND characteristics and limitations in return for the lower cost per TB. I would consider budgeting a replacement set of SSDs if analysis shows that the MLC life-cycle does not match the expected system life-cycle. Of course, I am also an advocate of replacing the main production database server on a 2-3 year cycle instead of the more common 5-year practice.

The difference in read IOPS at 512B is probably not important. If the ioDrive2 MLC models can drive 70K+ read IOPS at 8KB, then it does not matter what the 512B IOPS is.

Post-RAID?
One point from the press release: "new intelligent self-healing feature called Adaptive FlashBack provides complete chip level fault tolerance, which enables ioMemory to repair itself after a single chip or a multi chip failure without interrupting business continuity." For DW systems, I would like to completely do away with RAID when using SSDs, instead having two system without RAID on SSD units.

For a critical transaction processing system, it would be nice if Fusion could provide failure statistics for units that have been in production for more than 30 days (or whatever the infant mortality period is) on the assumption that most environments will spend a certain amount of time to spin up a new production system. If the failure rate for a system with 2-10 SSDs is less than 1 per year, then perhaps even a transaction processing system using mirroring for high-availability can also do without RAID on the SSD?

ioDrive2 and ioDrive2 Duo
I do think that it is great idea for Fusion to offer both the ioDrive2 and ioDrive2 Duo product lines matched to PCI-E gen2 x4 and x8 bandwidths respectively. The reason is that server systems typically have a mix of PCI-E x4 and x8 slots with no clear explanation of the reasoning for the exact mix, other than perhaps that being demanded by the customer complaining the loudest.

By have both the ioDrive2 and Duo, it is possible to fully utilize the bandwidth from all available slots balanced correctly. It would have been an even better idea if the Duo is actually a daughter card the plugs onto the ioDrive2 base unit, so the base model can be converted to a Duo, but Fusion apparently neglected to solicit my advice on this matter.

The original Fusion IO product line was the ioDrive with PCI-E gen 1 x4/8 interface. The next product line was the ioDrive Duo now supporting PCI-E gen 2 x4 or gen 1 x4 and x8.

When the first Fusion IO products first came out a few years ago, the ioDrivce performance was far above other SSD products of the time. Aside from massive parallelism and the PCI-E interface which other vendors have since adopted, the other significant feature of the Fusion ioDrive was use of memory to store metadata. Apparently, the latency of NAND is somewhere around 25 Ás for SLC and 50 Ás for MLC. On top this, NAND based SSD are not direct mapped (to ensure even distribution of writes). In a NAND-only SSD, there needs to be 1 or 2 metadata accesses to determine the actual block location.

Fusion-IO 2010 Oct

The general idea behind the Fusion-IO architecture is that the storage interfaces were not really intended for the capabilities of an SSD. The storage interface, like SAS, was designed for many drives to be connected to a single system IO port. Since Fusion-IO could build the SSD unit to match the IO capability of a PCI-E slot, it is natural to interface directly to PCI-E. All the major server system vendors (Dell, HP, IBM) OEM the Fusion-IO cards.

Below is the original Fusion-IO ioDrive card with a PCI-E gen 1 x4 upstream interface.

x

Below are the Fusion-IO ioDrive specifications. The prices are from the Dell website, (HP website prices in parenthesis).

ioDrive Capacity160GB320GB320GB640GB
NAND TypeSLC (Single Level Cell)MLC (Multi Level Cell)
Read Bandwidth (64kB) 770 MB/s770 MB/s735 MB/s750 MB/s
Write Bandwidth (64kB)750 MB/s790 MB/s510 MB/s550 MB/s
Read IOPS (512 Byte) 140,000140,000100,00093,000
Write IOPS (512 Byte) 135,000135,000141,000145,000
Mixed IOPS (75/25 r/w)123,000119,00067,00074,000
Access Latency (512 Byte)26 Ás26 Ás29 Ás30 Ás
Bus InterfacePCI-E x4
Price$8,173 ($7K)?$7,719 ($7.5K)$11,832 (?)

The specifications are somewhat improved over the specifications listed in 2009, listing 50-80μs per read. So presumably there were firmware enhancements, as the NAND chips should be the same. The latency specification is a little misleading, cited for 512 bytes. Other vendors cite 100μs for 4K? which incorporates 26 μs for SLC random access, plus the transfer time. I am assuming that Fusion-IO implements very wide parallelism, in both channels and die per channel.

Below is the Fusion-IO Duo, matched to PCI-E gen1 x4 or PCI-E gen2 x4 bandwidth, supporting both interfaces.

x

Below are the Fusion ioDrive Duo specifications.

ioDrive Duo Capacity320GB640GB640GB1.28TB
NAND TypeSLC (Single Level Cell)MLC (Multi Level Cell)
Read Bandwidth (64kB)1.5 GB/s1.5 GB/s1.5 GB/s1.5 GB/s
Write Bandwidth (64kB)1.5 GB/s1.5 GB/s1.0 GB/s1.1 GB/s
Read IOPS (512 Byte)261,000273,000196,000185,000
Write IOPS (512 Byte)262,000252,000285,000278,000
Mixed IOPS (75/25 r/w)238,000236,000138,000150,000
Access Latency (512 Byte)26 Ás26 Ás29 Ás30 Ás
Bus InterfacePCI-Express x4/x8 or PCI Express 2.0 x4
Price$17,487 ($14K)?$15,431 ($15K)?

The reliability specs are ECC for 11-bit per 240 bytes, uncorrected error 1 in 1020, undetected error 1 in 1030.

What I would like from Fusion-IO are a range of cards that can match the IO bandwidth of PCI-E gen 2 x4, x8 and x16 slots, delivering 2, 4 and 8GB/s respectively. Even better is the ability to simultaneously read 2GB/s and write 500MB/s or so from a x4 port, and so on for x8 and x16. I do not think it is really necessary for the write bandwidth to be more than 30-50% of the read bandwidth in proper database applications. One way to do this is to have a card with a x16 PCI-E interface, but the onboard SSD only connects to a x4 slice. The main card allows daughter cards each connecting to a x4 slice, or something to this effect.

Fusion-IO has an Octal card that is matched to the x16 PCI-E gen2? bandwidth. This card draws power from the dedicated power connector for high-end graphics. Apparently a x8 card draws more power than available from the PCI-E slot, and normal server system do not have the special graphics power connector.

One more thought on the Fusion-IO is using the PCI-E to PCI-E bridge chips. In my other blog on System Architecture, I mentioned that the 4-way systems such as the Dell PowerEdge R900 and HP ProLiant DL580G5 for Xeon 7400 series with the 7300MCH use bridge chips that let two PCI-E port share one upstream port. Could the Fusion-IO reside in an external enclosure? attached to the bridge chip. The other two ports connect to the host system(s). One the host would be a simple pass through adapter that sends the signals from the host PCI-E port to the bridge chip in the Fusion-IO external enclosure. This means the SSD is connected to two hosts. So now we can have a cluster? Sure it would probably involve a lot of software to make this work, who said life was easy.