PCI-E, SAS, FC,
RAID Controllers, Direct-Attach,
SAN, Dell MD3200, CLARiiON AX4, CX4, VNX, V-Max, HP P2000, EVA, P9000/VSP, Hitachi AMS
SSD products: SATA/SAS SSDs, PCI-E SSDs, Fusion iO, other SSD
Tom's hardware review Four SAS 6Gb/s RAID Controllers, Benchmarked and Reviewed discusses among others the LSI MegaRAID 9265-8i. The 9265 has the dual-core LSI SAS2208 ROC, and FastPath (feature) for accelerate I/O operations of SSDs, 465,000 IOPS is mentioned. The LSI web site states that the MegaRAID SAS 9285-8e and the 9265-8i are 2nd generation 6Gb/s SAS PCIe 2.0 controllers (versus 1st generation 6Gb/s PCIe 2 for the 9280 and 9260 controllers using the SAS2108 ROC). FastPath is optional(?) software ($165?) with either hardware or electronic key?
The 9285 mentions LSI SAS2208 Dual-core ROC 800MHz PowerPC, 1GB DDR 3 (1333MHz) SSD Guard
The LSI FastPath document cites: "With MegaRAID FastPath software enabled, SSD configurations tuned for small, random block-size IO activity — typical of transactional database applications — can sustain over 150,000 IO reads per second in RAID 0 configurations. This is two times the transactional performance of identical configurations when the MegaRAID FastPath software is disabled. This is particularly evident in 4K random reads and random writes; as well as 4K and 8K OLTP transaction-oriented benchmarks." The figure below is from the LSI FastPath document is based on the 9260-8i with 8 Intel X25E SSD drives.
SSD Guard is described as:
"SSDs are known for their reliability and performance. The LSI SSD Guard technology, that is unique to MegaRAID controllers, increases the reliability of SSDs by automatically copying data from a drive with potential to fail to a designated hot spare or newly inserted drive. A predictive failure event notification, or S.M.A.R.T command, automatically initiates this rebuild to preserve the data on an SSD whose health or performance falls below par. If a hot spare is not present or not assigned, MegaRAID Storage Manager (MSM) will recommend that the user insert a hot spare drive into an available slot.
Because SSDs are very reliable, non-redundant RAID 0 configurations are much more common than in the past. SSD Guard technology offers added data protection for RAID 0 configurations by actively monitoring the status of the SSDs. SSD Guard, together with MegaRAID FastPath software, allows users to take full advantage of the reliability and performance attributes of SSDs."
Reference: SCSI Trade Association
SAS started out with 3.0Gbit/sec signaling. Unlike SATA, SAS appears to be used only with a x4 wide connection. Most SAS adapters have 2 x4 ports. The HP Smart Array P800 has 4 x4 ports. The nominal bandwidth of a x4 3Gb/s SAS connection is 12 Gbit/sec. The realizable bandwidth appears to be 1.0-1.1GB/sec.
The LSI SAS Switch White Paper 092210 provides the following explanation of SAS bandwidth:
6Gb/s x 4 lanes = 24Gb/s per x4 port
24Gb/s (8b/10b encoding) = 2.4GB/sec per port
2.4GB/s per port x 88.33% (arbitration delays and additional framing) = 2160MB/sec per port
Note that the above is in decimal bytes/sec, not binary 1KB = 1024B, 1M = 1024K, etc.
Unfortunately, this is not matched with the bandwidth of a PCI-E gen 1 slot. To realize more than 800MB/sec from a single x4 SAS channel requires a x8 PCI-E gen 1 slot, which in turn, results in under-utilizing the PCI-E slot or not achieving balance between the 2 x4 SAS ports. Since most adapters have 2 x4 ports, the maximum realizable bandwidth in a x8 PCI-E gen 1 slot is 1.6GB/sec. Some of the early PCI-E SAS adapters have an internal PCI-X bus that limits realizable bandwidth over both x4 SAS ports to 1GB/sec.
Server systems usually have some combination of x16, x8 and x4 slots. No server adapter relevent to databases can use more bandwidth than that provided by a x8 slot, so each x16 slot could have been 2 x8 slots, for a waste of an otherwise perfectly good x8 slot. The x4 slots are usually a good match for network adapters. A PCI-E gen 2 x4 slot is exactly matched to 2 x 10GbE ports.
Matching the available x16 and x8 slots to storage controllers is not always possible. Sometimes it may be necessary to place one or more SAS storage controllers in the x4 slots, in which case it is important to distribute the number disks behind controllers in x8 and x4 slots proportionately as appropriate.
In the last year, 6.0Gb/s SAS adapters and disk drives became available. The same bandwidth mismatch situation between 3Gb/s SAS and 2.5 Gb/s PCI-E gen 1 also occurs with 6Gb/s SAS and 5Gb/s PCI-E gen 2. In addition, LSI Logic states that their 6Gb/s SAS controller has a maximum combined bandwidth of 2.8GB/sec over both x4 SAS ports.
The table below are estimates from the LSI slidedeck MegaRAID Performance Tuning and Benchmark Tips
|Port width||SAS 3Gbps||SAS 6Gbps|| |
|x1 SATA||245MB/s|| || |
|x4 SATA||908MB/s|| || |
|x1 SAS||260MB/s||520MB/s|| |
|x4 SAS||1040MB/s||2080MB/s|| |
The LSI reference did note that the estimates change with disk models, technology and drive revisions. There more recent SATA SSDs can deliver 265-275MB/sec on 3Gbps. This diagram seems to indicate that the realizable bandwidth on 8 3Gbps SAS lanes is 2.4GB/s, corresponding to 20% overhead. The upper bound is 4.8GB/sec over 8 6Gbps SAS lanes.
However the diagram also cites the nominal bandwidth of PCI-E, so presumably the 2.4 and 4.8GB/sec SAS bandwidths are also nominal, with realizable bandwidth around 80% of nominal.
The chart below shows a test by LSI. The 1000MB/sec upper bound on 1 x4 3Gbps SAS port appears to be the SAS limit. The 1500MB/sec upper bound on 2 x4 3Gbps SAS ports may be the controller limit, with 1600MB/sec being the PCI-E gen 1 x8 limit.
PCI-E gen1 and 3Gbps SAS
PCI-E gen2 and 6Gbps SAS
For direct-attach storage, the SAS adapter is also a RAID controller. Most SAS RAID controllers are built around LSI silicon controllers, notably the LSI SAS 1078 for 3Gb/s SAS and the new SAS 2008(2108) for 6Gb/s SAS and 5Gb/s PCI-E gen 2. Intel used to make a PCI-E to SAS RAID controller built around the 80333 IO Processor, but mysteriously dropped out of the market soon after releasing the new 81348 IOP in 2007. There might be another vendor as I am not sure who makes the controller for the HP P800.
The LSI SAS 2008
The LSI SAS 2x36
LSI has a 4 x4 PCI-E gen 2 6Gb/s SAS RAID Controller, listing a LSI SAS 2116. It is unclear if this is a variation of the 2008 or just two die in board.
The LSI slidedeck MegaRAID Performance Tuning and Benchmark Tips states that Direct IO is the correct setting on RAID controllers. "All read data is transferred directly to host memory, bypassing RAID controller cache. Any Read Ahead data is cached. All write data is transferred directly from host mempory bypassing RAID controller cache if Write-Through cache is set." The Cached IO " (All read and write data passes through controller cache memory on its ways to or from host memory, includes write data in Write-Through mode), legacy option is being eliminated in a future release.