NetApp – Data ONTAP storage architecture – part 1

In my last few posts I started to give you a brief introduction into NetApp clustered Data ONTAP (also called NetApp cDOT or NetApp c-mode). Now, it’s not that easy task, because I don’t know your background. I often assume that you have some general experience working with NetApp 7-mode (“older” mode or concept of managing NetApp storage arrays). But just in case if you don’t in this post I want to go through basic NetApp concepts of storage architecture.

From physical disk to serving data to customers

The architecture of Data ONTAP enables to create logical data volumes that are dynamically mapped to physical space. Let’s take a look at the very beginning.

Physical disks

We have our physical disks – which are packed into disk shelves. Once those disks shelves are connected to Storage Controller(s), the Storage Controller itself must own the disk. Why is that important? In most cases you don’t want to have a single NetApp array, you want to have a cluster of at least 2 nodes to increase the reliability of your storage solution. If you have two nodes (two storage controllers), you want to have an option to fail-over all operations to one controller, if the second one fails, right ? Right – to do so, all physical disks have to be visible by both nodes (both storage controllers). But – during normal operations (both controllers are working as HA – High Availability Pair), you don’t want one controller to “mess with” the others controller data, since those are working independently. That’s why, once you attach physical disks to your cluster and you want to use those disks, you have to decide which controller (node) owns each physical disk.

Disk types

NetApp can work with variety of different disk types. Typically you can devide those disks in terms of use:

  • Capacity – this category describes disks that are lower cost and typically bigger in terms of capacity. Those disks are good to store the data, however they are “slow” in terms of performance. Typically you use those disks if you want to store a copy of your data.  Disks types such as: BSAS, FSAS, mSATA
  • Performance – those are typically SAS or FC-AL disks. However nowadays SAS are the most popular. Those are a little bit more expensive but provides better performance results
  • Ultra-performance – those are solid-sate drives disks, SSD. They are the most expensive in terms of price per 1GB, hoewever they give the best performance results.

RAID groups

OK, we have our disks which are owned by our node (storage controller). It’s a good start, but that’s not everything we need to start serving data to our customers. Next step are RAID groups. Long story short RAID group consists of one or mode disks, and as a group it can either increase the performance (for example RAID-0), or increase the security (for example RAID-1).. Or do both of those (for example RAID-4 or RAID-DP). If you haven’t heard about RAID groups before, that might be a little bit confusing right now. For sake of this article think of RAID group as a bunch of disks that creates a structure on which you put data. This structure can survive a disk failure, and increase performance (comparing to working only on single disks). This definition briefly describes RAID-4. This RAID group is often used in NetApp configurations, however the most popular is RAID-DP. The biggest difference is that RAID-DP can survive two disks failure, and still serve data. How is that possible? Well, it’s possible because those groups are using 1 (for RAID-4) or 2 (for RAID-DP) disks as ‘parity disks’. It means those are not storing customer data itself, but they are used to store “control sum” of those data. In other words if you have 10 disks in RAID-4 configuration it means you have a capacity of 9 disks, since 1 disk is used for parity. If you have 10 disks in RAID-DP confiugration it means you have a capacity of 8 disks, since 2 are used for parity.


That’s the end of part 1. In part 2 I will go futher, explaining how to build aggregates, create volumes and serve files and LUNs to our customers.

Storage Performance – IO characteristics

What is IO

When it comes to performance issues the term you hear really often is IO. IO is a shortcut for input/output and it is basically communication between storage array and the host. Inputs are the data received by the array, and outputs are the data sent from it. To analyze and tune performance, you must understand the workload that an application and/or host is generating. Application workloads have IO characteristics. Those characteristics may be described in terms of:

  • IO size
  • IO access pattern
  • thread count
  • read/write ratio

In this post I would like to shortly go through those characteristics, because many people understands IO only as a “number of operations”,  without the awareness of what this number actually means and how to understand it. Very often when people are talking about IO, what they  might mean is IOPS – which is basically number of IOs per second. But to talk about IOPS and understand the number – you first have to understand and consider IO characteristics.

 IO size

IO request size has an affect on performance throughput – in general, the larger the IO size, the higher the storage bandwidth. What is very often overlooked is that in most cases tools are showing the average IO size. Bare in mind that most production workloads have a mix of IO sizes.

IO access pattern

In terms of access patterns we very often use terms:

  • random read/write – there is no sequential activity at all (near zero), the read and writes are purely random, almost impossible to boost performance with cache.
  • sequential read/write – the exact opposite – purely sequential access, with no randomness at all. In such environments using storaga cache can really boost the performance.

In the real word you will almost never find 100% random or 100% sequential data access.  Even in sequential environment there might be number of different sequential activities at the same time, which actually build randomness when switching between them.

IO access pattern may relate to IO size, with larger IO size (like 64kB etc) you most often deal with more sequential data access, whereas with small IO size (8kB for example) the access is most often random

Thread count

How many different activities are going on in the same time. If you have a single threaded IO access patterns, it means that host sends a write to a storage system, and waits for the acknowledge from the storage system that the write has completed. And once its completed it will send next write and so on. In real word most applications will produce multiple threads at the same time. They don’t have to wait for single response to send another request. If we go deeper here – one disk can do only one operation at a time. If multiple IO are targeted to the same disk we have a queue. Using queues storage system can optimize the way it works. The worst type of performance is when the tread count is 1 – how can you optimize single thread?

Read/write ratio

Writes are most expensive (performance wise) then reads. Mostly because the system has to determine where to put new chunk of data. Once it decides where to place the data the write itself is time consuming due to RAID write penalty (Of course it depends on the RAID level placed underneath).  With enterprise storage system it actually changes – very often writes are faster, because they are placed into cache, and then the acknowledge to the host is send (later on writes from cache to actual hard drives are send in optimized way by storage system). Whereas reads – especially random reads, are very often un-cached and have to be fetched from hard drives.

Read/Write ratio is really important in terms of replication. Obviously only writes are being replicated, reads are not.


Workload IO and (some of the most popular) RAID Types

RAID 1/0

  • Best for small random write-intensive workloads.  The RAID penalty with RAID 1/0 is only 2. That’s the lowest of all the RAID types (that actually give you some kind of failure protection)


  • Good mix of performance and protection. But the RAID penalty is 4, so  the performance is much worst with small random writes.
  • Best with client write IO size of 64 KB or lager – full stripe writes can really boost the performance, you can write the entire stripe without a need of reading the parity information – which of course lower the RAID penalty.
  • Best practice is to use RAID 5 for workloads that have random writes of 30% or less.


  • More protection due to two parity disks, but at the same time higher RAID penalty – which is 6.
  • Very often used with NL-SAS drives (or SATA drives) which are the cheapest in terms of TB/$, but are the slowest.

EMC VNX – RAID groups vs Storage Pools

With EMC training I have learnt some general guidelines, that should be followed to optimize and enable good performance on a VNX Unified storage system.

Drive types

When you chose a specific drive type, you should base your choice upon the expected workload.

  • FLASH – Extreme Performance Tier – great for extreme performance.
  • SAS – Performance Tier – perfect for general performance. Usually 10k RPM or 15k RPM.
  • NL-SAS – Capacity Tier – most cost effective, for streaming, agining data and archives and backups

RAID groups

When we want to provision storage on our hard drives, first we have to create either a RAID group, or a Storage Pool. Let’s first review RAID groups.

RAID Levels

Within EMC VNX RAID groups should have a maximum count of 16 drives (which is strange, since this rule does not apply to NetApp at all!).
Choosing the correct drive type is only one step, you also have to choose a correct RAID group.

  • RAID 1/0 – this RAID group is appropriate for heavy transactional workloads with a high rate of random writes (let’s say greater than 25-30%)
  • RAID 5 – I would say this one is most common one. Works best for medium to high performance, general-purpose and sequential workloads. And it’s much more space effective, since blocks are not mirrored, and parity takes only one extra disk (distributed on all disks thru the RAID group)
  • RAID 6 – Most often used for NL-SAS. Works best with read-based workloads such as archiving and backup to disk. It provides additional RAID protection (two disk may fail in one RAID group, instead of 1 with RAID5).

As I mentioned before, RAID groups can have a maximum count of 16 drives (not with NetApp). For parity RAID, the higher the count, the higher the capacity utilization (since there is only one parity disk in RAID5, and 2 parity disks in RAID6 regardless the RAID group size).

Preffered RAID configuration

With a predominance of large-block sequential operations the following rules are most effective:

  • RAID5 has a preference of 4+1 or 8+1
  • RAID6 has a preference of 8+2 or 14+2
  • RAID1/0 has a preference of 4+4

One more tip – when creating RAID groups, select drives from the same bus if possible to boost performance (always prefer horizontal over vertical selection).

Storage Pools

A storage pool is somehow analogous to a RAID group. In few words it’s a physical collection of disks on which logical units (LUNs) are created. Pools are dedicated for use by pool (thin or thick) LUNs. Where RAID group can only contain up to 16 disks, pool can contain hundreds of disks. Because of that, pool-based provisioning spreads workloads over many resources requiring minimal planning and management effort.

Of course pool has the same level of protection against disk failure as RAID groups, simply because RAID groups are used to build up pool. In fact, during pool creation, you have to choose which RAID configuration you preffer.

Pools can be homogeneous or heterogeneous. Homogeneous pools have a single drive type (e.g. SAS or NL-SAS) whereas heterogenous pools contain different drive types. Why EMC allow us to mix different drive types in one pool? Actually it’s pretty clever, le me go a little deeper into that:

Homogeneous pools

Pools with single drive type are recommended for application with similar and expected performance requirements. Only one drive (Flash, SAS, NL-SAS) is available for selection during pool creation.

Homogeneous Storage Pools

Homogeneous Storage Pools

Picture above shows three Storage pools, one created from Flash Drives (extreme performance tier), another created form SAS drives (performance tier) and the third one created from NL-SAS drives (capacity tier)

Heterogeneous pools

As mentioned previously, heterogeneous pools can consist different types of drives. VNX supports Flash, SAS, and NL-SAS drives in one pool. Heterogeneous pools provide the infrastructure for FAST VP (Fully Automated Storage Tiering for Virtual Pools).

Heterogeneous Storage Pools

Heterogeneous Storage Pools

FAST VP facilitates automatic data movement to appropriate drive tiers depending on the I/O activity for that data. The most frequently accessed data is moved to the highest tier (Flash drives) in the pool for faster access. Medium activity data is moved to SAS drives, and low activity data is moved to the lowest tier (NL-SAS drives).