This post was initiated by a comment from Vladimir Lavrentyev to my recent post on HDP.

First, Hitachi Dynamic Provisioning creates an HDP pool that maps across multiple RAID groups. There may be any number of RAID Groups in an HDP pool. Since HDP volumes are striped across the width of the HDP pool, they are striped across multiple RAID groups and have greater protection from multiple disk failures. With an HDP volume you could have multiple disk failures and not lose data as long as you did not have more than 1 disk failing in a single RAID Group. With an HDP Pool built from RAID 6 RAID Groups you could have two disk failures per RAID Group. The RAID generation for a RAID Group is performed in the back-end RAID director hardware. Unlike other thin provisioning systems that have no externalized storage virtualization, the USP V can leverage either internal or external RAID director hardware. If the storage is external, the overhead for RAID is off-loaded to the external storage system. Thin provisioning systems that use small chunks have additional overhead in managing the placement of chunks across RAID groups.

In terms of the tagging and mapping of pages or chunks, most thin provisioning systems use the same processors, cache, and busses that are used for data flow. This causes contention which will impact performance. In the USP V/VM, these processes are separated to avoid contention. The mapping information is maintained in a separate control store from the data cache. This control store is a “global” control store in that it has direct access from all the processors in the USP V/VM, so that an update that is made to the map by one processors can be seen by all the other processors. This direct access is separate from the access path to the data cache. All HDP mapping tables are present in the control store at all times so that the USP V/VM does not have to wait to retrieve less frequently used mapping segments from storage. HDP also uses a 42 MB page size which reduces the need to access these tables as frequently as other systems that use smaller chunk sizes. In these ways the USP V/VM eliminates overhead and reduces contention with the use of HDP.

So while there is overhead in HDP, as there is in all thin provisioning systems, this overhead is minimized and architected to avoid contention with data access.

But even where the OS or File System is not thin provisioning friendly,  customers are using HDP for its many other benefits.

Recently I visited a large financial customer in Switzerland, where the storage administrators have been critical about the usability of storage management software in the past. The biggest advantage they saw in HDP was the ease of provisioning storage. Instead of taking hours to carve LUNs out of RAID groups, they could provision storage in a matter of minutes out of a virtual pool of preformatted pages. They made the surprising statement that HDP made provisioning so easy that it nearly eliminated their jobs!

Claus Mikkelsen visited a financial customer in New York where they were excited about the performance improvements which come from the wide striping of pages across the width of the HDP pool. They saw a 10 – 15 min batch job run in less then a minute. Claus points out that a major task of provisioning, especially for data bases, is to provision for performance Since HDP stripes across all the disks in the HDP pool, and now with v05, automatically rebalances the stripe when new pages are added to the pool, performance tuning by manually balancing spindle usage is a thing of the past.

Another customer I talked to was a telco who provided storage as a service to their internal customers. Since their customers paid for the storage; IT did not over commit the storage pool. If a user paid for 10 TB, they set aside 10 TB in the HDP pool. However, when they provisioned storage for the user they used HDP to provision the pages that were actually used. This not only gave them the performance benefits of wide striping, but also gave them the ability to copy, move, replicate, and tier only the pages that were actually used, and not the whole over-allocated volumes. This resulted in lower operating costs. If a customer happened to under estimate the amount of storage they needed, IT could find enough virtual capacity to meet their new requirements in a matter of minutes instead of the days that it would normally take.  They could also charge them for that service which would be outside the normal SLA.

So while many customers think of HDP as thin provisioning which only applies to certain OS and file systems, many users are experiencing the dynamic benefits of HDP which are independent of OS and file systems.

This morning Hitachi Data Systems announced some new enhancements to the AMS 2000 modular storage platform. These new enhancements include Dynamic Provisioning, Dense Expansion Trays, AMS 2500DC for NEBS compliance, 8Gbs FC storage ports, and security enhancements. Since we began shipping the AMS 2000 platform in October of last year we have shipped over 60PB on this platform, making this one of our more successful storage products in the modular storage market. To understand why this product is has garnered such acceptance we must first understand the modular storage market and what sets the AMS 2000 apart from the rest.

Modular storage was created as external storage for open system servers. The basic architecture has remained the same for over twenty years. Modular storage is configured in drawers that fit into an industry standard 19 inch racks. One drawer contains two controllers which manage the I/O access from the servers, caching of the data to minimize the mechanical latencies of the disks, generation of RAID parity for data protection, and access of data from the disks. Controller drawers are connected to one or more drawers which contain the disk arrays. Today most disk drawers are connected to the two controllers through a FC loop. Each drawer, whether it is a controller drawer or a disk drawer is “modular” in that each has its own power and cooling. This makes it possible to add disk drawers simply by connecting to the FC loop, unlike monolithic storage where power and cooling is in the frame and disk arrays must be added in prescribed locations for power and cooling and cabling to back end directors.

While the Hitachi Adaptable Modular Storage 2000 is still modular storage in that it is a dual controller storage system which comes in drawers that fit into a 19 inch rack, nothing else remains the same. Attached is a summary of the major enhancements that differentiate it from your usual modular storage system.

The controllers in the AMS 2000 are active/active. An I/O can come through either controller and the workload is automatically load balanced between the two controllers. This eliminates the need to assign LUNs to one controller or another, making it simpler to configure the AMS 2000 over other modular storage systems.

The FC loops that connect the disk drawers have been replaced with Serial Attach SCSI with expanders which provided point to point connections which eliminates the overhead of loop contention and provides isolation of disk failures.

Hitachi Dynamic Provisioning will be available in the AMS 2000 in Q3 2009. This virtualizes the capacity in the AMS 2000, making it easy to provision storage requests in a matter of minutes, improve performance with automatic wide striping, and thin provision by eliminating unused capacity for most allocation requests.  This will increase the utilization of storage capacity and reduce operational costs.

High Density Expansion Trays are drawers which have been configured to contain 48 disks in a 4u drawer instead of the 15 disk that are contained in most modular storage drawers. With 1TB SATA disks one drawer would contain 48TB of raw capacity, reducing the cost of capacity to about $2/GB (list), ideal for tier 3 applications. The AMS 2000 can move data between tiers of storage with in the AMS without disruption to the application.

8 Gb/s FC host port options. Up to 16 x 8 Gb/s on the AMS 2500, provides the speed to support Virtual servers with their aggregated need for high speed connections, or high speed demands of multi-core processors.

A new model, the AMS 2500DC, was introduced with DC power and NEBS (Network Equipment Building Systems) compliance for use in Telco Central Offices. DC power is a growing requirement for “Green” data centers who hope to save power by reducing the number of voltage changes and conversions between AC and DC.

The addition of an external authentication capability on the AMS 2000 allows customers to leverage their existing infrastructure such as  Active Directory, as well as Multi-Factor authentication, without being locked into proprietary authentication solutions in the storage layer. Couple this with the AMS 2000s recent Common Criteria EAL-2 Certification, and customers now have Enterprise class security functionality in a mid-range platform. This certification is an EAL-2 and covers authentication, Role Based Access, account management, audit logging, and secure management actions via SSL (Note: The Common Criteria Certification for the AMS 2000 is pending).

While priced and configured as your common modular storage system, it has a number of enterprise features which can reduce costs and complexity while improving performance and security.

As the volume is moved into the pool it is moved in increments of 42MB pages and the pages are striped across all the disks in the pool. At the end of the move, the USP V/VM switches the internal paths to the new volume in the HDP pool. The pages are checked to see if they are all zeros. These zero pages are reclaimed and returned to the HDP pool for use by other applications. The old (fat) volume can be shredded (over written) to erase the old data and made available as free capacity. Depending on the configuration and location of the old volume, it may be added to the HDP pool to expand its capacity and wide striping performance.

For example, let’s say that we have a 400 GB volume which we migrate into a new 4000 GB HDP pool. After we move the volume into the pool, the microcode examines the pages that were assigned to the volume and finds that half the pages contain zeros. These zero pages are reclaimed and returned to the HDP pool, so now the volume has been thinned to 200 GB and the remaining free capacity of the HDP poll is 3800 GB. The old 400 GB volume can be shredded and added to the HDP pool, increasing its total capacity to 4400 GB and its available free capacity to 4200 GB. When this new capacity is added to the pool, the thin 200 GB volume is re-striped across this additional capacity to increase the effects of wide striping performance across these additional disks. In this way we can continue to free up more capacity and even increase performance until we run out of fat volumes.

Since this is all done within or behind the USP V/VM, the applications, the servers, and the SAN are not aware of the change from fat to thin. No changing of World Wide Names, no rezoning of the SAN, no exposure to configuration errors. The only difference that the applications and servers will see is an improvement in I/O response time and throughput since the thin volume is now wide striped across all the disks in the pool.

Most users experience between 30% to 40% reclamation of storage capacity and improvement in throughput by several hundred percent due to the automated wide striping of pages across all the disks in the pool. The HDP license is required, but it is free for the first 10TB. The pay back is almost immediate as you free up capacity in a matter of hours. However, some file systems are not thin provision friendly since they write metadata intermittently across the file space. What if you move a volume in to an HDP pool and you find that you have not reclaimed any capacity?  You will still experience the performance benefits of wide striping and automating load balancing. But if you are still not happy, you can back it out by simply migrating the thin volume back to its old fat condition, without disruption, and hope that your application users do not notice the change in performance.

There are other storage systems that do thin provisioning but unless they also provide the ability to do virtualization they will not be able to reduce your existing fat volumes without disruption to your applications, servers, and SAN. What happens in the USP V/VM stays in the USP V/VM and they don’t need to know.

The Storage Reclamation Service starts with a free assessment of the customers allocated but unused space and the types of volumes that can benefit from Dynamic Provisioning. This assessment is done with Hitachi Data Systems tools by HDS or HDS Partners. Once the assessment is done, HDS and the customer will agree on the percentage of storage space that can be reclaimed and the cost of the services to plan, design, implement, and integrate Dynamic Provisioning and Zero Page Reclaim. After the service is complete, and if the expected percentage of storage is not reclaimed; the service charges will be waived.

Of course this requires that you have a USP V or VM that has the Dynamic Provisioning license and that you have the Basic Operating Software V if you want to Dynamic Provision external storage behind the USP V/VM.

Currently there is a “Switch It On” promotion for virtualization of 3rd party storage systems behind the USP V which runs till the end of the year. With this promotion the Basic Operating System V, which enables virtualization of external storage, Tiered Storage Manager for policy based movement of storage across internal and external tiers of storage, and In-System Replication across internal and external storage systems, are free of charge for 3rd party storage capacity that is attached to the USP V by the end of this year. In addition there is a free “Dynamic Provisioning” license for 10TB on the USP V which can be used for internal and external storage systems.

This combination of free assessment, free capacity licenses for 3rd party external storage attachment, and risk free services for storage reclamation, is an easy way to upgrade your storage resources, show an immediate payback, and provide a return on your investment  for years to come. After the immediate payback of storage reclamation, customers will discover the operational benefits of Dynamic Provisioning in the ease of managing several pools of virtual storage capacity versus the traditional, labor intensive, management of hundreds of physical LUNs and RAID groups.

These results were not surprising to HDS since we have been advising our customers to do more with less. Since the introduction of the Universal Storage Platform, we have been advising our customers to increase the utilization of their existing storage assets through the use of storage virtualization. We have developed tools and methodologies around storage economics to help quantify the savings that can be gained through storage virtualization, Dynamic Provisioning, and tiered storage. With an average storage utilization of 30 to 40%, there is plenty of room to grow without buying a lot of new capacity.  Not only does this reduce capital expenditures, it also reduces operational costs, through simplification of provisioning, migration, and load balancing tasks.

While these tools can help save costs during the current downturn, they also provide customers with the agility to scale up when the economic recovery kicks in. Virtualization and Dynamic Provisioning will enable the seamless provisioning of new storage capacity and functionality without disruption to existing applications. Those who hunker down during this down turn, by shifting to lower cost, lower function storage, and micro managing their utilization will not have the ability to respond quickly to increased business demands.

With the USP V/VM you can put most of your storage capacity on lower cost commodity systems without sacrificing agility and enterprise functionality by virtualizing behind the USP V/VM. Increasing utilization and decreasing the purchase of storage is not bad, as long as you do not give up the capability to quickly scale up when the business requirement change.  Selling less storage through the application of storage virtualization, tiered storage, and dynamic provisioning, helps our customers to reduce costs and improve operational efficiency so that they will be better positioned to grow with the coming recovery.

When I see that I think of the presidential archives that President Bush transferred to the National Archives  on January 20th when he left office. There was reported to be about 100 TB which was 50 times more that the amount of digital data that was archived for the Clinton presidency. President Bush did not use email and did not have a laptop for security reasons.

With President Obama, they have not only  solved the security problem with Black berrys, but his administration has embraced social networking, with accounts on Myspace, facebook, and twitter.

In President Obama’s first 100 days in office he has probably generated more digital data than President Bush did in 8 year of office! When President Obama leaves office after eight years, there will probably be tens of petabytes of digital data to be archived.  President Bush’s archives are on a Hitachi Content Archive Platform, since it is the only content archive that can scale to petabytes. 

If you watch the video of the NBC special, it is interesting to watch the segment on Anatomy of a Talking point, to see how quickly this administration responded to the criticism for President Obama’s nomination to the Supreme Court, Sonia Sot0mayor.  I wonder how much of this will need to kept in the Presidential archives?

In another segment President Obama goes out to buy hamburgers for his staff. I know that menus for state dinners are part of the archives. Will he need to save his receipt for these hamburgers?

Another big hit is Dynamic Provisioning. A large bank who has consistently  pushed for ease of use,  had implemented HDP for a 400TB data warehouse application. While there was some benefit from thin provisioning the operations people were  overjoyed with the ease of provisioning that this provided. No more need to look for space on array groups, and formatting, and concatenating, and wide striping.  All this was done for them by Dynamic Provisioning.  One of the operations persons said that this nearly eliminated his job. Every USP V and VM has a free 10 TB license, so more and more users are turning it on.

The storage market in Switzerland is still very healthy. But like every where else they are looking for ways to reduce cost and these products meet their needs.

Since the USP V/VM is able to virtualize external storage, this will be the first continuous availability solution for both internal and externally  attached heterogeneous storage, all under  common management.

In fact this ability to cluster two separate control units over distance requires an  external disk attached to both control units to act as a quorum disk.  If the link between the two storage nodes is broken the quorum disk helps to identify which node is the correct node and which node needs to be fenced off.  If the quorum disk was internal to either of the two controllers it would not be able to provide the quorum function. So storage virtualization is required to make storage clustering work reliably.

What is the difference between High Availability Manager and business continuity solutions like TrueCopy and Hitachi Universal Replicator? The difference is that the secondary USP V/VM can be accessed immediately with High Availability Manager when the primary USP V/VM goes down for any reason.  With TrueCopy and Hitachi Universal Replicator, the applications have to be restarted on the secondary USP V/VM.

Imagine the work that is saved when a VMware server running multiple applications, can simply switch paths to a secondary storage controller when the primary storage controller fails.

In my next post I will talk about the benefits this provides for device migration.

In modular storage arrays with two controllers, LUNs are assigned to one controller or the other to avoid thrashing between the controller caches. The Hitachi AMS 2000 avoids this by switching the I/O request to the controller cache that contains the data and load balances the activity between the two caches.

So perhaps the V-Max is doing what the AMS 2000 has been doing for some time. That works for one V-Max node of two controllers. What gets harder to do is cluster multiple V-Max nodes and still present a global cache data image. Namely, have an I/O come in from one V-Max node and be routed to another V-Max node where the data resides in cache. There needs to be some sort of locking for cache consistency and some sort of arbitrator to tell which node has the correct status if the two of them become out of sync. That takes overhead if you have to go over an external wire instead of a backplane. Performance will be great if you happen to access the node where you data resides, but there will be a performance impact if it doesn’t.

If you happen to be going to EMC World next week, see if you can find out how this is done.