In addition to capacity allocation efficiency, which most of us are addressing with thin provisioning, Jon points out the need for capacity utilization efficiency, storage performance efficiency, data protection efficiency, and storage energy efficiency. Expanding on his thoughts, I have added storage management efficiency.

Capacity utilization efficiency is about the placement of data on the appropriate tier of storage, based on frequency of access, business value and cost of the storage. This could be addressed by automated tiering based on policies that are triggered by time or events.

Storage performance efficiency could be addressed by automated wide striping or page level tiering, where only the hot pages of a volume—rather than the whole volume—is moved to high performance tiers of storage.  Ray Luccesi has a great take on storage performance efficiency in his IOPs vs Drive Counts chart of the month, which he posted last week.

Data protection efficiency, which is measured in terms of Recovery Time and Recovery Point objectives (RPO/RTO), is a major area for improving efficiency. This has to do with replication, backup, recovery, archive, etc. If most of the data is static data, which is not being updated like most unstructured data, you only need two copies for redundancy. You can eliminate making the many snapshots and backups of the same unchanged data over and over again.  Brad Clarke commented on my post that the most important storage efficiencies to him were the ones which make replication less bandwidth hungry. He makes the point that when data volumes increase, the cost of disks to contain that capacity is relatively cheap—compared to the cost of the increase in bandwidth pipes that is required to replicate it.

Energy efficiency will be a big focus this year based on the record increase in carbon emissions in 2011 (5.9% increase) that was reported by The New York Times. Another factor is the nuclear problems at Fukushima, which has sifted demand from nuclear power to carbon fuels, and is raising the cost of energy, as well as the possibility of carbon taxes on top of the energy bill. Since storage is becoming a greater percent of the power consumption in the data center, storage energy efficiency is becoming a key consideration for buying decisions. Storage energy efficiency benefits from the other efficiencies cited above, but there are also efficiencies of 40% or more with storage systems like VSP, which use Small Form Factor SAS drives, dense drawers, front to back cooling, and the replacement of batteries with flash for protection of the cache.

Another efficiency that comes to mind is Storage management efficiency: the ability to manage heterogeneous storage arrays as a pool of common resources with a common set of tools, so that resources like capacity and performance can be shared rather than isolated in silos.

Are there other areas of storage efficiency that we should be considering?

Here are more posts on capacity efficiency:

• A Consensus on Storage Efficiencies
• How Much Does it Cost to Spend Money?
• The Tipping Point for Hard Disk Prices?
• Buying Disks or Buying Storage Efficiencies
• Look Beyond the Price of HDD
• Capacity Efficiencies, Again and Again
• How Thin Provisioning Contributes to Storage Efficiency

In view of Gartner’s recent prediction of a 5% to 20% increase in disk prices and shortages due to last year’s floods in Thailand, Hitachi Data Systems will extend and enhance the Switch It On III promotion that reduces the cost of virtualizing and managing heterogeneous storage on a Virtual Storage Platform. This promotion is targeted to helping customers increase utilization and reclaim capacity on existing third party storage in the face of raising disk prices and shortages. Customers who take advantage of this promotion will be able to reduce storage CAPEX and OPEX and increase their return on assets.

Photo by Kevin Kevin Pelletier

The Switch It On III promotion, which was due to expire at the end of March has been extended to June 30, 2012. The promotion reduces the licensing and maintenance costs for virtualizing heterogeneous storage systems. It also includes price reductions on thin provisioning, dynamic tiering, disaster recovery, and in-system replication software, as well as management tools for tuning and Command Director that provides an end-to-end application to infrastructure view of utilization and service level objectives.

This promotion is available on new purchases of VSP and existing VSP systems that have not already virtualized external storage.  Hitachi storage virtualization enables customers to fully utilize the resources that they already have in their multivendor environment. It creates a single pool of heterogeneous storage capacity that they can control and optimize with a powerful suite of management tools. It lets them take advantage of cost saving features like thin provisioning and data mobility with their existing storage systems and provides them with a sustainable storage architecture that that can protect and grow into the future.

For specific details on this promotion contact your Hitachi Data Systems representative or reseller. Also, check out more details on Switch It On III.

Also, here are some other blog posts on storage efficiencies:

• Look Beyond the Price of HDD
• Buying Disks or Buying Storage Efficiencies
• Capacity Efficiencies, Again and Again

The most common theme is the explosion of data, and the need for storage efficiencies. Jon Toigo says that 70% of the capacity on every disk today doesn’t need to be there–40% should be archived and the other 30% should be reviewed and probably deleted.

Joe Kovar of CRN predicts that growth in storage capacities decline as users implement more storage efficiencytechnologies like thin provisioning, deduplication, and cloud. An optimistic prediction is that the impact of the Thailand floods on disk shortages will wane sooner than expected, and the expectation is that disk shortages will wane by the second half of this year. I agree that the impact may wane sooner as users implement storage efficiencies. However, I believe there will be a fundamental change in the pricing of storage capacity, as I posted last week.

David Chapa believes storage is becoming more and more affordable to the masses, through the adoption of small business cloud services. He recognizes that home office users now have several terabytes of data stored locally and the increasing costs of managing that data, similar to the enterprise. While we have seen consumer prices of disks more than double in the last two quarters—$79/TB to $190/TB was quoted at a recent Gartner conference—the total cost of managing storage far exceeds the cost of acquisition. Cloud services can reduce the cost of this management and make the total cost of storage more affordable, even though the cost of the disk may increase. See what David Merrill says about procurement costs.

What do you think about storage efficiencies and the impact of disk prices on storage costs?

Here are more posts on capacity efficiency:

• How Much Does it Cost to Spend Money?
• The Tipping Point for Hard Disk Prices?
• Buying Disks or Buying Storage Efficiencies
• Look Beyond the Price of HDD
• Capacity Efficiencies, Again and Again
• How Thin Provisioning Contributes to Storage Efficiency

FORTUNE worked with Great Place to Work Institute to conduct the most extensive employee survey in Corporate America. While this survey applies to the United States, we have similar results across Hitachi Data Systems globally. The Great Place to Work Institute ranked HDS #5 in Poland and #13 in France.  In Silicon Valley—one of the most exciting places to work in technology—we ranked #3 and placed in the top 10 for the last three consecutive years!

How do you define a great workplace? Great Place to Work Institute, who has been doing research on this for over 25 years, has this to say:

“Trust is the defining principle of great workplaces – created through management’s credibility, the respect with which employees feel they are treated, and the extent to which employees expect to be treated fairly. The degree of pride and levels of authentic connection and camaraderie employees feel with one are additional essential components.”

HDS has an entrepreneurial culture where we are constantly improving and innovating. At the foundation of our innovation is trust, an integral part of the Hitachi culture (Hitachi Spirit) that we have inherited from our parent Company Hitachi, Ltd:

Hitachi Spirit is what distinguishes us as the employer and partner of choice. It’s more than our foundation, more than a poster on the wall. It’s how we operate every day, how we get things done. It’s who we are.

See what our employees are saying on the top workplaces website.

Above is a chart showing disk drive capacity increases going back to 1980, which is truly phenomenal. Due to this rate of capacity increase, the industry has enjoyed an annual price erosion of about 20% to 30% per year on disk media. However, you can see that the density curve is starting to slow down as we approach the limits of current perpendicular recording technologies.

With this historic price erosion, most data centers depreciate their enterprise storage over three years while midrange storage is typically depreciated over five years considering it has been cheaper to buy new than to maintain the old after that time frame. If this price erosion starts to slow down, data centers may need to extend their depreciation to seven years. By this time, the disks will be in the five to ten TB range, so keeping the media longer may not be a bad idea.

However, there is a lot more technology that goes into storage systems than the disk technology and the rate of that technology has been increasing rapidly with thin provisioning, data mobility, tiering, replication, and closer integration with the application layer through APIs, plugins, client/providers, adapters, and snap-ins. That means that a five to seven year life cycle for storage systems will make your storage system non-competitive. The reason why enterprise storage is capitalized over a shorter period than lower cost modular storage is because of the higher technology cycle of enterprise storage.

An approach to solving this is to separate the disk capacity from the enterprise storage system controllers, so that you can keep the storage system controllers current with systems technology on a three year cycle, while you refresh the disk capacity on a five to seven year cycle. Since the storage media is still the bulk of the cost of a storage system, the longer depreciation cycle will help to reduce the capital costs. You can do this with an enterprise storage control unit, which also has the capability to virtualize external storage systems. This is what we provide with VSP.

What are your thoughts? Are the price increases that we expect this quarter just an anomaly and will we go back to enjoying the price erosion that we have enjoyed for the last 50 years? Or has this changed forever?

Is this a tipping point in the way we capitalize storage assets?

The shortages have been felt the most in the consumer markets. At the Gartner Data Center Conference in Las Vegas, a speaker cited the costs of a TB disk at the U.S. retailer Fry’s had gone from $79 to $190. Last November, the Storage Architect reported that a 2TB SATA drive that he had bought before for £65 was then listed at £150 on Amazon. Consumer markets run on very low margins, so the price can increase dramatically in response to any shortages.

In the enterprise space, the shortages have been real but the prices have been more stable. Some drive types have increased on the order of 5-15%. Consult your vendor to see what the current status is. Hopefully we will see the supply situation return to normal by the end of 2Q.

The Storage Architect supports my view of focusing on storage efficiency during this period by using thin provisioning and other efficiency methods in his post Drive Prices Increase – Who Will Suffer Most?

“If your vendor doesn’t offer it, then there are plenty out there who do.  As prices rise, it may be time to look again at implementing these features and fixing the processes that stop you using them today.”

Investing in storage virtualization through VSP adds another dimension to storage efficiency by extending these new capabilities to existing storage systems. If your current storage system does not provide thin provisioning, which can reclaim 40% or more of allocated but unused capacity, you don’t need to rip and replace it. Just by attaching it behind VSP, VSP can see your existing LUNs, and move them into a dynamic provisioning pool where the unused pages in the LUN can be reclaimed (Zero Page Reclaim) while your application is running.

So instead of buying additional disks during this period of shortages, invest in storage virtualization with VSP, which not only frees up the capacity you need today from your existing storage assets, but positions you for sustainable growth into the future. See what Claus Mikkelsen and David Merrill have to say about storage efficiencies.

Garth is best known for the research paper that he authored with David Patterson and Randy Katz in 1988, “A Case for Redundant Arrays of Inexpensive Disks (RAID)”, which was the catalyst for the RAID storage industry. Once this paper was published and presented at conferences, it took only a few years for all the major storage vendors to deliver RAID storage systems and for customers to adopt this new technology. The rate of this adoption was phenomenal.

When I asked Garth about this, he said the reason RAID was adopted so quickly was that this was delivered as a paper and not as a patent. It was freely available to the industry. Garth and Randy also included a taxonomy that defined RAID levels 1 to 5 and mathematical calculations to determine Mean Time To Failure (MTTF), a key factor in fault tolerance.

It also was helped by the availability of relatively inexpensive 5.5-inch disk drives and the premise that a RAID array of inexpensive disk drives could replace more expensive enterprise storage systems with the same reliability and performance. The industry was quick to drop the term “Inexpensive” in favor of “Independent” and RAID was redefined as Redundant Array of Independent Disks.

Tracing RAID’s Origins

Actually, the concept of RAID was introduced much earlier. Garth does not claim that he invented RAID. The earliest patent on RAID was filed by Norman “Ken” Ouchi of IBM, who was issued U.S. Patent 4,092,732 titled, “System for recovering data stored in a failed memory unit” in 1978. (Claus Mikkelsen and I worked with Ken at IBM.)

This patent described what Garth and his colleagues later defined as RAID 5. Ken’s patent also mentioned mirroring (RAID 1) and dedicated parity (RAID 4) as prior art at that time. So RAID has been around for some time but was not adopted until the RAID paper in 1988 which gave it a name, taxonomy, and a financial justification.

RAID as a fault tolerance mechanism for storage is running out of gas as the densities of disk media increase and the probability of multi drive failures increase. RAID levels up to RAID 5 only protect against a single drive failure in a RAID group. As densities increase, the probability of a drive failure increases and the RAID rebuild time also increases which affects performance due to drive contention and the increased probability of another drive failing during the rebuild. There is also an increasing problem with uncorrectable read errors as densities increase.

Getting Familiar with Erasure Codes

While everyone is familiar with RAID as a method for protection of a drive failure, many are not familiar with the term “erasure code”.  The image on the left come from IEEE to illustrate the concept.

In information theory, an erasure code is a Forward Error Correction (FEC) code for a binary channel (where data is transmitted as one of two symbols, usually a 0 or 1) that can reconstruct symbols that are erased.  It can be used for networks as wells as storage.

RAID is really a simple form of an erasure code where a parity or check sum is appended to a number of records, so that if one record is lost, it can be reconstructed by summing or XOR the remaining records and parity.

If we want to correct more than one error, additional redundancy must be added and the calculation now becomes a polynomial. This is where you will begin to hear more about erasure codes. RAID 6 is a polynomial erasure code that was introduced with large capacity drives in the last decade to protect against dual drive failures. RAID 6 has two redundancy records, so it requires more overhead than RAID 5 in capacity and processing.  As a result it has not been widely adopted until recently.

RAID 6 also helps with uncorrectable read errors. Today, we strongly recommend the use of RAID 6 with RAID pools where data is stripped across many RAID groups, since a dual drive failure in one RAID group would create data loss in all the applications that are using this pool. The cost of an additional parity drive in each RAID group is relatively inexpensive compared to the application down time and the cost of recovering an entire provisioning pool.

However, RAID 6 is not a long-term panacea since it only protects against dual drive failures. With the increasing rate of drive densities, it won’t be too long before we get concerned over three or more drive failures in a RAID group. Storage vendors are working to address these long term requirements.

While storage systems vendors source their disks from the same disk vendors, the reliability of the disk in a storage system will vary depending on how well the system vendors scrub the drives for errors, the effectiveness of their proprietary error detection and recovery software, their maintenance practices, and their proprietary implementation of erasure codes. Users will need to consider the track record of the vendor’s disk availability and then consider the costs and performance trade offs of different erasure codes.

Today, Nikkei, the leading business magazine in Japan, announced that Hitachi VSP was the winner of the 2011 Nikkei Superior Products and Services Awards. This year, over 20,000 products were submitted and only 44 were chosen for the award. Nikkei chose VSP for its usability and high performance through virtualization to reduce CAPEX and OPEX.

This is the latest addition to the list of awards VSP has received, which collectively recognize its innovation. Among these awards are the Information Age Innovation Award for 2011 and the TechTarget Best Enterprise Storage Platform Award for 2010.

The best awards are those that come from our customers, who have made VSP the most successful product in Hitachi Data Systems history. Most customers, who take advantage of the full value of VSP with virtualization of their existing assets, can see a payback in nine months or less, and a 40% reduction in capacity demand.

So if you are looking to increase your storage resources while struggling with budget constraints, contact your HDS account representative or HDS Partner to see what VSP can do for you: http://www.hds.com/corporate/contacts/

In order to meet these challenges, IT will focus on technologies that will enable them to grow through better use of their existing assets. During the last few months I have been blogging about ten storage trends that will develop out of this focus.

As my last post of 2011, I am summarizing them here with links to the extended posts.

1. Storage Efficiency: Global economic uncertainty and supply shortages in the first half of 2012 will require IT professionals to achieve better returns from their existing assets rather than buying new ones. There will be a greater focus on storage efficiency technologies such as dynamic or thin provisioning, dynamic tiering, archiving, and the extension of these technologies to existing assets with storage virtualization. http://blogs.hds.com/hu/2011/11/2012-a-focus-on-increasing-storage-utilization.html
2. Consolidation to Convergence: Consolidation will give way to convergence. Over the past few years IT has focused on consolidation, with much of the low-hanging fruit already executed on. In order to gain further cost savings, the focus will be on convergence of server, storage, networks, and applications. Application programming interfaces (APIs), which offload workload to storage, can make servers and memory more efficient. Orchestration software will help to converge the management, and automate the provisioning and reporting across local, remote, and cloud based server, storage, and network infrastructures. http://blogs.hds.com/hu/2011/11/2012-trend-consolidation-to-convergence.html
3. Transparency: Applications and infrastructure will be more transparent with each other in order to facilitate convergence through open interfaces like APIs, client/providers, and plugins. HDS provides Hitachi Command Director software, which gives applications a view into the service level, utilization, and health of the storage infrastructure behind the virtual storage that they are using.  http://blogs.hds.com/hu/2011/11/2012-trend-greater-transparency-for-end-to-end-management.html
4. Storage Computers: Storage systems will need to become storage computers as more functions are being driven down to the storage level.  Old storage architectures with general purpose controllers which service all these new functions along with the normal I/O workload will not be able to scale. New storage architectures with separate pools of processors will be required to handle these additional functions. http://blogs.hds.com/hu/2011/11/2012-trend-rise-of-the-storage-computer.html
5. Energy Efficiency: Power, cooling and carbon footprints will become even more critical as energy demand increases and countries begin to impose carbon taxes. IT will be asked to shoulder their share of the energy burden and will need to consider the energy savings of small form factor drives and front-to-back cooling. http://blogs.hds.com/hu/2011/12/2012-trend-energy-efficiency.html
6. Closing the Consumption Gap: The consumption gap between technology and IT operations will be an area of focus as businesses drive IT to implement technologies at a faster pace in order to realize the benefits that are already available with current technology. There will be a greater need for services to offload over-committed IT staff and accelerate the adoption of new technologies. http://blogs.hds.com/hu/2011/11/the-technology-consumption-gap.html
7. Storage Scaling: Sever and desktop virtualization will increase the need for enterprises to scale up storage systems non-disruptively as virtual machine demands increase. With more virtual eggs in one basket, modular storage systems will need to be replaced by high availability enterprise storage to service the tier one demands of virtual servers. Scale-out storage architectures will not be able to meet the scale-up demands of server and desktop virtualization.  http://blogs.hds.com/hu/2011/12/2012-trend-an-increasing-need-for-scale-up-storage.html
8. Virtualized Migration: Disruptive device migrations will be replaced by new virtualization capabilities that will eliminate the need to reboot. http://blogs.hds.com/hu/2011/12/2012-trend-virtualized-migration-of-storage.html
9. Cloud Acquisition: Cloud acquisition, based on self-service, pay per use, and on demand will begin to replace the current three-to-five year acquisition cycle of products, as convergence begins to create blended pools of resources. http://blogs.hds.com/hu/2011/12/2012-trend-cloud-acquisition.html#more-5098
10. Big Data: The hype for 2012 will continue to be around Big Data. The explosion of unstructured data and mobile applications will generate a huge opportunity for the creation of business value, competitive advantage, and decision support if this data can be managed and accessed efficiently. The massive size of Big Data will make it difficult to use traditional, relational databases, or desktop visualization products. Object based content platforms and large-scale file servers will be required to store this data. There will be greater adoption of content platforms in preparation for Big Data analytics. http://blogs.hds.com/hu/2011/12/2012-trend-big-data.html

These are just a few of my personal thoughts regarding 2012. Please let me know what you think will be the major focus areas in storage next year.

“Big” is a relative term. What may be big for one company may not be big for another. Data is big when it becomes difficult to work with using relational databases or desktop visualization packages. The size of Big Data makes it impractical to replicate, backup and mine through traditional means. Instead of moving or replicating big data for use by analysis programs, analysis programs will have to work directly with the original data using massively parallel or map reduce software. Big data may be more about the intersection of many data stores.

Big Data is best stored in an object store built on a virtualized storage infrastructure so that it is not affected by changes in the infrastructure. An object store is a container, which holds data along with the meta data that describes the data, and the policies that govern it. An object store disaggregates the data from the application or instrumentation which created it, and essentially virtualizes the data so that it can be accessed and repurposed by other applications.

Since Big Data is predominantly unstructured data that is not updated, once it is ingested into an object store, it only needs to be replicated once for protection. Some data sources maybe acquired over NFS, which will require a high performance filer that can scale to multiple PBs and millions of objects—like the HNAS 3200.  HNAS also provides the ability to stub out to the HCP content platform as a seamless extension of the file system into an object store.

The interest in Big Data will drive greater adoption of object stores like HCP and large scale file servers like HNAS from Hitachi Data Systems. It will also open up a huge market for analytics. HDS has partnered with SAP with certification of a reference architecture for their in-memory transaction analysis system, HANA, using our Hitachi Blade servers and AMS storage.

For Hu’s other 2012 trends, visit this bit.ly bundle: http://bitly.com/vXGP2T