I hope that more of option C comes about, because as an industry we are over provisioned and under utilized. Poor utilization has several explanations, and there is not enough room here to go into these reasons, but when average utilization (not allocated) are still hovering in the 30 percentile, there is more that can be done. Economic tough times are encouraging methods and technologies to reclaim existing capacity and to avoid unnecessary capital spending.

Is the idea of using what you already have ‘storage reclamation’ or ‘storage restoration’? A dictionary view of the two terms are different, and perhaps our view of giving back storage to the enterprise can be classified differently as well. The bottom line is this: a TB reclaimed (or restored) is a TB that does not need to be purchased. You can quickly determine the street price of XX TB of capacity, and that would be what you can NOT spend if you are able to recover or restore or reclaim XX TB from what you already have. Using better what you have also improves Return on Asset (ROA).

This is not a new topic on my blog – there is an old blog entry on stop buying storage; and an entry on physical reduction, which is different from reclamation. Then a stand-alone entry on basic disk reclamation; and one from the Storage Architect.

The Economics of Reclamation Process and Technology is very simple:

• Just like server virtualization, storage virtualization can reclaim stranded capacity between frames and present a common pool of storage to various hosts and data service functions
• When tiered storage is available, higher tier capacities can be reclaimed by demoting data or volumes to a lower tier at the appropriate age
• Presenting an integrated archive solution can reclaim disk capacities on any/all tiers and send them to a dedicated, low cost (indexable) tier for long-term cold storage
• Thin volumes can spoof the OS or application by presenting virtual volume capacity while only spending physical pages when real data is needed or written
• Zero page reclamation can be applied when apps or operating systems write out pages of zeros to mark a section of the volume
• Data de-duplication will help with all the data copies that exist
• Then there is the old-stand-by: delete the stuff

These above bullet points (technologies) can be implemented stand-alone, but when bundled present a sum-greater-than-the-parts result in space reclaimed and saved over time. Perhaps these ideas are of interest, but you don’t know where to start. Perhaps you do not know your potential for capacity reclamation in your SAN or NAS environment. Since you cannot improve what you cannot measure, a baseline for the reclamation opportunity is usually the first step.

HDS is announcing this week a new service called Storage Reclamation Services, or SRS. A feature of this service included a limited assessment to an environment up to 5 servers. It is certainly worth a look to see (in a sample size) what the reclamation potential would be through thin provisioning and zero page reclaim. You too can be an economic rock star when you cancel the disk purchase in your budget and reclaim enough capacity to meet your growth needs with what you already have on the floor. That is what Wells Fargo did last year; you can read about it here. Now that Thin Provisioning or Hitachi’s Dynamic Provisioning is available on a modular storage array, there are new entry price points to make this technology closer at hand.

Here are some common trends that I see with customer engagements around the world, as it applied to space reclamation

• Allocation rates to hosts are at a relatively high ratio (60-80% for some), but the underlying usage rate within the allocated volumes is much lower, around 30%
• We have a banking client who is moving active volumes to be thin volumes, and is see a give back rate of usable capacity in the 35-45% range
• Reclamation has everything to do with the OS and the applications that you are running
• Reclamation and zero page reclaim tend to be seen as a one-time event, but periodic analysis of volumes and file systems is suggested
• Incentives and encouragements are needed to get applications or users to move to thinned or virtualized environments
  • Lower chargeback is always a good carrot
  • Provisioning time can be much faster (even applications can provision themselves) and be a good motivator
  • Let the storage consumer see the per-unit-cost-difference to the company with this new technology. Everyone is more socially and economically sensitive to spending these days
• Try before you buy. Make sure the right approach for reclamation is what works best in your environment.

Let’s take this to storage metrics, or Econometrics. Are there storage and management metrics that are non-measurable? Metrics that are based on macroeconomic metrics, and without any real counting? Let me list a few of my favorites:

•   TB-per-person under management – This seems to be simple, divide all the capacity between the number of people supposedly managing storage. Simple right? This metric is full of holes since some of the storage admin team (if there is a dedicated team) also tend to look after servers, backup, DR tests, development, compliance requirements etc. These fuzzy metrics can be used internally (at best) to show how much more storage has been bought without adding headcount. It does not fairly represent the effective management nature of the staff and the data.

•    Total cost of Storage – I do a lot of work in this space, but the metric does not describe the cost of data or information on the disk. TCO of storage is the container cost, but if the container is largely empty or full of copies or aged data, then the metric is meaningless. We have to better measure cost of 1st instance data or information. TCO per TB is very measurable (sum of all the costs divided by the total raw or usable capacity). TCDO (the D is for data) or TCIO (I is for information) are more difficult measurements, but far more meaningful to the data owners.

•    Total cost per Transaction – Measuring costs per TB or MIP are great for the infrastructure, but the business really cares about how much does a transaction cost me to process? IT spend as a percent of total company revenue is a related metric. Transaction costs may be non-measurable due to the problem in isolating and defining a transaction.

•    Total cost of acquisition - now this seems simple enough since we can query the procurement department on what was spent to but XX TB of storage. Again, we are measuring containers, and low cost containers may or may not adequately cover the business needs. Advanced containers cost more, but may do more in terms of operational , data protection, migration and discovery functions for the business. TCA of data or information is more meaningful, but potentially hard to isolate from the raw costings.

Any ideas on storage or IT metrics that are popular but non-measurable?

Premise 1: DAS disk architecture is cheaper to buy
•    In the SAN costs, overhead can be high with switches, HBA, directors, ISL ports
•    In most SANs you will find modular or enterprise-class Controllers, RAID overhead, Cache, FC ports that add to the $/TB rate
•    Software costs are usually very different in the DAS model since the protection and management functions have moved into the app layer, and are controlled by the host (as opposed to a dedicated storage controller with cache)
•    The unit level DAS disk drive is cheaper, and the basic TCO appears to be lower cost

Premise 2: Price does not equal cost
•    Acquisition cost is only 20% of the TCO
•    HDS has documented/measured 33 different types of ownership costs (download paper here)
•    Let’s not confuse TCA and TCO. There is usually a better TCA storage with DAS but TCO can be very different.

Premise 3: Cloud computing seems to have created a renaissance or return to older DAS architectures
•    Cheaper to build into nodes
•    Removal of RAID and software overhead is popular with some cloud architectures
•    Scaling and data protection is done by adding more CPU nodes (with disk attached)
•    Data management, protection, copies etc. is done at the app level, with local CPU overhead

So what about DAS TCA then? The unit costs are less; the functions have changed with scalability, management, protection overhead moving to a different part of the computer. Can we use the same $/TB metrics or $/CPU node metric as before when comparing other architectures? I have found that when node or cloud based architectures use naked disk or DAS, there needs to be introduced new overhead costs to support the data protection and scalability requirements of the environment. Raw disk is not usable disk unless there are protections and investments to make the TB usable. Take for instance a cloud DAS architecture that relied on many cluster nodes to provide copy protection and drive failure (in place of RAID) protection. These can be very low cost nodes, but the total sum of extra nodes to provide 1 TB of usable capacity may require 2-5X the raw requirement to meet the business needs. All of a sudden you need many low end disk nodes to provide the basic storage function found in a higher-cost counterpart.

In looking at overhead rates for cloud nodes (CPU and disk nodes) at a recent client, we found a cross over point in terms of TCA, when compared to virtual, thin and tiered FC SAN storage architectures. At a low node count (in the hundreds) DAS had a better TCA. There was a cross over point at around 1,000 nodes. Not a very good cross-over, since the data suggests that DAS and SAN achieved relative parity at around 1,000 nodes and that parity was sustained up to about 6,000 nodes.

In the above graphic, we calculated TCA for usable and written-to storage. DAS was a flat-line relative to TCA for usable, but in order to protect data and meet scaling requirements, the written TCA was about 9x the cost. SAN had predictably higher TCA at low quantities, but achieves relative parity at 1000 nodes.

Now this is just TCA, next posting will extend the cost models to include TCO with such factors as power, cooling, management, waste, migration time/effort etc.

A recent article from Forrester analyst Andrew Reichman was a significant reference point on this topic, as he poses the question as to whether we need SANs anymore. Without a comprehensive review, I do agree and disagree with some of the findings.  Where he is right with the indictment on SAN:
•    Stranded capacity still exists on the arrays
•    Tiered islands has continued, with larger and larger islands of disparate capacity
•    Some micro optimization (10-20:1 host to array ratios)
•    High costs of SAN overhead
•    Relatively poor utilization still
•    Provisioning time is still too high

But are these shortcoming the fault of the SAN and the transport architecture or the underlying storage architectures?  Some areas of DAS deficiencies that have been addressed with SAN and basic storage architectures are:
•    High performance
•    Better availability, data path and the storage array
•    Scalability
•    Consolidation
•    Remote boot
•    Server-less backup
•    Data mobility
•    Better DR
•    Central command and control

There is much to discuss about SAN and DAS architectures – for a quick overview I will refer you to a webtech seminar from Hu Yoshida on the tech differences of SAN and DAS, and more specifically how advanced storage architectures have filled-in on some of the SAN deficiencies that were promised a decade ago. That webinar (recorded last week) is available here.

My own interest tends to be in the economics of DAS and SAN. I have an upcoming webcast on ‘SAN vs DAS, a TCO Perspective’ and it is scheduled for June 10. You can register on-line here.

The next post installment will focus on one element of the economics of these architectures – Total Cost of Acquisition or TCA. Subsequent posts will deal more with the variations of TCO costs. But in case you cannot wait….

TCA Outline
•    DAS is cheaper than SAN at the unit level
•    SAN can have a very high overhead cost (HBA, path mgmt, Switch, ISL, directors, SAN SW, cables, cable and patch panels) when amortized by port
•    Short stroking and other techniques are often used to squeeze better performance on DAS
•    Growth rate will have an impact on DAS TCA
•    Where is the cross over point in terms of total usable capacity where SAN is lower TCA than DAS

TCO Outline
•    Cost of performance as a factor
•    DR and data protection costs
•    Cost of management, encryption
•    Costs and effort for provisioning
•    Thick vs. Thin
•    Power consumption per usable TB
•    Migration, end of life costs to retire assets
•    Use some of the key 33 types of money to create a comprehensive TCO analysis: TCA, lMaintenance, Power, Space, Cooling, Cost of waste, Provisioning time, Total estate, Admin, DR Protection, BAckup, Archive, Migration, ILM

A few weeks ago a large client was working with us to develop some new econometrics. We had calculated TCO and TCDO — that was a straightforward exercise. Then they asked us to create a storage carbon footprint measurement, TB-per-metric tons of CO2. To them, power, space, labor, A/C, conditioning can all be summed by TB-per metric tons of CO2.

The calculation is straightforward - the CO2 emission factor for electricity is 0.524 lbs CO2 per kWh, but the source of your electricity will have an impact on this ratio as well (nuclear, coal, wind, hydro).  Try it out, add up the storage arrays, SAN switches, tape libraries, media servers, backup servers, DWDM, operator consoles power and then divide by the TB (usable) to report on your tonnage per TB (or TB-per-tons) of CO2. Don’t forget to add in the cooling costs to the CO2 calculation (you can convert from BTU to kVA by dividing BTU by 3413).

It should not come as a surprise that some storage architectures, configurations etc. can produce LESS CO2 per TB than others. Over provisioned volumes, virtualized capacity, archive, de-dupe, SSD can all significantly contribute to an improved power consumption bill, and therefore an improved storage/carbon footprint.

Our parent company Hitachi and HDS have numerous programs around the green data center. Check out some of the links.

I also recall global and national calls for information archive related to the SARS virus, flight information (China to Toronto in the early days) for the first passengers coming into North America with the virus. SARS patient record data, research and imaging data all consumes tremendous amount of data and information. I don’t know if I can fully understand the data retention, indexing, search and data management efforts in-play this week at CDC or WHO.

I can only hope they did not skimp in their storage and data retrieval investments over the past few years. I recall an oft quoted concern from the early days of NASA, and one of their astronauts, climbing on top of a rocket – “Would you want to put your life on the top of two million parts, each designed and manufactured by the lowest bidder?”

It seems to me that some in the storage world have become priests and priestesses of voodoo techniques, in producing savings that seem attractive but really do not work over time. Here are my top voodoo claims that I see repeatedly in the area of storage savings, costs, (in CAPEX, OPEX) and everything in-between:

1.   You can save more money than you have ever invested, or will ever invest in storage – We have to be careful in developing business case or payback savings that do not pass the goofy-test. Now I know that the TCO of storage is much more than the TCA (total cost of acquisition), but we have to have reasonable savings in areas such as power, cooling, waste, management etc. A $1M investment that saves a cool (hard) $5M in the first year is indicative of some black magic calculations. Cost benefit analysis has to be believable, with money that is relevant and applicable to your environment.

2.   Operational investments create immediate savings, and the payback will be early next week. People are slow to change. Teams take longer, and entire organizations…… well we measure organization change with solar calendar(s). We may want to put in place new chargeback systems, or provisioning systems, or reclamation systems; but the personnel and operational changes will take much longer than a set of tools (that you are being sold) will take. Savings have to be tempered with the mood or willingness of organizational change. Now the good news is that today’s challenging economic climate is tearing down a bunch of political and attitudinal walls in the storage team and IT organization, in order to find and realize savings. Just be aware that a tool or architecture by itself will generate the savings potential, but people will often be the long lead item in realizing the full savings

3.   There is an old saying “if a tree falls in the forest, and no one is there to hear it, does it make a sound?”. The next voodoo principle has relation to this adage: “if an organization saves a bunch of money with transformation and architectures and operational improvements, but no one is there to measure (and take credit for) the savings, did you really save any money?”. We cannot improve what we cannot measure. Investments to save money without the needed measurement systems (econometrics), will produce voodoo savings. You need transparency and accountability in your costs and saving projections.

4.   There is always a new snake oil salesman in the camp. The seductive allure of quick money, quick savings with unproven technology and process is a dangerous pattern that we are seeing today in the storage world. Lots of new (and potentially viable) architectures, management claims, performance claims certainly do exist, but only time will sift out the real/beneficial solutions from the imposters. If you need valid, tangible, measurable savings in your IT or storage estate now, then you have to consider proven solutions. By proven, I mean referenceable, time-tested and operationally-viable strategies.

• Virtualization (host, storage, desktop)
• Management automation
• Space and resource reclamation (throw away, demote, de-commission, archive)

5.    Performance capabilities that tend to break 1-2 newtonian laws, or natures laws of light, gravity etc.

6.   “Gee, let’s wait until the next disaster hits before we try to quantify all those soft dollars related to the cost of an outage…..”

7.   “You can’t save me people or labor expense! I will just work my team a little hard, that’s all. Bill-bob over there can handle 1, maybe 2 PB more of capacity if he comes in earlier on Sunday…. “

There are more of these voodoo principles, I will have to do a ‘desktop google’ to find more that come from vendors, integrators and clients alike.

• storage resource management
• thin provisioning
• data deduplication
• intelligent archiving

The article goes on to describe how storage capacity can be orphaned and some typical techniques to reclaim capacity were outlined. It is interesting that the topic of storage virtualization was missing as a proven method to reclaim stranded storage. Another way to stop buying storage, is to reduce the amount of data created in the first place (I took some time off earlier in the month, so perhaps I missed the announcement that we have repealed SOX??). Most IT departments do not have control of the data growth appetite, nor of business demands, application demands etc. I have written for years about improving what we already have invested, and thus create a short storage-capital-purchase-holiday. But this is short-term, improving ROA during the current capital and credit crunch.

An interesting view of orphaned storage, or stranded capacity will vary you look at your own data/storage stack:

Storage Capacity Definition

• Changing RAID levels will impact the delta between RAW and Usable
• Array virtualization will reduce the waste between usable and allocated
• Thin provisioning will save space differences between allocated and used
• Deduplication will impact utilized and application content
• Archive will reduce both allocated and utilized volumes
• SRM will help in managing and monitoring all the areas above

The right architecture is needed to right-size the total storage estate. Look at short term and long term options to improve ROA and overall utilization.

Transparency – econometrics is the key to transparency. Using unit cost measurements to explain where the costs are coming from. Cost is so much more than price, and all these costs have to be stacked up, measured and verified in order to see the true cost of goods (COGs). If you need help defining all the types of money, see this short outline of 33 types of costs that HDS uses for definition.

Accountability – users need to account for what they consume. Bad behaviors will continue when no risks or rewards are present. DBA and application owners need to account for capacities that are underutilized, or aging etc. Re-charge or chargeback mechanism (even dummy invoices) are helpful in assigning local costs for resources consumed.

If users see actual usage rates, and the associated costs, there is a new light on asset optimization and the ability to be sensitive to ROA. IT organizations that implement some type of chargeback or monthly dashboard can encourage better storage consumption behaviors. You cannot improve what you cannot measure, so having the right levels of transparency and accountability can enable better insight into the political, organizational and technical aspects of storage ownership costs.

The common message within my work this week (and most weeks) is changing the architecture to allow you to be smaller, faster, and more cost efficient.

Tightening budgets are calling for more creative tactics to identify, measure and reduce costs. Squeezing budgets is like squeezing into tight jeans (bad analogy) but the facts are that we can reconfigure many current infrastructure architectures and re-purpose arrays to be virtualized, enable thin volumes and dynamic tiers. There are measurements taken that can show the potential to be in a storage set of pants 3-4 sizes smaller. Here are 2 quick examples of what was determined to be the result of moving to a virtual, thinned storage platform:

• One client plan was to go from a 1900TB raw storage estate to 1200. That is a 35% impact.
• Another situation storage estate went from 250 TB to 186TB, a 26% reduction.

RAID is not the problem. Yes, I am seeing some clients migrate volumes from RAID 1 to RAID 5 for reclamation, but all things being equal, there are other areas to look at. Even ignoring the data reduction potential (de-duplication, archive, deletion, quotas), there is space reduction. In the graphic below, you can notice that there are 2 areas to address:

• Usable but un-allocated
• Allocated but not used

Keeping the amount of data the same, and the ratio of RAID overhead the same, virtual and thinned storage can attack directly the red and yellow sections of the storage estate. The results:

• Reduced floor space, removing 7-10 frames
• Reduced power
• No need to do a tech refresh, they will just decommission the arrays, even though the capacity is already there and ready to be re-purposed
• Retire the HW and SW maintenance
• Less asset headaches, CMDB management, provisioning and touch effort

Now the other option with this reclaimed capacity is to grow into the recovered space over the next few years or months. I was a little surprised in these 2 clients’ interest in decommissioning the arrays all together and reducing the footprint now. They plan to still squeeze more organic growth out of the existing infrastructure with other efforts (mentioned above) and are really interested in short-term gains by retiring assets right now. Rather than take the reclaimed space for organic growth, they are opting to decommission arrays now and reduce the OPEX costs now. I will be interested to see if they can get into an even smaller pair of pants…..

This just in from my colleagues in Thailand.