But there is a related topic I’ve been wanting to write about for a while now, and it relates to the trade-off in memory management at it relates to HDP and page size. But before I dive into it, let me take you back in time and (re)introduce some buzzwords and concepts from the past, in this case, the late 1970’s and early 1980’s. The two concepts were: “Access Density” and “Short Stroking”. Access density is a measure of IOPS per capacity of a hard drive and Short Stroking was what a lot of folks were doing to compensate for declining Access Density. The two terms are, or course, still used but rarely outside the domain of DBA’s. Put 10 DBA’s in a room and you’ll know what I mean (I used to be a DBA so I’m kinda allowed to poke them).

The problem at the time was that drive capacities on those 14-inch diameter behemoth drives were increasing, but they couldn’t spin the drives faster than 3200 RPM. Spinning a 14-inch drive reminds me of those circus guys spinning dinner plates on the ends of sticks: the tolerances by today’s standards were pretty bad. The problem was increasing capacity with the same RPM. Sound familiar? That is the Access Density problem.

The answer to this dilemma was “Short Stroking”, meaning just putting less data on the drive (to reduce the IOPS demand). It even included physically placing data in the center of the platters so that the heads wouldn’t have to seek (stroke, hence the term) as far. And you all thought Short Stroking was a description of my golf game? It actually is, but that’s a discussion left for another day…

Enter the new decade. Anyone see a repeat here? It’s been quietly happening again, since the drive folks are stuck at the 9-year old 15,000 RPM drives. Unfortunately, that will not change; there is no 20,000 RPM drive in our future. And while we all want utilization rates to increase from the measly 20%-30% industry average, the fact is that unless we start adopting available technology, it will not, for many applications.

I’ve spoken with many customers that were delighted with their 36GB 15K drives, tolerated their 76GB counterparts, but limit the amount of data stored on their 300GB or larger drives. Meaning if something is not done, utilization rates might actually go down further. In other words, they’re Short Stroking, especially in the database world. Performance still rules, and the drive guys won’t be giving us any relief, and flash drives are still too pricey. So what do we do?

Well, HDP, and Thin Provisioning in general, address the utilization rate problem, I think we’ll all agree on that one. But does it address the Access Density problem? Read on…

Our implementation of HDP lays out data in 42MB stripes (32MB stripes on the HDS AMS2xxx midrange array, that was announced today) in a wide striping scheme that offers as much as a 700% improvement (measured) in workloads (or more!!). Access Density, what’s that? Having a 700% improvement in workload throughput is equivalent to having 105,000 RPM drives (very crude arithmetic and workload assumptions, but you get the point; don’t beat me up on this!). We’re seeing many of our customers adopting HDP only for performance reasons, and not only for thin provisioning reasons. A customer I spoke to last week had a batch job that went from 10-15 minutes to less than 1 minute using HDP wide striping (150,000-225,000 RPM drives?).

And it gets better…

Considering that the major effort in provisioning, especially databases, is to “provision for performance”, now you don’t even have to do that. I’ll defy anyone to manually provision in a way that will outperform the wide striping we do with HDP. So now I’m telling our customers to just throw the data out into a HDP pool and let us manage for performance. For the thin provisioning argument, there are some caveats with some older file systems, but for performance, there are no drawbacks.

So with HDP we have:

1. Thin provisioning – 30%-40% improvement is not uncommon
2. Massive improvements in performance – orders of magnitude
3. Provisioning for performance – What’s that?

Maybe soon we can get all the world’s DBA’s to have a bonfire-party to burn all those old whitepapers they’ve been clutching for 30 years…..I’ll provide the matches…

It’s not that I don’t enjoy the give-and-take with Barry. It’s that Barry is using our announcement last week as an avenue to pose questions that he already knows answers to. That is, feeding the FUD machine that EMC has so ineptly been clinging to for quite a while.

I had to select between 3 choices here - Answer his increasingly inane questions (few of which actually related to our announcement last week), post to his blog equally inane questions about lagging Symmetrix/DMX architecture, or just bring a halt to this silliness. I originally thought I’d select the second option and “pepper” him with a million detailed questions about EMC’s architecture and its limitations but have decided on option three to end this once and for all. I have a “day job” that requires taking care of our customers, working with engineering and development, product management, global services, marketing, and corporate communications. What is also on my plate is blogging. What is not on my plate is having to answer the most detailed of silly questions from a competitor. Barry, as you concluded one of your recent posts to me - “inquiring minds want to know.” What is your day job? And do you really have this much time on your hands?

I will never preclude answering Barry, and look forward to the next exchange. What I will not do is engage in this silly banter in public. I have neither the time, nor the inclination, to do so.

Back to my day job…

In the past 3 days (today included), we’ve fielded questions, had many dozens of conference calls, spoken to the media, financial analysts, and industry analysts. It’s been a fun few days, and the interest is very high, but I would like to emphasize that in fielding these questions and interest, customers top the list as does any analyst representing customers. Somewhere further down on that list are the questions that come from our competition: EMC and IBM. It’s not that there’s no love here, it’s just a basic priority thing (‘scuse me Mr. ReallyBigBank customer, I’ll get back to you tomorrow after I answer EMC’s questions, first).

Barry’s post the other day took exception to our less than prompt answers to his good questions. They are good, and I will answer them here, although I suspect the motivation is pot stirring and information gathering to fuel the expected FUD machines. I’ll ignore his dripping sarcasm and vitriol and get to the basics of his questions. Really, you should all read his post!

But if Barry would allow, and to compensate for the above, I have to have a little fun first. Barry’s post on this started out by saying: “…I hate to be a pest, …” which got me thinking that since this whole HDS announcement started with an anagram, I could anagram “STORAGE ANARCHIST” into “’TIS A STRANGE ROACH”. (Sorry, Barry, but I couldn’t resist!!)

But onto the questions:

• Q: Why didn’t you include the GA date in the announcement? Are you hiding something? A: Yes, we are trying to conceal the GA date and you have no idea how much trouble I’ve gotten into by saying over and over that it’s 4Q this year. Why wasn’t it in the formal announce? I dunno, I don’t write those, but the information is all over the place.
• Q: Are there really NO host requirements? A: I may have misled here. What I’m saying is that there are no additional host requirements since I’ll assume path failover software is pretty standard these days. Initially we’ll support HDLM and perhaps others by GA.
• Q: How does a host know to start sending I/O’s to a different target? A: Path failover that supports HHAM will have owner-paths and non-owner paths after all owner-paths fail then path failover accesses the non-owner paths which are on the secondary controller. I would think you could have guessed that!
• Q: And exactly how does consistency and compliance work when TSM can’t relocate a volume that is being replicated? A: Consistency is maintained by TrueCopy Sync. HAM takes another pre-caution by checking the Quorum disk before a failover to insure that the pair(s) in question haven’t been suspended.
• Q: Oh, and please - you are the undeniable expert in all things IBM mainframe. Are you suggesting that Hyperswap is the solution to tech refresh migrations for CKD devices? A: Nope, I’m not. I’m just suggesting that the functions are similar in the sense that it can swap devices, and as you know, the similarity stops there. Hyperswap is an OS function, to begin with.

Peace…

But I would like to crisply answer a number of the questions that have come up in the last 36 hours and try to recreate the egg from the scrambles.

Firstly, there is no host requirement, and we support all platforms except z/OS. z/OS has Hyperswap which is very similar to AM when you think about it.

So it’s transparent to the apps which is a very good thing. No agents, no restrictions, no impact on performance (other than that imposed by synchronous replication; we’re used to these kinds of things).

You simply define which LUNs you wish to include (much like you would do with any replication product), the array copies the data over and the LUN is either moved (if this is a migration scenario) or kept in a  “mirrored” state, available in case a failover is initiated. So it’s not double the storage or double anything other than what data you wish to move or protect. Again, just like any other replication scenaro. The only additional requirement is that the quorum disk needs to be external to the USP-V.

Keep in mind, this “magic” is done entirely within the embedded software of the USP-V and does not require additional investment other than the quorum disk (keeping in mind that pricing is not finalized).

Availability is Q4, and yes, it has been tested by customers. This announcement is not a slide deck of what sounds cool in the future, nor is it vaporware, this is reality. Sorry guys…

Is AM for everyone? probably not, but the interest I’ve seen from NDA’ing this for a number of months would indicate that interest is a lot higher than some of these blog comments might think (or hope).

There it is, transparent failover from one array to another in the event of loss of access. It’s as simple as that…

Part of the problem is we (HDS) can scale our USP-V to hundreds of TB of internal storage and even to the PB range for our virtualized storage (yes, we do have customers that have scaled to the PB range). Great stuff, but it does elicit the EiaB cry, and for that reason alone, many of our customers have placed arbitrary limits on the amount of storage they’re willing to place under the control of a single array. The answer? “If you could cluster these things we wouldn’t be so concerned!!”

As our announcement today spelled out, we now have storage clustering, meaning there should never be a reason for a customer to lose access to critical data whether that data resides internally to the USP-V, or as external virtualized storage. It’s pretty straightforward, where if access is lost, failover is initiated and the applications never miss a beat. Cache coherency is maintained through local mirroring and an external quorum disk.

It also serves the purpose of being able to non-disruptively move data (internal and virtualized external) between USP-Vs. That means future data migrations will all be non-disruptive.

So, no loss of data access. Ever. And, no outages for any data migrations. Ever. And all of this done with embedded software on existing arrays. Forklifts need not apply.

Well, let’s see if the “blog-as-billboard” idea can work. So we obviously have a big announcement next Wednesday May 27th and although I can’t tell you what it is, I thought I’d try a different approach. Since I’m a big word-gamer, I thought I’d anagram our announcement and see who can come up with an answer. Anybody want to try to figure out what we’re announcing? Then solve this anagram:

REGRADES OUR CLASSY TREAT

We currently have a reseller in the form of Sun Microsystems (or is that Oracle Microsystems?) and they have very much been in the news recently. We (Hitachi) also have an OEM arrangement with HP, and after Dave Donatelli’s departure from EMC to join HP as head of their storage and server business, that created a bunch of news as well. And, quite seriously, we do a fair amount of business through these two partners.

But recently the media, the pundits, the analysts, and the guy at my local coffee shop, all seem to think that this spells doom and gloom for HDS and Hitachi storage in general. The reasoning goes a little like this:

If Oracle consumes Sun, Mr. Ellison will decide to dump HDS as a partner in preference of his highly invested Pillar Data Systems or existing Sun midrange storage. Another “theory” is that Oracle will remain a software-only company and will divest itself of storage altogether. There are many more conspiracy theories here, but the common conclusion is that HDS is toast as far as storage and Sun is concerned.

Then we have the Donatelli story. Decent guy, smart guy, and very capable guy, I’ve met him and like him, but he suddenly (or at least it appears sudden) decides to bolt to HP. OMG!! What’s that going to do to Hitachi storage!! I had to chuckle at Dave Vallente’s blog on this (it’s a very good one and a must read; great insight). But he had a comment (the source of my chuckle) that said: Hitachi must be freaking out a bit– what a month; one OEM gets acquired by a Oracle who just wants to commoditize all hardware and your other OEM just put the dreaded enemy in charge– yikes!

But we were all quite surprised here. Should we be “freaking out”? We weren’t, at least I wasn’t, nor were any of my colleagues here. We were acting just like everyone else was, which was crowding around the water cooler and theorizing what all this meant. We even canceled our “Should We be Freaking Out” meeting.

So let me digress for a moment. Confession: I’m a dyed-in-the-wool BMW guy. I’ve had a few and I can give you a long list of why. There are many other great cars out there, from great car companies, but I’ve just settled in on BMW. Now let’s say that my local BMW dealership (we’ll call him Bob’s BMW to protect the innocent) decides to suddenly become “Bob’s Trabant”. Would I suddenly be driving a Trabant (sorry, with all the consternation going on in the auto industry I picked a brand that is not only defunct, but one we can still make fun of)? If Bob doesn’t want to sell me my next BMW, I’ll go to Julio’s BMW in the next town. Really!!!

Let me also give you some HDS history to back up my claim. Hitachi used to have two distributors in Europe. One was (the current) HDS, the other was Comperex. Someone, at some point, decided which company got to operate in which country. They basically split the continent in half. I visited Comperex once in December, 1999 at their headquarters in Mannheim, Germany. Nice meeting with nice people but then a few weeks later a similar crisis struck!! Suddenly, in February 2000, the announcement came that Comperex would now be a reseller for EMC, not Hitachi. Did Hitachi suddenly lose half of Europe? Hardly. What happened next was HDS started setting up sales offices in hotel rooms in previously Comperex countries (we’re now well beyond the local Holiday Inn’s and occupy legitimate offices). The end result was that Hitachi storage really didn’t skip a beat. Customers that were used to buying Hitachi storage from Comperex just went to HDS and bought the storage from us. It was a scramble, but the transition went smoothly under the circumstances.

The point is that customers buy the storage they want and the cars they want. If they want to buy Hitachi storage, they will. If they want to buy EMC storage, they will. And if they want to buy IBM storage, ditto.

So whatever you read about the Oracle thing or the Donatelli thing, getting “conspiracy” on its impact on Hitachi storage is premature.

We’re doing, and will continue to do, just fine.

Back to the water cooler to catch up on the latest conspiracy rumors. Wonder what’s next…..

But we mostly identify it with aircraft. When bad stuff happens, like USAir’s flight # 1549, it was the black box that helped correlate pilot evidence that the aircraft had an unfortunate “aviary moment” on its way to the Hudson River runway. Fortunately, all survived quite nicely.

But here is a device that hour after hour, day after day, and year after year, is constantly recording data so that when the data is needed, it is available. No one really cares about all of the data it has captured over the years, just that when “specific” data is needed, it is retrievable.

Today’s ever-increasing mountains of data are no different. We’re creating data at rates unfathomable even just a few years ago. On the one hand, we can’t just erase stuff we don’t think we’ll ever need again because, one day after we do, we will (need it). So we’re lulled into thinking we’ll just keep everything forever. That’s not right either, so it’s no wonder “archiving” has been a big focus recently. I think “archive” is an unfortunate term since it implies a one-way street. Sort of like landfill: put it there and eventually cover it up. But when you think about it, it’s really all about the retrieval. Accumulating these mountains of data without easily and quickly being able to retrieve exactly what’s needed, when it’s needed, is a wasted exercise.

So today, HDS (re)announced the capabilities of our Hitachi Content Archive Platform (HCAP), but more importantly, how it has been used by our customers. Why did we do this (other than because it’s a really cool product)? Well, like we do with all of our products, we are constantly interviewing and surveying our customers to discover the good, the bad, and the ugly so we can improve, and when the results come back overwhelmingly positive, as in this case, why not announce it?

Too often, products like this get ignored in the overall datacenter priorities, and they shouldn’t be. Being able to relatively quickly and easily install a “black box” (yes it is, see picture) with your defined policies so that it seamlessly archives data no longer needed and (and this is the REALLY important part) brings it back when it is needed, is not only crucial, but a big time and cost saver. You don’t need to understand the “internal workings”, just that when you need to retrieve the data, it is easily available. But know that one of the great things about HCAP is that all of the archived data is pooled in a way that allows direct retrieval using search commands that bypass the need for the original archiving software. Yes, just simple search.

Other findings in our survey: some companies were able to archive over 80% of their online data, almost half were able to achieve a return on investment (payback) in less than 12 months, 20% ingest more than a million objects per month, and almost all were able to manage their entire HCAP environment with a single administrator. So take a look at the announcement. And if you really want to understand the “internal workings” of HCAP, go no further than your nearest HDS representative or check out the website.

I’m getting a little ticked off over this banter of what company was first with what technology, and Barry Burke, the self-proclaimed storage anarchist, made some comments on our own Christopher Bertrand’s blog that need some serious corrections. I’m not sure where he gets his information, but he should be questioning his sources. The topic here is replication, internal and remote. Let’s take a trip through history to set the record straight…Barry claims that EMC invented it all which is not only false, but he is now hiding behind his proclaimed “legal rulings” to justify his claims, which are also bogus.

The first intelligent function introduced to storage systems was a mainframe capability called Concurrent Copy. CC was GA’d in early 1992 and was based on a patent filed in early 1987. I know this because I filed that first patent. It was based on a technology I (we) called “write intercept” that everyone now wants to refer to by its bovine name of “copy on write”. That’s right, the first CoW patent is 22 years old.

CC essentially introduced intelligent storage subsystems to the world. This CC intellectual property was licensed to Hitachi, Amdahl, and STK and a couple of other companies that no longer exist (in addition to the now non-existent Amdahl and STK!). EMC did not license CC, but did come out with a “compatible” version about 18 months later. What’s interesting is that in the very early EMC documentation on CC, it basically said “see the IBM manuals for details”. Who copied who, here? You were 18 months late to delivering intelligent storage function and you had to refer your customers to your competitor for information?

What came next, over the next 4 years, is a little less crisp and I’ll explain why. Both IBM and EMC were developing remote replication technologies. EMC was developing synchronous SRDF and IBM was developing synchronous PPRC as well as asynchronous XRC. The reason these times were muddy was that early distance replication technology was truly groundbreaking and was having to deal with a lot of new concepts like surviving “rolling disasters” and understanding the underlying reactions of DBMS’s in response to these failures. Both companies called these products “GA” but in fact all 3 products (SRDF from EMC and PPRC and XRC from IBM) were in controlled availability as we were researching and learning in the process. So I’ll give PPRC and SRDF a “toss up” in terms of novelty and availability (actually, no, PPRC had some very specific technology advantages that showed an understanding of data integrity issues that were absent in SRDF, but perhaps I’ll leave that for another blog), but XRC was a clear winner in asynchronous replication and a “first”.

Then came the “bombshell of 1996” when we all woke up one morning to learn that IBM was going to OEM the STK Iceberg. The threat here, to EMC, was that the Iceberg, because of its log structured file system, was able to deliver the first true “snapshot” capability and it was now in the domain of IBM, not just STK which, at the time, was still a relatively minor storage player. IBM now had 2 ways to clone volumes: the just discussed CC, and now Snapshot. IBM marketed Snapshot very heavily, integrated it into CC via the Virtual Concurrent Copy (VCC) interface. EMC’s response to this threat was TimeFinder. TimeFinder, by the way, was again NOT new since both IBM and Hitachi had the same capability since the early 1980’s called “Dual Copy” (now called RAID 1). But EMC did demonstrate that they, too, could clone a volume. Wow. Really. So far, not a not of “newness” here from EMC’s side, eh?

This era was mainly a mainframe IBM/EMC battle. HDS’ interest in this was to stay compatible with the IBM mainframe on both the processor side and the storage side. Thus, this period, especially with replication technology, was all mainframe. But things started changing in the late 1990’s when Hitachi, after licensing the XRC IP from IBM, decided to get aggressive in delivering technology not locked to the IBM mainframe. True Copy Asynchronous (TCA) in 1999, which supported both mainframe AND open systems, was the first step in this new strategy. Wanna bet when SRDF/A was delivered? Yes, 2003! That’s 4 years after HDS’ TCA!! Who is leading the technology here?

Now about Barry’s comments on EMC’s multihop solutions…this, I would argue, was to address the fact that EMC had no distance replication technology to counter TCA. Multihop was nothing but a bunch of complex scripting around SRDF and TimeFinder that, through iterations of suspend/resume, and as many as 7 volume copies (sometimes more), delivered remote replication in the absence of an asynchronous technology. It was basically “bucket brigade by Rube Goldberg”.

Now, Barry also makes reference to certain legal actions and who won what, but again, he is, again, completely uninformed. In addition to being involved in the design and architecture of all of the technology I have just quoted, I was ALSO involved in EVERY legal engagement between IBM and EMC, HDS and EMC, and even a very brief one between HDS and IBM. All of them ranging from defending IP, giving depositions, serving testimony, and actually negotiating the settlement in the last 2 between EMC and HDS, and in no case, at no moment, was there ever an acknowledgement, conclusion, or ruling, that EMC preceded either HDS or IBM in any replication technology. Never! I would love to disclose more detail on these events but they are still embargoed and THAT would get me into a ton of trouble, and in spite of that fact that many of my friends are attorneys, I never, ever, again want to get involved in another legal squabble between a bunch of storage companies.

So, Barry, please be careful with your facts, because they are not even close to the mark. We all get caught up in our marketing campaigns, slogans, and claims. In fact, a while ago at a conference, your EMC speaker even claimed EMC “invented” the term “rolling disaster”. Did you also invent sliced bread?

But anyway, I’m here to refute pretty much everything you claimed or implied in your comments to Christophe’s blog. You guys have excellent marketing and no one will doubt that. You also manage to get the most out of your (somewhat delayed) technology. But pretending that EMC has been this technical dynasty that is inventing and leading all the way is a bit much for me to stomach, I’ve been in this industry for 43 years, I’ve experienced a lot, and I know better.

In the end, I saw the announcement as:
• An announcement that claims parity with HDS, but 2 years later (more in some cases)
• A roadmap for the future with some deliverables coming next calendar year
• Performance, performance, performance. Does that mean EMC are confident that they’re good enough to join the SPC? Oh, I forgot, we’re still talking about slideware here…
• Oh yeah, they saved some provisioning clicks (which is a good thing, we’re all working that angle)

And as my buddy Hu pointed out this morning, that is again true. There is really nothing wrong with the EMC roadmap here, but until it evolves, it is just that: a roadmap. The last major roadmap that I can recall from EMC was the InVista announcement almost 7 years ago! Is it now exempted from grief under a statute of limitations? Perhaps.

But there was little substance in the announcement that I caught. Storage virtualization is clearly aimed at supporting heterogeneous storage. V-Max does not support this, unless you define being able to mix Seagate and HGST drives in the same array as heterogeneous. No substance on thin provisioning advancements. There was some mention that they can do synchronous replication at any distance; we’ve been doing that for years, too.

So the question of the day is why did EMC announce this, and why now? My conclusion is that it helps hold onto their current install base which is at jeopardy every time we install a USP-V at their existing customers. It’s a way of telling them that they’ll catch up to HDS in a year or 2, but they’re trying to catch a moving target. Consistently, they are “catching up with the past”.

I’ll post more on this subject as new details emerge, but today, most of HDS is relieved.