Another point of evidence comes from the body of court cases and new regulations all of which put pressure on companies to look again at their unstructured information with fresh eyes. I’m sure that everyone reading this knows about ENRON or perhaps may even remember Sarbanes-Oxley which is effectively the United States government’s attempt to increase both accountability and transparency in the financial markets. Of course there is also the urban legend that Kenneth Lay was so guilty that he died of a self induced heart attack before his trial could be completed — and if it wasn’t one it sure is now. However, while dramatic and definitely in the news, when compared to the Federal Rules of Civil Procedure (FRCP), SOX is “chump change.” My reason for stating this comes from both the ambiguity and the broad applicability of the FRCP. Applicability includes for any organization’s main parts in addition to any agents, subsidiaries, or affiliates and even those who are overseas. Further it is also applicable to any organization (public companies, private companies, and educational institutions) who can get into Federal court for some reason. The only thing that the FRCP really says is that organizations have to be prepared for a litigation.

• A records retention policy is really required and more importantly adherence to it
• The ability to quickly put items on “litigation hold” is required
• With the records retention policy in effect it is also a good idea to provide evidence that your organization is following the policy through potentially documented audits

In short if you can say to yourself that you’ve documented what you are supposed to do, you are doing it and you can prove it you are in good shape.

Now there has been a lot written about the FRCP in 2006, 2007 and even now, and I’m sure by this time your saying like, what the, you’re not saying anything new here. If you have come to that conclusion you’d be right, however I was using this point to illustrate that we as a culture are now considering how to protect and preserve, from a real archival perspective, ESI. Essentially, we need to look at the paper world as an example of what not to do, learn the lessons and apply them to ESI. I do want to point out something fairly unique here: namely that when companies are forced into looking at things through regulation it can often lead to efficiencies that they had not thought of in the past. For getting the unstructured world in order, I firmly believe that we will need to develop unstructured reporting tools which don’t really exist today. Instead, a mishmash of content management systems has moved into center stage as the thing to fix this critical issue. Well, I think that logical approach is doomed from the beginning. One of the first things to consider, if content management could solve this problem it would already have. The issue that I see with content management systems is that a priori organizations have to think of all possible outcomes for work flow, structure, permissions and hierarchy. Pointedly that logical approach is flat wrong in two ways.

• The world of unstructured information is a very messy one and vast and people largely don’t know or understand the contents or even what they have, so how can an upfront strategy work if scale and scope aren’t really known?
• Building a structure with messy information is really an emergent property of people interacting with one another and their information, not a premeditated action.

I do want to provide an example of where point in time observation can lead to incorrect logical assumptions about premeditation. Specifically, ants gathering food. If one were to look at a fully formed ant trail where the ants were clearly taking food from a source and bringing it back to the nest, one could wrongly assume that there was an intelligent controller “directing” the ants towards getting the food and bringing it back to the nest. This assumption would be completely wrong and assumes the controller had a premeditated thought directing the ants to complete the task. In actuality the ants have some relatively simple programming and social interaction on their side. Essentially we can think of every ant as knowing about food, knowing how to get back home, knowing how to leave scent trails, and knowing how to follow scent trails. When they run into food they combine these simple programs to gather the food and return to home while leaving scent marks. If one adds a randomly distributed set of ants across the field then it is probable that many ants will run across the scent trail, follow it back to the food, and return to the next while strengthening the trail. Repeat this process over many ants and an organized ant trail emerges. So in the natural world this is a case of organization being an emergent property of time and social interaction, and not something thought of a priori.
Another reason why premeditated application of structure fails the giggle test: there is an entire generation of workers (like almost 80 million) entering the workforce and they are trained by Google and others in the use of modern unstructured information reporting tools such as blogs, RSS, Wikis, digg, etc. In essence it is through the social interaction between these people that their structure emerges. So my point is that we as IT professionals today need to start bringing these tools into our companies and teams so that we can start speaking early to these new comers into the work force. I also firmly believe that as we add these tools into our bag of tricks, we will be solving working with unstructured information and remedying the failed experiment of content management.

I’m sure that everyone is aware of the recent events in Myanmar and China over the past several weeks. Well the situation is horrible, truly horrible. When they are combined, in my opinion, it has the potential to approach the scale as the tsunami of several years ago, in terms of the impact of human life. Well like Katrina in the US Hitachi is there providing what it can in the form of relief. For China in the form of heavy moving equipment and cash, and for Myanmar in the form of cash.

For me this is a little more personal as much of family was impacted by Hurricane Katrina. Fortunately we were blessed by no loss of life and the gift of insurance. During that time Hitachi also gave generously to the cause and specifically supplied heavy moving equipment to assist in the reconstruction. While certainly I’m an employee of the company that has done this, I can remember getting a call from my Mom saying that she saw some of Hitachi’s equipment all over the city of New Orleans providing assistance to remove debris, so it is also personal.

I’ve been back to New Orleans just this year and while my parents have moved back into their house there is still a long way to go for New Orleans to become something new, and Hitachi has indeed played a part in doing that. On a local corporate level several HDS personnel were personally involved in Katrina as I imagine Hitachi employees in Sichuan are doing what they can. So in that sense, I am reminded about being proud to work for Hitachi, and I want to wish well those doing this hard good work.
If you are interested in the links for the donations Hitachi has made to the disasters in China and Myanmar please see the following links.

• http://www.hitachi.com/New/cnews/080514a.html
• http://www.hitachi.com/New/cnews/index.html

Regards,

Michael C. Hay

“Within the IT industry, the Wikibon community believes that Hitachi, Ltd. has the most comprehensive and fully implemented corporate green plan in place,” says David Vellante, president and CEO of IT Centrix and co-founder of the Wikibon Project. “Hitachi’s progress on its emission neutral strategy is impressive and genuine. Initiatives such as the collaboration of various Hitachi groups for a new datacenter design in Yokohama underscore the firm’s commitment and are great drivers for change. Within storage the USP V controller re-designs and the implementation of virtualization, thin provisioning and support for external devices that spin down, have helped improve utilization and reduce power consumption by 63% over previous generations, a substantial milestone that sets an example of leadership for the industry.”

Why is it that Hitachi’s offerings are so mature in comparison to the industry? Well, we could go all wavy hands and stuff, or we could draw a natural straight forward conclusion. The Kyoto Protocol was signed in Japan; Hitachi like Toyota is a Japanese company, Japan “gets it” therefore, Hitachi “gets it”. That being said, now I have to prove it. Well for literally over a thousand years there have been well defined Japanese arts which celebrate the Earth. Bonsai, Ikebana, and even some of the early Japanese religions celebrate the Earth spirit. Aspects of the Earth are used in ritual purification for Aikido and Zen. In short, “getting it” has been in the Japanese culture for a really long time. Hitachi is a company that is applying being green in delivery of the products we produce, in our development and manufacturing facilities, and frankly in our office building where less cooling is used for humans. In summary, I think that Hitachi was built green from the beginning and has a natural Japanese legacy to take advantage of.

“With EMC scaling down the Centera unit and the future of Centera unclear, the chance to join Caringo, which understands the potential of CAS, and partner once again with Paul Carpentier was too good of an opportunity to pass up,” said Van Riel. “The need for CAS solutions is ever increasing and I am excited to participate in the next chapter of the technology at Caringo.” (source: Storage News Letter)

It should be noted that Van Riel is one of the key Centera architects representing the “soul” of the technology. This is a big loss to EMC and an even bigger loss to Centera customers. I have to say if I was an EMC customer, I’d be really worried!

• Encryption of ingested objects over HTTPS
• Encryption of replicated objects when using object level TCP/IP based replication between to archives
• A packaging format called AOP (Archive Object Package) which can be utilized by users wanting to backup the archive via NDMP
• Finally, AES encryption of data in-flight and at-rest over the SAN fabric, with the intention of protecting sensitive data on drives finding their way out of the data center

The bottom line is that we are leaps and bounds more secure than the competition. Which brings me to the title of the post.

EMC recently announced “CentraStar (R) 4.0″ including the following somewhat vague descriptions of their feature set.

Well, I don’t know where to start. If you look at the announcement in detail you can see that wow they can now support 750GB drives and up to 25 million objects per drive. Okay HCAP can support up to 2TB LUNs (2.7x more capacity than EMC) and is future proofed for up to 16TB LUNs (nearly 22x more capacity than EMC) in the future. So in essence when I look at what EMC shows then so what. As to the number of objects per drive, I honestly don’t get it. How many per node I ask? I can make some assumptions and let you all know what I think that the number is (100 million), but again when I compare this to HCAP (400 million) again we are 4x their size, but again that’s my estimation, if EMC had provided more detail in their announcement I could be more accurate, but as it stands, Centera doesn’t scale, sorry.

The other point of EMC’s announcement is secure administration. Well wow I guess copying is the best form of flattery, then I feel very flattered. Further, it’s nice to know that a company who purchased RSA chooses to copy Hitachi rather then innovate on their own.

• Pick you favorite operating system and associated scheduler
• Pick your favorite scripting language on that platform — there are many to choose from, I happen to like Python
• Determine the directory that you automatically want to watch for expired files, in our example we’ll call it /fcfs_data/foo
• Well, it just so happens that HCAP keeps a metadata directory called /fcfs_metadata/foo/.directory-metadata/info/expired containing zero byte representations of files that either aren’t under retention or have their retention period completed.  If these representations are deleted then the corresponding files/objects are removed from the system
• Write your script that will be run from the scheduler you selected which will look at /fcfs_metadata/foo/.directory-metadata/info/expired, seek out newly expired files and delete them

With all of that done you’ve got a simple approach to automating deletion of expired objects/files.  I’d also like to suggest that you log the heck out of this script and broadcast it to a centralized logging infrastructure like syslog.

If you’d like to see what this would look like in Python, see below.

Script Disclaimer:

ALL OF THE DESCRIBED SCRIPT IS PROVIDED WITHOUT ANY EXPRESS OR IMPLIED WARRANTY, INCLUDING WITHOUT LIMITATION ANY WARRANTIES THAT IT IS FREE OF DEFECTS, MERCHANTABLE, FIT FOR A PARTICULAR PURPOSE OR NONINFRINGING AND RECEIPIENT WILL NOT SEEK ANY INDEMNIFICATION OR OTHER REMEDY AGAINST HDS OR ARCHIVAS IN CONNECTION WITH THE SCRIPT.  FURTHER NEITHER HDS NOR ARCHIVAS WILL HAVE ANY OBLIGATION TO SUPPORT OR MAINTAIN THE SCRIPT OR THE STEPS DESCRIBED IN THE SCRIPT.

Today Hitachi is announcing quite a bit in the area of file storage platforms. To start with are core updates in our underlying platforms. From the debut of the Essential NAS Platform (ENP), a hardware update on the High Performance NAS Platform, and a software revision in the Content Archive Platform; to the entrance of a new product suite for us: the Hitachi Data Discovery Suite that includes two offerings. I want to provide some amount of detail, but not what we are stressing in the announcements. This will be more about behind the scenes kinds of things.

High Performance NAS Platform (HNAS)

Before I get too far into this I do want to bring up a point close t o my heart which I hear on a regular basis: well isn’t HDS just going to drop Bluearc like you did with NetApp? To start with the relationship with BlueArc is 1000% different. My colleague, Shmuel Shottan CTO of BlueArc, is my personal mentour and good friend; however, that really doesn’t summarize the total relationship with BlueArc. The two companies enjoy a symbiotic relationship which spans mutual respect, trust and a shared vision. You will see that in the intentional alignment between our two companies as well as the fact that HDS has engineers with source code access at the BlueArc site in the UK. This was done because when both companies got down to it we realized that in order to implement our mutual strategies in a timely manner the best thing was to increase the engineering resources in key areas like offloading some portion of the full content indexing process and performing HSM migrations to/from externally attached NAS and active archive devices. Beyond the shared strategy and co-development activities HDS owns an equity stake in BlueArc, something that was not the case with NetApp. So hopefully this tells our user base that our relationship is entirely different and growing in a positive direction. Okay just because I don’t want to hear myself talk and all don’t take my word for it here’s what Shmuel had to say about our partnership.

I consider myself extremely fortunate for having the opportunity to work closely with HDS and with Michael in particular over the last 18 months. Being called a mentor by Michael is a great honor. relationships last only when they are symbiotic and reciprocal. I have indeed received much more than I transmitted. While it seems as if Michael and I are trying to compete for the “most humble” award, my observation is sincere. The relationship between the companies is a true partnership. BlueArc is more than a technology provider or just a NAS component provider to Hitachi. Together we have improved our product offerings and embarked on a roadmap to deliver value through tighter integration. BlueArc has gained much more than just increasing its routes to market. Michael has provided me with enormous and invaluable insight to what Hitachi’s customers actually need. This closed loop review has allowed me to channel the innovations we “dream” at BlueArc towards building products that solve the customer’s needs. - Shmuel Shottan Chief Technology Officer, BlueArc

Essential NAS Platform (ENP)

Well there is a lot to say here, and like HNAS, I’ll be focusing on stuff which is “behind the scenes”. One thing that I did want to relate is the attention paid into making this modular NAS device highly reliable. Unlike some of our competitors, in the modular NAS space, who merely use simple heart-beating mechanisms to make their NAS devices achieve HA, the ENP actually has hardware offloaded HA and support mechanisms, existing below the firmware and independent of the internal kernel. In the unlikely event that one of the nodes goes out to lunch the other one can actually force the takeover by issuing a command over the HA channel killing the second node. This kind of attention to detail and processor separation is not something that you will find on another competitive modular NAS device, by the way if you aren’t getting it I mean NetApp. I cannot stress enough the level of attention to detail across the board especially when it comes to reliability. It is kind of hard to relate this level of attention to detail, because one doesn’t find it valuable until there is an outage and you realize that that Hitachi stuff didn’t go down. So it is kind of one of those things that everyone needs but is hopefully rarely used, I guess in that sense it is kind of like life insurance. There are after all two main types of life insurance whole life and term life. Generally whole life is an absolute guarantee potentially with a lot of bells and whistles. Where as term is generally less expensive with less bells and whistles. The ENP is rather like term life, it gets the job done it is reliable and it is less expensive than the competition in the same market segment.

Content Archive Platform (HCAP)

As previously mentioned HCAP is a really strong product capable of solving many use cases, like archiving and Web 2.0 storage, in a single bound — Superman reference intended. Not only is the product rock solid, but the engineering team that makes HCAP is second to none. So my behind the scenes point here is really a shout out to the engineering team. In fact one of them, Jack Orenstein, is already engaged in a speaking spot at PGCon since HCAP makes use of patented technology on top of Postgres. One other topic I want to point to is the reliability of the system, pointing back again to Hitachi’s attention on extreme reliability. Soon we will be publishing a white paper on the patent pending technology within HCAP making it 500x more reliable than competitive products. This work was through the development of a mathematical algorithm to right place the data for extreme reliability. When coupled to SAIN the combination of the HCA software we are getting the best of both worlds serious protection from Hitachi backed ASIC driven RAID plus protections from the software stack. (Note that when the paper is available I’ll do a deeper dive on it. Further, this technology has been deployed for nearly a year now.)

Data Discovery Suite (HDDS)

This is a net new offering by Hitachi, breaking new ground for us. Backing the development effort is Hitachi Software Engineering corporation working in collaboration with Hitachi Data Systems — code speak for HDS being embedded as an integral member of the engineering activities. There are a lot of things contained within this product and I’m quite personally proud of it. The most striking and important thing is the focus on usability for non-IT users. We really did spend a lot of time with HR personnel, Intellectual Property/Patent specialists, corporate lawyers, employment attorneys and IT personnel to understand the challenges in this space. The usability is literally pushed all the way down to the user’s desktop via a Microsoft Vista Desktop gadget. In essence a non-IT worker can span a search across multiple NAS and archive devices without being aware of that fact. Further we looked at common metaphors which these users could grok to assist them with file recovery: shopping carts. Essentially in the Data Discovery Suite, there is a concept called a collection. Users may add many files into this collection, and when they are ready download the entire collection as a ZIP package that includes all of the files added into the collection regardless of the location. I can go on and on here, but I’ll reserve things for a future post since this is getting a little bit long.

Hopefully by this point I’ve piqued your interest and you are thinking: okay Michael what’s your point.  The point that I’m trying to drive to is debunking the criticism about why Hitachi has discrete products in a variety of areas.  Okay, let me go back to a part of the first sentence: use the right tool for the right job.  I think that some of our very worthy adversaries get this point like EMC and IBM.  They have discrete tools that solve specific problems, and if you look at IBM they even have many different file systems, some in their OEM products via NetApp, one called GPFS, another called JFS, etc.  Each different tool solves a different problem for IBM.  With EMC again they have different products for different areas, for high scale enterprise class workloads they have the DMX/Symmetrix and of course for other markets they have the Clariion.  Hitachi has a similar product segmentation with our enterprise class rockstar, USP-V(M), our highly reliable modular, AMS, the USP-V-like High Performance NAS platform, ultra-scale active archive, Hitachi Content Archive Platform (HCAP), etc.  You see, early on we recognized that to optimally solve problems different tools are required.  Hitachi’s sales force has long dealt with that approach, we just don’t talk about it.  After all like all of the other storage vendors, we’ve offered two major block storage architectures and have ingrained in our corporate culture how to attack the market with these two products.  We are doing a similar thing with our file storage platforms now with HNAS, HCAP, and our NAS-Blades, different tools for different jobs.  Early on we recognized that if you really needed to store data for long periods of time, a simple file system was insufficient you must have a different product so we went and purchased Archivas last year.  For highly performant NAS there are a whole set of reasons why we work with the company that we have an equity stake in, BlueArc, to resolve really high scale file I/O types of problems.  You see we recognized that if one wanted to consolidate many NetApp filers onto one platform, the traditional bus architecture would simply not do, a different tool was required.
Okay, hopefully you are following my logic different tools and different products are required to solve different problems.  Also I hope that you are getting that I’ve mentioned IBM, EMC, and Hitachi as recognizing this to be the case.  While mentioned, I’ve not stated that NetApp yet understands this.  They are acting like their hammer, OnTap, can nail anything in, screws, nails, thumb tacks, bolts, hooks, etc.  NetApp’s name for this approach is “Unified Storage”.  And they literally are putting everything into their hammer, storage optimization, snapshots, block storage, clustered global namespace (Spinnaker ain’t no global file system, it is a clustered global namespace.), replication, FC-block storage, iSCSI-block storage.   Wait a minute those last two items in most other companies that offer storage products represent at least a different product altogether.  A widely known secret is that NetApp added iSCSI to their filers because Microsoft removed support for running Exchange and SQL server on CIFS shares, and almost as an afterthought, Fibre Channel was added allowing NetApp to dip their toes into the block storage market.  To me this confirms NetApp consistently targets their hammer, OnTap, at any market even remotely within their aim.

So, I’ll end with, do you want to buy a storage device from a vendor who wants to nail in your screws, or a vendor that uses the right tool in the right market?

On Linux the desktop search engine is Beagle, a C#/Mono port of Lucene, which is really closely tied to the GNOME desktop environment. While it is possible to set Beagle up to repeatedly scan the file system, a more efficient configuration approach is to work with a Linux kernel service called INOTIFY. This service is set up to generate an event stream telling the search engine about added, deleted or modified files. This allows the search engine to target the indexing process only at those file system objects that need to be added to the index, removed from the index or updated in the index. This means that the system is effectively event driven and not set up to regularly poll the file system. Generally speaking for a lot of different problem domains having an event driven approach is superior to a pure polling approach.

The next target on the list is Apple’s Spotlight, which like Beagle integrates to something analogous to INOTIFY: FSEvents. This is an internal only interface maintained by the kernel that allows various user space applications to subscribe to file change events generated by the kernel. Like INOTIFY it removes the need for having to repeatedly poll the file system for modified files. Further history shows that some of this technology originated in BeOS sometime in the past; an article at ArsTechnica provides a great history of core file system features in OS X.

The final search service to review is the Windows Desktop Search (WDS) which really came into the “spotlight,” pun intended as always, for Vista. It was kind of always there, but not as important until the birth of Vista. Like all of the other search engines they have implemented a similar strategy of looking at a journal or event stream to construct an event stream to improve the efficiency of the full content index process. In the case of WDS, it relies on the NTFS USN Journal which keeps track of the added, deleted and changed files so that the search/indexing infrastructure can be more efficient.

Okay with all of that said and the recognition that enterprise class search engines like FAST, Autonomy, and Google are all out there one has to ask why there aren’t a lot of NAS or file storage devices out there with some journal or eventing mechanism that is capable of improving the efficiency of the full content indexing process? Well I think that one of the reasons is that Hitachi has patent pending technology in this area and sports an implementation in HCAP. Due to the previous articles on this topic I’m sure that you are aware of the capabilities of the system being a Web 2.0 storage grid architecture, and part of the capabilities include an eventing mechanism that runs in parallel on each of the storage nodes feeding the full content index a stream of events related to added, deleted and changed files. This allows the system to implement an approach analogous to the ones described above leading to a more efficient indexing process. Further we view the full content indexing facilities within HCAP as a component of the storage infrastructure. Since that is the case, we’ve acquired rights to the query API for this component so that interested companies, parties and application vendors can take advantage of the full content index to solve new customer problems and implement novel use cases. Finally, since Hitachi has patent pending technology in this area for file storage devices, I would certainly not be surprised to hear that we were applying it to more of our platforms.

• Petabyte scaling under a single system image – yes
• Fewer points of management – yes
• Ingestion of 10s of thousands of objects/second – yes
• Rest style protocol for access – yes
• Implementation of capacity optimization features – yes
• Implementation of value added services (capacity balancing, automated node management/control, garbage collection, authenticity checking etc.) – yes
• Usage of commodity components and/or RAID backed storage and ability to sell software independently of hardware – yes
• High performance media streaming – yes
• Local and wide area content distribution/replication – yes
• Low latency rich media streaming – partially

These are merely a few requirements I wanted to relate, but the point is that we can and do solve these problems for our customers now.  We have live systems that scale into the hundreds of terabytes, meaning that like Amazon we have a mature platform: quite pointedly EMC is just now approaching their science experiment stage with both Maui and Hulk.

As to what this orderable product is that can do all of this, drum roll please: it is the software running at the core of the Hitachi Content Archive Platform, internally we code name the platform “Prime” (as in Optimus Prime, or Transformers which are “More than meets the eye”).  The talented engineers behind the HCAP and Prime have been hard at work carefully taking the pulse of the industry, and have made something that very much mirrors what could back an online storage service like S3.  This very novel system has seen significant improvements such as core capabilities that make it ultra reliable, protect data privacy with encryption, utilization of various capacity optimization features (e.g. single instance storage, etc.), and finally being as the system is web based object storage at its core it can be highly customized to meet user requirements or deployed out of the box.

I do want to provide some level of detail on peta-scaling of the platform.  As of HCA V2, Hitachi deploys what we call SAIN (SAN + Array of Independent Nodes) disaggregating the storage from the nodes, meaning that storage and front end processing nodes scale independently.  Specifically this means that each node can sport up to 64 LUs at 2TB each and also includes all of the goodness you’d expect from a SAN attached system such as multi-pathing, encryption of data in-flight/at-rest, swapping of the LUs between nodes, and proven RAID backed full featured Hitachi storage all leading to maximum reliability, performance and efficient scaling.  When storage capacity scaling is coupled to software/node scale out – we’ve tested node scaling to 80 nodes and have no architected limit – a true peta-scale system emerges of over 10PB realizable today – actually the total addressable capacity of the system is 80 nodes x 16TB/LU x 64LU/node = 81920TB but hey whose counting. However, if users want to perform their own hardware procurement, we can and do sell the software apart from the Hitachi storage and nodes.  But, due to the fact that the maximum number of hard drives an x86 class system can hold is 48, it means that to create a similar 10PB scale system would require 1.3 times the number of nodes or 107 nodes – note as of today I’m only aware of SUN’s Thumper that can carry a maximum of 48 hard drives, and I’m assuming each drive can be 1TB of capacity if there is another DAS system than can have more then let me know.  So while it is possible and we do sell software independently of the nodes, at really large capacities the scaling out based on a DAS architecture starts to make a lot less sense then what Hitachi has with SAIN – and yes the pun is intended for SAIN, because those that do not sport a SAIN architecture are inSAIN.