It's All About Data Classification and Searching

1. It's All About Data Classification and Searching I don't know if this has been discussed elsewhere but I felt like I had an epiphany so there... The way I see it, in a decade or two the most important technology regarding data will be Data classification and search technologies. Consider this: At the moment, all the rage is archiving and storage tiers. The reason is that it simply is too expensive to buy the fastest disks, and even if you do buy them they're smaller than the slower-spinning drives. For more information, visit: http://klassify.in/ Imagine if speed and size were not issues. I know that's a big assumption but let's play along for a second... (let's just say that there are plenty of revolutionary advances in the storage space coming our way within, say, 10-20 years, that will make this concept not seem that far-fetched). Nobody would really care any longer about storage tiers or archiving. Backups would simply consist of extra copies of everything, to be kept forever if needed, and replicated to multiple locations (this is already happening, it's just expensive, so it's not common). Indeed, everyone would just leave all kinds of data accumulate and scrubbing would not be quite as frequent as it is now. Multiple storage islands would also be clustered seamlessly so they present a single, coherent space, compounding the problem further. Within such a chaotic architecture, the only real problems are data classification and mining. I.e. figuring out what you have and actually getting at it. The where it is is not quite such an issue - nobody cares, as long as they can get to it in a timely fashion. I can tell that OS designers are catching on. Microsoft, of all companies, wanted a next- gen filesystem for Vista/Longhorn, that would really be SQL on top of NTFS, with files stored as BLOBs. It got delayed so we didn't get it, but they're saying it should be out in a few years (there were issues with scalability and speed). Let's forget about the Microsoft-specific implementation and just think about the concept instead (I'd use something like a decent database on raw disk and not NTFS, for instance). No more real file structure as we know it - it's just a huge database occupying the entire drive. Think of the advantages: Far more resilient to failures Proper rollbacks in case of problems, and easy rebuilding using redo logs if need be Replication via log shipping

2. Amazing indexing Easy expandability The potential for great performance, if done right Lots of tuning options (maybe too many for some). With such a technology, you need a lot more metadata for each file so you can present it in different ways and also search for it efficiently. Let's consider a simple text document - you're trying to sell some storage, so you write a proposal for a new client. You could have metadata on: Author Filename Client name Type of document - proposal Project name Excerpt Salesperson's name Solution keywords, such as EMC DMX with McData (sorry, Brocade) switches Document revision (possible automatically generated) ... and so on. A lot of these fields already are to be found in the properties of any MS Word document. The database would index the metadata at the very least, when the file is created, and any time the metadata changes. Searches would be possible based on any of the fields. Then, a virtual directory structure could be created: Create a virtual directory with all files pertaining to that specific client (most common way people would organize it) Show all the material for this specific project Show all proposals that have to do with this salesperson ... and so on. Virtual folders exist now for Mac OSX (can be created after a Spotlight search), Vista (saved searches) and even Gnome 2.14, but the underlying engine is simply not as powerful as what I just described. Normal searches are used, and metadata is not that extensive for most files anyway (mp3 files being an exception since metadata creation is almost forced when you rip a CD).

3. It should be obvious by now that to enable this kind of functionality properly you need really good ways of classifying and indexing your data and actually create all the metadata that needs to be there, as automatically as possible. Future software will probably force you to create the metadata in some way, of course. Existing software that does this Data classification is fairly poor, in my opinion. Please correct me if I'm wrong. The other piece that needs to be there is extremely robust search and indexing capabilities. Some of that technology is there (google desktop and its ilk) but natural language search has to be - well, natural, but unambiguous at the same time.