Saturday, September 1, 2007

Miniature data warehousing is now possible


By Harikrishna S. Aravapalli

The moment you hear the phrase "data warehouse", often the first thought or first visualization that comes to mind is that of a huge server infrastructure, with thousands of gigabytes (or even terabytes) of data being churned in, processed, and churned out. This also creates an illusion that unless the business intelligence reporting systems do not connect to these servers and also to the tons of data that they hold, we cannot derive any meaningful operational and analytical data from them.

However, with the rising maturity of virtualization software, the easy availability of the high-capacity mobile storage devices and feature rich mobile phones, it is possible -- with a little innovation -- to create personalized business intelligence and data warehouse systems, thus leading to what I call the "miniaturization of the data warehouse systems". This article discusses how that might happen.

Typical characteristics of Large Data warehouses

Let's briefly discuss the typical characteristics of a large data warehouse (DW). A large DW accesses data from multiple source systems. A large DW has an intermediate staging area. It has an ODS (operational data store) to hold short term data. It has a large database which holds a significantly large amount of historical data and also live data. There are ETL (extract, transform, and load) tools to move the data from one stage to the next stage. Finally there are business intelligence tools and technologies which connect to these large DW systems and present the data in the form of reports to business users.

Maturity of Mobile storage devices and virtualization technologies

Storage devices have matured enough to the point of holding a few gigabytes of data in small mobile storage devices like USB flash drives and even up to the terabyte level in external in USB hard drives. Even smaller USB hard drives (like you'll find in an iPod) can hold 160GB or more.

To further boost the potential usage of these small, mobile storage devices, there is OS virtualization software which can run on these mobile storage devices. These OS virtualized mobile storage devices can then be used to connect to any host PC to utilize the hardware resources, thus converting the OS virtualized mobile storage devices into mobile PCs which can run in a plug-n-play model. There are already vendors which can provide these kinds of systems.

Concept connectivity for creating a miniature data warehouse

Now that we have mobile storage devices of reasonably large capacities (greater than 100GB) coupled with the fact that there is virtualization software readily available, the key question to answer is how can we create a "miniature data warehouse" that is mobile, personalised and yet has all the features of a regular business intelligence and DW applications.