Big Data: Why Traditional Data warehouses fail?

Over the years, have been involved with few of the data warehousing efforts.   As a concept, I believe that having a functional and active data ware house is essential for an organization. Data warehouses facilitate easy analysis and help analysts in gathering insights about the business.  

But my practical experiences suggest that the reality is far from the expectations. Many of the data warehousing initiatives end up as a high cost, long gestation projects with questionable end results.   I have spoken to few of my associates who are involved in the area and it appears that quite a few of them share my view.

When I query the users and intended users of the data warehouses, I hear issues like:

• The system is inflexible and is not able to quickly adapt to changing business needs.
•  By the time, the changes get implemented on the system, the analytical need for which the changes were introduced is no longer relevant.
• The implementors of the datawarehouse are always looking for the requirements for which the model can be defined and implemented. But the fact is we are in no position to foresee all the requirements upfront and want the system to be flexible to the changing needs.
• Most of the data ware house systems I use are legacy systems which are so complex and so unmanageable to work with. I am better off build in point solutions which will address my specific need quickly.
• The latency of system is high.  There are cases where we need to do quick analysis, but we find that it takes a long time before the data is available in the warehouse for analysis. 

It is not that, the implementors of the system do not have their own peeves.  Some of them go like this:

• We are building systems which are optimized to provide answers.  Unless the users tell us what are the kind of answers they intend to gather from the system, how can I model a system.
• By definition, Datawarehousing and OLAPs are constructed to be read intensive and for sparse writing.   The way you accomplish this is by ensuring batch loading.  Data Latency is inherent in such concepts and data warehouses are not intended for real time or pseudo real time analytics.
• User understimate the efforts and costs to ensure data quality and integrity.  It takes significant time, effort and cost to ensure the integrity of the data stored in the ware houses.  Everyone is underestimating the impact of having "Dirty Data" in the warehouse.
• Everyone should understand that at a very fundamental level, schemas are static and predefined.   Everything is built over this assumption. 
• Any changes that would impact a change in schema and storage will have a cascading impact and implementing such changes will take time.
• The number of people and groups we need to interact with is mind boggling.  We need to simultaneously work with, Business owners,  the data owners,  the analysts.  While we are talking to these folks, they are also working in parallel to develop point solutions which address their specific needs.  There is always conflict of priorities.

Looking at it objectively, I find that both sides of the argument are rooted in logic and deserves merit.  But what is more obvious is that these arguments offers a good insight on what are the problems which affect such initiatives and why many of the ware housing projects fail.  


But the mentioned issues also provides an opportunity to start looking at building a system which would bridge this growing gap between users and implementors of the system.