Background
The challenge was to build a real-time dashboard and reporting solution that combined enterprise and external data including consumer and social media data in order to present a unified insight into an enterprise.
Solution
Data is pulled off of third party servers, blogs, discussion boards, social media sites, etc. This is achieved using APIs and crawling frameworks. The data collected therein is persisted to stores that support rapid retrieval.
Pre-processed data is transformed into a data warehouse for fast and flexible evaluation.
Expressive and user-friendly querying languages are used to create interesting mash- ups from data in warehouses. They constitute the primary mode of communication between the presentation and processing layers.
Volumes
Sales and external data running potentially into terabytes of data with multi-year retention for batch queries.
Technologies
Angular JS + D3 JS, Phonegap for iOS, Mondrian over Postgres, Pentaho Kettle
