Businesses are generating and ingesting an unprecedented volume of data to be analyzed. Needed is a scalable infrastructure that processes and parses extremely high volume in real-time and calculate aggregations and statistics - look at banking trade data, where volumes can exceed billions of messages a day.

Firms are approaching 'the wall' in terms of scalability with relational databases, and must stop imposing relational structure on analytics data and map raw trade data to a data model in memory or low latency, preserve the mapped data to disk, and handle ad-hoc data requests for data analytics.

A scalable solution of this magnitude requires the 'Big Data' paradigm with a NoSQL database.

Four challenges:

• Volume - process more data while retaining historic data trending analysis.
• Variety - deal with more sources -- trade data, messages, web logs, machine logs, geo tags and custom tags. The scope of trade data includes unstructured, semi-structured and multi-structured data.
• Velocity - unprecedented throughput is required as we support and/or produce financial fraud sensors, early warning sensors, etc.
• Vitality - provide a secure, persistent data store, while offering the agility in order to adapt quickly to changing business needs.

Joe discusses these challenges, and introduces NoSQL databases, describing how they are capable of scaling far beyond relational databases while maintaining performance.

He then shares a real-world case study that details the architecture and technologies needed to ingest high-volume data for real-time analytics along with a step-by-step overview to help the DAMA conference participants determine the best products in the flooded Big Data market.