Issues and Considerations

Big data technologies describe a new generation of technologies and architectures, designed to economically extract value from very large volumes of a wide variety of data, by enabling high-velocity capture, discovery, and/or analysis.  Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesnt fit the strictures of your database architectures.

Data has high rate of input,  large quantity of elements/events, Media streaming, Mobile devices, Financial streams and Traffic monitoring.

• Includes structured, semi, and unstructured •  Necessitates new data model and file formats •  Involves, real-time, analytic, and search data.
• Involves data maintenance functions (e.g. purging, etc.)
• The average investment firm with fewer than 1,000 employees has 3.8 petabytes of data stored, experiences a data growth rate of 40 percent per year, and stores structured, semi-structured, and unstructured data.
• Typically involves data distribution, movement, etc., across multiple data centers and geographies.
• Can be on-premise, cloud, or hybrid hence getting a big data technology that provides two out of three can be challenging.

Approach

To gain value from this data, you must choose an alternative way to process it. Datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze. All definitions have one thing in common: new technology is needed for big data.

• Real-time – transactional, online, streaming, low latency data
• Analytic – aggregated data from real-time feeds or other sources; many times batch in nature
• Search – supporting data, both external and internal, used for locating desired information and/or objects (e.g. products, documents, etc.
• Research done by McKinsey & Company shows the eye-opening, 10-year category growth rate differences between businesses that smartly use their big data and those that do not.

Value Proposition Characteristics

• Velocity – includes the speed at which data comes in, and the number of events/elements being stored
• Variety – involves structured, semi-structured, unstructured data
• Volume – can equate to TB-PB’s of data
• Complexity – typically entails the difficulty distributing the data (e.g. multi-data centers, cloud, etc.) and managing the data traffic/movement (ETL, Migrations, etc.)

Offering