CISSE Advanced Level
Big Data Management cheat sheet
Every chapter's key points, formulas and definitions on one page, for the last week before the exam. Each heading links to the chapter that explains it.
Introduction to big data
- Big data is data whose volume, velocity or variety is beyond what a conventional database on a single server can store and process economically, so it needs distributed storage and parallel processing.
- The five V's are Volume, Velocity, Variety, Veracity and Value; the first three describe the data, veracity describes how far it can be trusted, and value is the reason for collecting it at all.
- Data comes in three structural forms: structured (fixed format, such as a table), semi-structured (self-describing, such as JSON or XML) and unstructured (text, images, audio and video).
- Cheap storage, commodity hardware, mobile money, smartphones, sensors, social media, cloud computing and open-source software are the drivers that made big data both possible and necessary.
- The main challenges are storage and processing at scale, data quality, security and privacy, integration and skills; each one is also an opportunity for an organisation that manages it well.
Key terms
- Volume
- The quantity of data generated and stored, measured in terabytes, petabytes and beyond. Volume is the reason data must be spread across many machines rather than kept on one disk.
- Velocity
- The speed at which data is generated and must be processed. Card payments, mobile money transfers and sensor readings arrive continuously and often need a response in seconds, not at the end of the month.
- Variety
- The range of forms and sources the data comes from: database tables, log files, emails, JSON from mobile apps, images, audio, video and GPS positions from smartphones, often all feeding the same analysis.
- Veracity
- The trustworthiness of the data: how accurate, complete and consistent it is. Big data is often noisy, with duplicates, missing fields and conflicting records, so its quality must be measured before it is used.
- Value
- The business benefit extracted from the data through analysis, such as insight discovered, patterns recognised and decisions improved. Data that is stored but never turned into value is only a cost.
- Distributed storage
- Splitting data into pieces and storing them across many machines, with copies kept so that a failed machine loses nothing. HDFS is the classic example, covered in the chapter on the Hadoop ecosystem.
- Parallel processing
- Dividing one job into many tasks that run at the same time on different machines, then combining the partial results. MapReduce and Spark both work this way.
- Data locality
- Moving the computation to the machine that holds the data rather than moving large data across the network to the computation.
- Batch processing
- Processing a large, bounded set of data in one run, such as a nightly job that recalculates every customer's credit score. High throughput, but results are hours old.
- Stream processing
- Processing each record, or a small window of records, as it arrives, such as flagging a suspicious transfer within seconds. Low latency, covered with velocity management in the chapter on big data mining and warehousing.
- NoSQL databases
- Non-relational stores (key-value, document, wide-column and graph) that relax the fixed schema and scale out across a cluster.
- Schema-on-read
- Storing raw data first and applying a structure only when it is read for a particular analysis, the opposite of a relational database's schema-on-write.
- In-memory processing
- Keeping working data in RAM rather than on disk so that repeated calculations and interactive queries run much faster, the approach Spark takes.
- Data mining and machine learning
- Techniques that search large datasets for patterns, groups, associations and predictive relationships, such as fraud scoring or market-basket analysis.
The rest of the sheet covers
- Big data architectures
- Big data acquisition, cleaning and storage
- Big data mining and warehousing
- Big data and the Hadoop ecosystem
- Pattern mining over big data
- Big data processing pipelines
- Big data design
- In-memory data management
- Distributed computing
- Big data policy frameworks
- Deployment of big data
- Exam technique and revision
- Glossary