CISSE Advanced Level
Big Data Management flashcards
Read the question, say the answer to yourself, then turn the card to check. The first 10 of 61 cards are free; the rest come with the subject.
Card 1 of 61
Jump to a card
Complete Flashcard Reference
Review every card in this deck. Open any card to reveal its answer. The first 10 are free; the other 51 come with the subject.
What are the five V's of big data?
Volume: the quantity of data. Velocity: the speed at which it is generated and must be processed. Variety: the range of forms and sources. Veracity: how far it can be trusted. Value: the business benefit extracted through analysis.
What are the three layers of the Lambda architecture?
Batch layer: keeps the immutable master dataset and precomputes accurate batch views. Speed layer: processes recent data as it arrives into real-time views. Serving layer: indexes the views for low-latency queries, and a query merges batch and real-time views.
Distinguish ETL from ELT.
ETL transforms data in a staging area before loading the cleaned result into the target. ELT loads raw data into a scalable platform first and transforms it there, keeping the raw data for later use.
Name the five OLAP operations.
Roll-up: summarise to a higher level. Drill-down: move to finer detail. Slice: fix one dimension at one value. Dice: select ranges on two or more dimensions. Pivot: rotate rows and columns.
What do the NameNode and DataNodes do in HDFS?
NameNode: the master; holds the namespace and the map of files to blocks and blocks to DataNodes, but no file data. DataNodes: store the blocks, serve reads and writes, and send heartbeats and block reports to the NameNode.
What is the Apriori property and how is it used?
Every subset of a frequent itemset must be frequent, so any superset of an infrequent itemset is infrequent. Apriori prunes candidates that have an infrequent subset without counting them.
What limits the throughput of a pipelined data pipeline, and what does pipelining improve?
Throughput is limited by the slowest stage, the bottleneck: once the pipeline is full, items leave no faster than that stage releases them. Pipelining improves throughput, not latency: each item still takes as long to pass through. Example: stages of 4, 6 and 3 minutes give one batch every 6 minutes, so 10 batches take 13 + 9 x 6 = 67 minutes instead of 130.
What are the four families of NoSQL data store, with an example of each?
Key-value: a value fetched by its key (Redis, DynamoDB). Document: self-describing JSON or BSON documents in collections (MongoDB, CouchDB). Wide-column: rows by key with columns grouped in column families (HBase, Cassandra). Graph: nodes and relationships with properties (Neo4j).
How does a column store differ from a row store, and which workload suits each?
A row store keeps each record's values together: suited to transactions (OLTP) that insert, update or read whole records. A column store keeps each column's values together: suited to analytics (OLAP) that scan a few columns over many rows, reading only those columns and compressing them well. Example: a 2-column query on a 20-column table reads one tenth of the data in a column store.
What are the steps in estimating the storage nodes a Hadoop cluster needs?
Raw data per day x retention days = data held. x replication factor (3 by default in HDFS). x (1 + working space, typically 20 to 30%). / target disk utilisation (about 70 to 80%). / usable disk per node, rounded up, plus a spare. Example: 500 GB x 180 = 90 TB; x 3 = 270; x 1.25 = 337.5; / 0.7 = 482.1; / 48 TB = 10.04, so 11 nodes plus a spare.
Distinguish structured, semi-structured and unstructured data, with an example of each.
This answer comes with Big Data Management. Buy the subject
Name four drivers of big data.
This answer comes with Big Data Management. Buy the subject
How does the Kappa architecture differ from Lambda?
This answer comes with Big Data Management. Buy the subject
Distinguish a star schema from a snowflake schema.
This answer comes with Big Data Management. Buy the subject
Distinguish a data warehouse from a data lake.
This answer comes with Big Data Management. Buy the subject
What is data enrichment?
This answer comes with Big Data Management. Buy the subject
Name the dimensions of data quality. Which one checks the format?
This answer comes with Big Data Management. Buy the subject
What is data lineage and why does it matter for audit?
This answer comes with Big Data Management. Buy the subject
What is velocity management? Name three of its techniques.
This answer comes with Big Data Management. Buy the subject
Distinguish a tumbling window from a sliding window.
This answer comes with Big Data Management. Buy the subject
How many blocks and replicas does a 1,000 MB file use in HDFS with 128 MB blocks and replication factor 3?
This answer comes with Big Data Management. Buy the subject
How does HDFS place three replicas under its default rack-aware policy?
This answer comes with Big Data Management. Buy the subject
List the phases of a MapReduce job in order.
This answer comes with Big Data Management. Buy the subject
What are the main components of YARN?
This answer comes with Big Data Management. Buy the subject
How do resilient distributed datasets make Spark fault tolerant?
This answer comes with Big Data Management. Buy the subject
What does lift measure, and why is it needed alongside confidence?
This answer comes with Big Data Management. Buy the subject
How does FP-growth find frequent itemsets without candidate generation?
This answer comes with Big Data Management. Buy the subject
Distinguish sequential pattern mining from temporal data mining.
This answer comes with Big Data Management. Buy the subject
Give the effects of poor data partitioning on a distributed processing job, and two remedies.
This answer comes with Big Data Management. Buy the subject
Name the three multi-tenancy schemes for data and the trade-off between them.
This answer comes with Big Data Management. Buy the subject
Hive is listed with NoSQL data stores. What is it really, and how does it differ from HBase?
This answer comes with Big Data Management. Buy the subject
What is the difference between a Hive managed table and an external table?
This answer comes with Big Data Management. Buy the subject
In pymongo, why must a document be inserted before checking that a new database exists?
This answer comes with Big Data Management. Buy the subject
What is late materialisation (late reconstruction) in a column store?
This answer comes with Big Data Management. Buy the subject
A 12-row county column with four distinct values is stored as fixed 8-byte strings. What does dictionary encoding with bit-packed codes reduce it to?
This answer comes with Big Data Management. Buy the subject
Why does run-length encoding depend on sort order?
This answer comes with Big Data Management. Buy the subject
What are the rules of a Lamport logical clock, and what is its limitation?
This answer comes with Big Data Management. Buy the subject
How do you tell from vector clocks whether two events are ordered or concurrent?
This answer comes with Big Data Management. Buy the subject
What does the CAP theorem say, and what quorum rule gives consistent reads in a store such as Cassandra?
This answer comes with Big Data Management. Buy the subject
Compare the message cost of distributed mutual exclusion algorithms for one entry to the critical section.
This answer comes with Big Data Management. Buy the subject
Summarise the principles of data protection in section 25 of the Data Protection Act, 2019.
This answer comes with Big Data Management. Buy the subject
What do sections 18, 26, 30, 31, 41 and 43 of the Data Protection Act, 2019 cover?
This answer comes with Big Data Management. Buy the subject
Distinguish anonymisation from pseudonymisation, and define k-anonymity.
This answer comes with Big Data Management. Buy the subject
Differentiate distributed file storage from object storage in a big data environment.
This answer comes with Big Data Management. Buy the subject
Name the cloud service models and deployment models, with a big data example of each service model.
This answer comes with Big Data Management. Buy the subject
How do you decide between an on-premise cluster and pay-per-hour cloud capacity on cost?
This answer comes with Big Data Management. Buy the subject
What are the stages of the big data life cycle, and what does a system's architecture decide about them?
This answer comes with Big Data Management. Buy the subject
How does big data management differ from traditional data management in the way it scales and handles machine failure?
This answer comes with Big Data Management. Buy the subject
Distinguish shared-memory, shared-disk and shared-nothing parallel database architectures. Which one did big data platforms adopt?
This answer comes with Big Data Management. Buy the subject
What is a Bloom filter, and what kind of error can it make?
This answer comes with Big Data Management. Buy the subject
What are the six phases of CRISP-DM, and why is it described as iterative?
This answer comes with Big Data Management. Buy the subject
As techniques of volume management, how does partitioning differ from sharding?
This answer comes with Big Data Management. Buy the subject
How is the support of an itemset counted when each item in a transaction carries only a probability of being present?
This answer comes with Big Data Management. Buy the subject
What does Amdahl's law say about moving a job that is 90% parallel onto 10 nodes?
This answer comes with Big Data Management. Buy the subject
Distinguish at-most-once, at-least-once and exactly-once delivery in a streaming pipeline.
This answer comes with Big Data Management. Buy the subject
Why is a raw timestamp a poor HBase row key for sensor data, and how is the key designed instead?
This answer comes with Big Data Management. Buy the subject
Main memory loses its contents when power fails. How does an in-memory database protect committed data?
This answer comes with Big Data Management. Buy the subject
Under the Data Protection Act, 2019, distinguish a data subject, a data controller and a data processor.
This answer comes with Big Data Management. Buy the subject
How can a credit model penalise a region or group even when the protected attribute is excluded from it?
This answer comes with Big Data Management. Buy the subject
Compare dedicated and shared resources for a big data platform, with a typical use of each.
This answer comes with Big Data Management. Buy the subject
Why is licence price the wrong basis for choosing between open-source and proprietary big data software?
This answer comes with Big Data Management. Buy the subject
How to revise with these cards
- Answer out loud or on paper before you turn the card: recalling is what fixes it.
- Go through the deck in short sessions of ten to fifteen minutes, every day.
- Shuffle once you know the order, so you are not answering from memory of the sequence.
- When a card catches you out, read the chapter it links to, then try the topic in a past paper.