Part A: Foundations
Introduction to big data
Key takeaways
- Big data is data whose volume, velocity or variety is beyond what a conventional database on a single server can store and process economically, so it needs distributed storage and parallel processing.
- The five V's are Volume, Velocity, Variety, Veracity and Value; the first three describe the data, veracity describes how far it can be trusted, and value is the reason for collecting it at all.
- Data comes in three structural forms: structured (fixed format, such as a table), semi-structured (self-describing, such as JSON or XML) and unstructured (text, images, audio and video).
- Cheap storage, commodity hardware, mobile money, smartphones, sensors, social media, cloud computing and open-source software are the drivers that made big data both possible and necessary.
- The main challenges are storage and processing at scale, data quality, security and privacy, integration and skills; each one is also an opportunity for an organisation that manages it well.
Every organisation keeps data, but some now keep so much of it, arriving so fast and in so many shapes, that the tools built for ordinary record keeping stop working. A mobile money operator recording every transfer, a hospital network streaming readings from bedside monitors, and an online retailer logging every click on its website all face the same problem: a single database server cannot hold the data, cannot process it quickly enough, and was never designed for half of the formats it arrives in. This paper is about managing that data: the architectures, tools, designs and policies that let an organisation store it, process it and turn it into decisions.
What big data means
Big data is a term for datasets so large, fast-moving or varied that traditional data management tools, typically a relational database running on one powerful server, cannot capture, store, manage and analyse them within an acceptable time and cost. The definition is relative rather than a fixed size: what counts as big depends on the tools available and on the task. A dataset that a laptop spreadsheet cannot open is big for that user; the term in this paper means data that needs a cluster of machines working together.
The idea is usually explained through a set of characteristics that each begin with V. The analyst Doug Laney described the first three (volume, velocity and variety) in the early 2000s, and later writers added veracity and value to make the five V's most syllabuses now teach.
Some writers extend the list. Variability describes data whose meaning or flow rate changes over time, such as social media volumes that spike during an election or a national exam release. Visualisation is sometimes added because big data is only useful once people can see what it says. These extensions are useful in an essay, but the five V's above are the core an examiner expects.
Exam tip: matching a scenario to a V
Short questions often describe a situation and ask which V it shows. Read for the clue word: 'many different sources' or 'different formats' is variety; 'per second' or 'in real time' is velocity; 'petabytes' or 'millions of records' is volume; 'missing', 'inconsistent' or 'unreliable' is veracity; 'insight', 'better decisions' or 'pattern recognition leading to effective operations' is value. Name the V and give one sentence linking it to the clue.
Structured, semi-structured and unstructured data
The three structural forms of big data
| Form | What it means | Examples | Typical store |
|---|---|---|---|
| Structured | Data in a fixed format with a predefined schema of rows and typed columns, so it can be stored, accessed and processed directly | Bank ledger entries, payroll tables, KRA tax return fields, a hospital's admissions table | Relational database, data warehouse |
| Semi-structured | Data that carries its own tags or keys describing its structure, but no rigid schema shared by every record | JSON from a mobile app, XML invoices, email headers, web server logs, sensor messages | Document stores such as MongoDB, HDFS, data lakes |
| Unstructured | Data with no predefined model, which must be processed (for example by text mining or image analysis) before it can be queried | Customer emails, scanned documents, call recordings, X-ray images, CCTV video, social media posts | HDFS, object storage, data lakes |
Most organisational data is unstructured or semi-structured, which is exactly the data a relational database handles worst. XML and JSON are the standard examples of semi-structured data because each record names its own fields with tags or keys.
Drivers of big data
Big data did not appear because organisations suddenly decided to keep more records. A set of technical and economic changes made it both possible and necessary.
- Falling cost of storage and computing Disk and memory became cheap enough that keeping everything is often cheaper than deciding what to throw away.
- Commodity hardware and scale-out design Clusters of ordinary servers, rather than one expensive machine, can store and process data in parallel, the design idea behind Hadoop.
- Mobile phones and mobile money Each call, SMS, data session and M-Pesa transfer leaves a record, so a Kenyan telecom operator collects very large transaction and network datasets every day.
- The Internet of Things Sensors on vehicles, water meters, cold-chain fridges, farm equipment and hospital monitors send readings continuously.
- Social media and the web Posts, reviews, clickstreams and search logs reveal customer opinion and behaviour at a scale no survey could reach.
- Digitisation of government and business Online tax filing, e-procurement, digital health records and electronic payments turn paper processes into data.
- Cloud computing Storage and processing can be rented on demand, so a small firm can run a large analysis without buying a data centre.
- Open-source software Hadoop, Spark, Kafka and the NoSQL databases are free to use, which lowered the barrier to entry for every organisation.
- Competitive pressure Firms that use data to price, target and serve customers better force competitors to do the same.
Concepts and techniques
Big data management rests on a small number of ideas that recur in every later chapter. It helps to meet them together here before each is developed in its own chapter.
A useful way to see how these fit together is the big data life cycle: data is generated by its sources, acquired (ingested) into the platform, cleaned and stored, processed and analysed, and finally presented as reports, dashboards, alerts or automated decisions. Each stage has its own tools, and the architecture of a big data system is the decision about which tools sit at each stage and how data flows between them.
Trends in big data management
- Cloud-first platforms Organisations increasingly rent managed storage, warehouses and processing services instead of running their own Hadoop clusters, paying for what they use.
- Data lakes and lakehouses Raw data of every form is kept cheaply in a lake, and lakehouse designs add warehouse features such as tables, transactions and fast SQL on top of it.
- Real-time analytics Streaming platforms let firms act on events as they happen, for example blocking a fraudulent card payment before it completes.
- Artificial intelligence and machine learning Big data is the raw material for models that predict churn, score credit, detect fraud and read documents.
- Edge computing Some processing moves to devices close to the data source, such as a sensor gateway on a farm, so only summaries travel to the centre.
- DataOps and automation Pipelines are tested, versioned and deployed like software, so that data arrives reliably and errors are caught early.
- Data mesh Some large organisations give each business domain ownership of its own data as a product, instead of one central team owning everything.
- Privacy and regulation Laws such as Kenya's Data Protection Act, 2019 make data protection a design requirement rather than an afterthought, covered in the chapter on big data policy frameworks.
Applications of big data
Big data applications by sector
| Sector | Application | Kenyan illustration |
|---|---|---|
| Mobile money and banking | Fraud detection, credit scoring from transaction history, customer segmentation, agent liquidity planning | A mobile money operator scores small digital loans from a customer's history of transfers and repayments; a bank flags unusual card or app activity in real time |
| Revenue administration | Matching data from many sources to find under-declared income and non-filers | A tax authority such as KRA can compare filed returns with withholding, import and electronic invoice records |
| Healthcare | Disease surveillance, patient monitoring, predicting readmissions, stock planning for medicines | A hospital network monitors patients' vital signs from bedside devices and alerts nurses when readings move outside safe limits |
| Retail and e-commerce | Market-basket analysis, recommendations, dynamic pricing, demand forecasting | A supermarket chain mines till receipts to decide which products to place together and what to promote |
| Logistics and transport | Route optimisation, fleet tracking, delivery time prediction, fuel monitoring | A courier or haulage firm tracks its trucks by GPS between Mombasa and Nairobi and predicts delays |
| Agriculture | Yield prediction, weather-based advice, input targeting | A farm advisory service combines weather and soil data to send planting advice by SMS |
| Public sector | Planning services, detecting procurement fraud, managing traffic and utilities | A county government analyses service requests and revenue collection data to plan resources |
| Telecommunications | Network capacity planning, churn prediction, targeted bundles | An operator analyses call and data records to decide where to add network capacity |
About the illustrations
The Kenyan examples describe the kinds of use these organisations make of data, to show where the ideas apply; they are not descriptions of any named organisation's internal systems.
Challenges and opportunities
Big data brings real difficulties, and an exam answer is stronger when each challenge is explained rather than only named. The table pairs each challenge with the opportunity, or the management response, that answers it.
Challenges of big data and the matching opportunities
| Challenge | Why it is hard | Opportunity or response |
|---|---|---|
| Storage at scale | Data outgrows one server and grows every day | Distributed file systems and cloud object storage that scale out by adding machines |
| Processing speed | A single machine cannot scan petabytes or keep up with a stream | Parallel frameworks (MapReduce, Spark) and stream processors |
| Data quality (veracity) | Duplicates, missing values and conflicting sources mislead analysis | Systematic cleaning, validation rules and data quality monitoring |
| Integration of varied data | Sources use different formats, codes and identifiers | ETL and ELT pipelines, common identifiers and master data management |
| Security and privacy | Large stores of personal data are attractive targets and are regulated | Encryption, access control, anonymisation and compliance with the Data Protection Act, 2019 |
| Skills shortage | Data engineers and data scientists are scarce and expensive | Training, managed cloud services and simpler tools such as SQL on Hadoop |
| Cost and return | Clusters, licences and staff cost money before any value appears | Starting with a clear business question, open-source tools and pay-as-you-go cloud |
| Governance and ethics | Unclear ownership, unknown lineage and biased models create risk | Data governance roles, lineage tracking and ethical review of models |
The opportunities are the other side of the same coin. An organisation that manages big data well can make decisions on evidence rather than intuition, personalise products for millions of customers, detect fraud while it is happening rather than at audit, run operations more efficiently, and create new products, such as digital credit, that would be impossible without data. For a public body the opportunity is better targeted services and less leakage of revenue.
Big data compared with traditional data management
How big data management differs from traditional data management
| Aspect | Traditional data management | Big data management |
|---|---|---|
| Data form | Mainly structured | Structured, semi-structured and unstructured |
| Schema | Defined before loading (schema-on-write) | Often applied when read (schema-on-read) |
| Scaling | Scale up: buy a bigger server | Scale out: add commodity machines |
| Processing | Centralised on one server | Distributed and parallel across a cluster |
| Typical size | Gigabytes to a few terabytes | Terabytes to petabytes |
| Latency | Periodic reports | Batch and real-time side by side |
| Fault handling | Reliable hardware and backups | Assumes machines fail; replication and recomputation recover automatically |
Exam tip: building a two-mark answer
A two-mark 'explain' question usually wants a point plus its reason or effect: one mark for the idea, one for why it matters. For 'Explain one challenge of big data', write: 'Data quality (veracity): big data from many sources contains duplicates and missing values, so analysis on uncleaned data can produce wrong conclusions.' A bare list of words earns one mark at most.
Self-check
(a) Define big data. (2 marks)
(b) Distinguish between velocity and veracity as characteristics of big data. (2 marks)
(c) Classify each of the following as structured, semi-structured or unstructured data: an XML invoice, a recorded customer care call, a payroll table. (3 marks)
(d) Explain two drivers that have increased the volume of data held by Kenyan organisations. (4 marks)
Show worked answer
(a) Big data is data whose volume, velocity or variety is too great for conventional tools such as a single-server relational database to store, process and analyse within acceptable time and cost, so it needs distributed storage and parallel processing.
(b) Velocity is the speed at which data is generated and must be processed, for example mobile money transfers arriving every second. Veracity is how trustworthy the data is, its accuracy, completeness and consistency, for example records with missing or conflicting fields.
(c) XML invoice: semi-structured, because tags describe the fields but there is no rigid schema. Recorded call: unstructured, audio with no predefined data model. Payroll table: structured, fixed rows and typed columns.
(d) Mobile phones and mobile money: every call, SMS and M-Pesa transfer creates a record, so operators accumulate very large transaction datasets daily. Digitisation of services: online tax filing, digital health records and e-procurement turn paper processes into electronic data that is kept and analysed. (Other valid drivers: IoT sensors, social media, cheap storage, cloud computing.)
Self-check
A county government wants to use data from its revenue collection system, citizen complaints sent by SMS and WhatsApp, and traffic cameras to plan services.
(a) Identify which of the five V's is most clearly illustrated by the mix of sources, and justify your answer. (2 marks)
(b) Explain two challenges the county is likely to face and suggest how each could be addressed. (4 marks)
Show worked answer
(a) Variety: the county combines structured revenue records, semi-structured or unstructured text messages and unstructured video, so the data comes in many forms from different sources.
(b) Integration: the sources use different formats and identifiers, so the county needs an ETL pipeline and common identifiers (such as a ward code) to combine them. Privacy: complaints and camera footage contain personal data, so the county must restrict access, anonymise where possible and process the data lawfully under the Data Protection Act, 2019. (Data quality and skills shortage are also valid, each with a response.)