Why this chapter matters
Choosing the right structure directly affects correctness, lookup speed, and how easy transformations are to reason about.
What you will learn
- Use lists for ordered sequences and iteration.
- Use dictionaries for key-based lookup and mapping.
- Use sets for uniqueness checks and membership testing.
Understand the core ideas
Data structures are design choices, not just syntax choices. A list preserves order and supports sequential processing, so it is good for rows in arrival order. A dictionary gives key-based access, which is ideal when order_id should map to one canonical record. A set stores unique values and supports fast membership checks, which is useful for deduplication and validation. A tuple signals fixed shape data that should not change. Picking the right structure early lowers complexity because your operations align with data semantics.
In automation pipelines, structure choices affect both correctness and performance. If you repeatedly scan a list to check whether an ID already exists, runtime can degrade and duplicate handling can become inconsistent. Using a set for seen IDs and a dictionary for indexed records gives deterministic behavior and simpler logic. Error handling also benefits from structure discipline. Keep a separate list for validation errors with record identifiers and reasons, instead of mixing failed rows with cleaned rows. This preserves traceability and supports targeted reprocessing.
Key terms
- list
- An ordered mutable sequence used for row collections and iterative processing.
- dictionary
- A key value mapping used for direct lookup by identifiers.
- set
- An unordered collection of unique values used for deduplication and membership checks.
- tuple
- An ordered immutable sequence used to represent fixed structure data.
Merge two extracts with deduplication and indexing
You have extract_a and extract_b, each a list of order dictionaries. Some order_id values overlap and some records are missing required fields like amount. Build a clean merged view for downstream aggregation.
- Initialize seen_ids as an empty set, orders_by_id as an empty dictionary, merged_rows as an empty list, and validation_errors as an empty list.
- Iterate through both extracts in sequence. For each row, validate required keys. If a key is missing, append an error entry and continue.
- If order_id is already in seen_ids, skip or replace based on a documented rule. If new, add it to seen_ids, store row in orders_by_id, and append to merged_rows.
- After iteration, use orders_by_id for fast record retrieval and merged_rows for stable export order. Persist validation_errors for audit and correction.
A common misconception
Claim: A list is enough for everything if the dataset is not huge.
Correction: Even moderate workloads benefit from appropriate structures. Sets and dictionaries improve clarity and consistency, not only speed, especially for uniqueness and lookup rules.
Lessons in this chapter
- Lists and tuplesPick mutable versus immutable sequence types correctly.
- DictionariesRepresent row-like records and keyed indexes.
- SetsDetect duplicates and run fast membership checks.
- Guide: data structuresBuild a clean record model for analysis tasks. Read the full guide →
Study task
Chapter checkpoint
Which structure is best for repeated membership checks on unique IDs?
A set, because it stores unique values and provides fast membership testing.