Why this chapter matters
Network calls are failure-prone, so defensive API handling is required for dependable automation.
What you will learn
- Make GET requests with explicit params and timeout.
- Handle status errors and malformed payloads safely.
- Normalize API JSON into stable tabular records.
Understand the core ideas
API integration brings external data into your pipeline, but network boundaries are inherently unreliable. A robust requests workflow sets explicit timeout values, checks status codes, and validates payload structure before transformation. Avoid assuming a successful HTTP response always means usable business data. Fields can be missing, nested differently, or semantically changed without warning. Keep request parameters explicit and stable so calls are reproducible. For repeated jobs, log request metadata such as endpoint, params, and response timing for diagnostics.
Error handling is central in this chapter. Catch request-level exceptions such as timeout and connection errors, and separate them from content-level validation failures. Retries should be bounded and selective, usually for transient transport failures, not for schema mismatches. When JSON validation fails, persist a sample payload and fail with a precise message so fixes are fast. After validation, normalize records into stable tabular fields with explicit mappings. This preserves downstream consistency even when source APIs evolve around optional fields.
Key terms
- timeout
- A maximum wait duration that prevents indefinite blocking on slow or stalled network calls.
- status check
- Validation of HTTP response codes to detect failures before parsing payload content.
- schema validation
- Verification that required JSON keys and value types match expected structure.
- normalization
- Conversion of nested or irregular payload data into stable analysis-ready records.
Fetch exchange rates with guarded requests and JSON checks
A daily job pulls rates from a public endpoint. Required fields are base, date, and rates. The pipeline must avoid hangs, handle transient failures, and write normalized records for analysis.
- Call requests.get with explicit params and a timeout value, then use raise_for_status to fail fast on non-success HTTP responses.
- Catch requests.Timeout and requests.ConnectionError separately from HTTPError so operational logs can distinguish transient network faults from server responses.
- Parse response.json and validate required keys. If keys are missing or rates is not a mapping, raise a validation error that includes endpoint and missing fields.
- Flatten the rates mapping into rows with columns date, base, currency, and rate, then write to CSV or JSON with deterministic field order.
A common misconception
Claim: If the endpoint usually works, retries and schema checks are unnecessary overhead.
Correction: Intermittent failures and payload drift are common in real APIs. Basic guards are low cost and prevent major downstream reliability incidents.
Lessons in this chapter
- Request patternsBuild safe HTTP calls with predictable behavior.
- Response validationVerify status and schema before use.
- JSON to tableFlatten payloads into analysis-friendly records.
- Guide: API data fetchingIntegrate third-party data into local reporting. Read the full guide →
Study task
Chapter checkpoint
What problem does a request timeout solve?
It prevents scripts from hanging indefinitely and allows controlled retry or failure behavior.