Modular Object-Oriented ETL Data Pipeline

A robust, production-grade Extract, Transform, Load (ETL) data pipeline built from scratch in Python. This project ingests raw library loan data, processes it through a rigorous multi-gate validation and cleaning engine, and securely persists the structured records into an SQLite database.

The architecture is designed with a strong emphasis on Object-Oriented Programming (OOP) best practices, clean separation of concerns, and enterprise-level defensive programming.

🚀 Key Architectural Highlights

• Object-Oriented Design & Encapsulation:
  • Structured the pipeline into dedicated, modular classes (LibraryExtractor, LibraryTransformer, LibraryLoader, and a central orchestrator).
  • Applied encapsulation to break down complex data transformation logic into specialized private helper methods, ensuring high code readability and maintainability.
• Defensive Programming & Fail-Fast Schema Enforcement:
  • Utilized direct indexing for data ingestion to instantly catch and log schema breaks (such as missing or renamed columns) rather than using silent fallback methods.
  • Designed targeted try…except runtime error boundaries to safely isolate file I/O locks, type coercion errors, and unexpected data corruption.
• Modern Python Tooling & Centralized Logging:
  • Leveraged pathlib.Path for clean, cross-platform file path management across Windows, Mac, and Linux environments.
  • Configured a centralized logging infrastructure (logging module) to track pipeline execution lifecycles, structured warnings, and critical system failures in real-time.

⚙️ Pipeline Lifecycle

• 1. Extract (extractor.py): Ingests raw data from CSV files with early file existence checks and robust error logging.
• 2. Transform (transformer.py): Runs records through a 4-gate validation pipeline:
  • Gate 1: Strict schema validation and missing field checks via direct indexing.
  • Gate 2: Duplicate loan ID tracking and removal using set lookups.
  • Gate 3: Robust type coercion with safe default fallbacks.
  • Gate 4: Core business logic constraints (e.g., validating positive borrow durations).
• 3. Load (sqlite_loader.py): Persists clean, validated data efficiently into an SQLite relational database using batch execution (executemany).
• 4. Orchestrate (main.py): Acts as the central system controller, managing global error handling, exit codes, and end-to-end execution logging.