Data Science & Analytics Engineering

Write a production-ready Python ETL pipeline for [USE CASE, e.g., "ingesting daily sales data from a REST API and loading to PostgreSQL"]. Requirements: 1) Extract: API pagination handling, retry logic with exponential backoff, rate limit handling, 2) Transform: Pandas/Polars transformations, data validation (Pydantic models), type casting, 3) Load: upsert to PostgreSQL using SQLAlchemy, 4) Error handling: try/except with specific error types, failed record logging, 5) Logging: structured logging with timestamps and record counts, 6) Configuration: environment variables for credentials, 7) Idempotency: safe to re-run without duplicates. Full Python with type hints and docstrings.

Write a dbt staging model for [SOURCE TABLE, e.g., "raw.stripe_charges"]. Requirements: 1) Source definition in schema.yml with freshness test, 2) Staging model SQL: select only needed columns, rename to snake_case, cast to correct types, parse timestamps to UTC, handle nulls with coalesce, 3) Schema.yml for the staging model: column descriptions, not_null tests, unique tests, accepted_values tests, 4) Staging model should be materialized as view, 5) Add a surrogate key using dbt_utils.generate_surrogate_key, 6) Add loaded_at metadata column. Include the full SQL and schema.yml.

Write a Python function that generates a comprehensive ML model evaluation report for a [TASK TYPE: binary classification/multiclass/regression] model. Include: 1) For classification: confusion matrix, ROC-AUC curve, precision-recall curve, classification report (precision, recall, F1 per class), calibration curve, 2) For regression: residual plot, actual vs. predicted plot, error distribution, MAE/RMSE/R² metrics, 3) Feature importance: SHAP summary plot, top 20 features, 4) Model comparison table if multiple models provided, 5) All plots saved to /reports/figures/, 6) Summary statistics printed to console. Python with matplotlib, seaborn, shap, sklearn. Full function with docstring.