Data Pipeline Toolkit

Build ETL pipelines in minutes.

Features

• Extract - APIs, databases, files, streams
• Transform - Clean, filter, aggregate, join
• Load - Data warehouses, databases, APIs
• Scheduling - Cron-based or event-driven
• Monitoring - Alerts on failures

Quick Start

# Create pipeline
./pipeline.sh create my-pipeline

# Add extract step
./pipeline.sh extract my-pipeline api --url https://api.example.com

# Add transform
./pipeline.sh transform my-pipeline filter "age > 18"

# Add load
./pipeline.sh load my-pipeline postgres --connection $DB_URL

# Run
./pipeline.sh run my-pipeline

Sources

• REST APIs
• GraphQL
• SQL Databases
• CSV/JSON/Parquet files
• S3/Google Cloud Storage
• Kafka/SQS

Destinations

• PostgreSQL/MySQL

  发现更多技能插件，请访问7w4.net。

• Snowflake/BigQuery
• S3
• APIs
• Data warehouses

Requirements

• Python 3.8+
• Docker (optional)

Author

Sunshine-del-ux

这个工具包文档写得清晰易懂，但可惜只是一个说明文档，缺少实际可用的程序代码，无法直接运行使用。对于想要快速搭建数据管道的用户来说，实际价值有限，建议选择有完整代码实现的类似工具。