pipectlYAML-defined data pipelines

Read from files or stdin. Transform through ordered steps. Write to files or stdout.

Get Started

View on GitHub

🔗

Composable

Chain any number of steps in any order. The output of each step is the input to the next.

📄

Multiple payload formats

JSON, JSONL, and CSV — with automatic conversion between them at any point in the pipeline.

⚡

Built-in steps

Filter, normalize, redact, cast, sort, dedupe, validate, convert, make HTTP calls, and more.

🔒

Built-in redaction

Mask or SHA-256 hash sensitive fields before they leave your pipeline.

✅

JSON Schema validation

Validate records against a JSON Schema and fail fast if the data doesn't conform.

🪶

Minimal by design

A single binary, no runtime dependencies, no framework overhead. Just YAML and your data.

Quick example

yaml

# customer-intake.yaml
id: customer-intake
input:
  format: csv
steps:
  - normalize:
      fields:
        first_name: capitalize
        last_name: capitalize
        email: lower
  - filter:
      field: country
      equals: AU
  - redact:
      fields: [credit_card]
      strategy: mask
  - select:
      fields: [first_name, last_name, email, plan, credit_card]
output:
  format: jsonl

bash

pipectl run customer-intake.yaml < customers.csv

jsonl

{"first_name":"Alice","last_name":"Smith","email":"alice@example.com","plan":"pro","credit_card":"****************"}
{"first_name":"Bob","last_name":"Jones","email":"bob@example.com","plan":"free","credit_card":"****************"}