Getting Started

Installation

macOS (Homebrew)

brew tap pipectl/pipectl
brew install --cask pipectl

Windows

Direct download: grab pipectl_<version>_windows_amd64.zip from the Releases page, extract, and add the folder to your PATH.

Scoop (if you have Scoop installed):

scoop bucket add pipectl https://github.com/pipectl/scoop-pipectl
scoop install pipectl

Linux

Download a .deb or .rpm from the Releases page, or use the .tar.gz archive on any distribution.

Debian/Ubuntu:

sudo dpkg -i pipectl_<version>_linux_amd64.deb

Fedora/RHEL:

sudo rpm -i pipectl_<version>_linux_amd64.rpm

Docker

Images are published to GitHub Container Registry for both amd64 and arm64:

docker pull ghcr.io/pipectl/pipectl:latest

Run a pipeline with stdin:

echo '[...]' | docker run --rm -i ghcr.io/pipectl/pipectl:latest run pipeline.yaml

Run with local files mounted:

docker run --rm -i \
  -v $(pwd):/data \
  ghcr.io/pipectl/pipectl:latest run /data/pipeline.yaml --input /data/input.json

Go install (from source)

go install github.com/pipectl/pipectl/cmd/pipectl@latest

Verify the installation:

pipectl --help

Upgrading

macOS (Homebrew)

brew upgrade --cask pipectl

Windows

Direct download: download the new pipectl_<version>_windows_amd64.zip from the Releases page, extract, and replace the existing binary in your PATH.

Scoop:

scoop update pipectl

Linux

Download the new .deb or .rpm from the Releases page.

Debian/Ubuntu:

sudo dpkg -i pipectl_<version>_linux_amd64.deb

Fedora/RHEL:

sudo rpm -U pipectl_<version>_linux_amd64.rpm

Docker

docker pull ghcr.io/pipectl/pipectl:latest

Go install (from source)

go install github.com/pipectl/pipectl/cmd/pipectl@latest

Your first pipeline

Create a pipeline file:

# greet.yaml
id: greet
input:
  format: json
steps:
  - normalize:
      fields:
        name: capitalize
  - select:
      fields: [name, email]
output:
  format: json

Create some input:

echo '[{"name":"alice smith","email":"ALICE@EXAMPLE.COM"},{"name":"bob jones","email":"BOB@EXAMPLE.COM"}]' > people.json

Run it:

pipectl run greet.yaml < people.json

Or use --input instead of stdin redirection:

pipectl run greet.yaml --input people.json

Output:

[{"email":"ALICE@EXAMPLE.COM","name":"Alice Smith"},{"email":"BOB@EXAMPLE.COM","name":"Bob Jones"}]

Validate without running

Use --dry-run to check a pipeline is valid and see the planned steps without executing anything:

pipectl run greet.yaml --dry-run

Pipeline: greet
Steps:
  1. normalize
  2. select

Write output to a file

By default pipectl writes to stdout. Use -o to write to a file instead:

pipectl run greet.yaml -o output.json < people.json

Enable verbose logging

Use --verbose to see per-step detail — record counts, field operations, sort results — written to stderr:

pipectl run greet.yaml --verbose < people.json

Suppress output

Use --quiet (or -q) to suppress all diagnostic output. Only the final payload is written to stdout (or your output file). Useful for scripting or piping output to other tools:

pipectl run pipeline.yaml --quiet < input.json | jq .

Profile step timing

Use --timing to print a per-step summary table to stderr after execution. The table shows each step's name, duration, and record counts (in/out) — useful for spotting slow steps and verifying that filter or limit steps are behaving as expected:

pipectl run pipeline.yaml --timing < input.json
# STEP                          DURATION        IN      OUT
# 1. filter                         2ms       1000      847
# 2. sort                           5ms        847      847

--timing output is suppressed by --quiet.

Substitute variables

Use --var KEY=VALUE to substitute ${VAR} tokens in pipeline YAML before it is parsed. This lets you write reusable pipelines and supply environment-specific values at runtime:

# pipeline.yaml
id: example
input:
  format: ${INPUT_FORMAT}
steps:
  - limit:
      count: ${LIMIT}
output:
  format: json

pipectl run pipeline.yaml --var INPUT_FORMAT=jsonl --var LIMIT=50 < data.jsonl

--var can be repeated as many times as needed. Any ${VAR} token left unresolved after all substitutions are applied causes an error at startup.

Next steps

• Read Core Concepts to understand how pipelines, steps, and payloads fit together
• Browse the Step Reference to see all available steps
• Explore Example Pipelines for real-world patterns