Steps

A step is a single transformation applied to the pipeline payload. Steps run in the order they are listed. If any step fails, the pipeline stops.

All steps

Step	What it does	JSON	JSONL	CSV
assert	Assert record count or field existence	✓	✓	✓
cast	Convert field types — int, float, bool, string, time	✓	✓
convert	Convert payload format — json ↔ jsonl ↔ csv	✓	✓	✓
count	Print current record count to stdout	✓	✓	✓
dedupe	Remove duplicate records by key fields	✓	✓	✓
default	Fill missing or empty fields with a default value	✓	✓	✓
filter	Keep records matching a condition or nested all/any group	✓	✓	✓
http-request	Send payload to an HTTP endpoint as a side effect, continue with the same payload	✓	✓	✓
http-transform	POST/PUT/PATCH payload to an HTTP endpoint, continue with the response	✓	✓
limit	Truncate to N records	✓	✓	✓
log	Print record count and samples to stdout	✓	✓	✓
normalize	Normalise string fields (lower, upper, trim, capitalize…)	✓	✓	✓
redact	Replace field values with mask, sha256, or REDACTED	✓	✓	✓
rename	Rename fields	✓	✓	✓
select	Keep only the specified fields	✓	✓	✓
sort	Sort records by a field, ascending or descending	✓	✓	✓
validate-json	Validate records against a JSON Schema	✓	✓

Step syntax

Each step is a single-key object in the steps list:

steps:
  - normalize:
      fields:
        email: lower
  - redact:
      fields: [password]
      strategy: mask
  - filter:
      field: country
      equals: AU

The key is the step type. The value is the step's configuration. Each list item must contain exactly one step type.