Performance

pipectl is built for small-to-medium batch and ETL work — shaping data in CI, transforming webhook payloads, local dev tooling, config-driven glue between systems. It is not a replacement for a distributed engine like Spark: pipectl reads the whole payload into memory, runs it through the pipeline in a single process, and writes the result back out. That design is what makes pipeline behavior simple and predictable (steps like sort, dedupe, and count-based assert can just look at the whole payload), but it also means memory use scales with input size — there's no streaming mode.

The numbers below quantify that envelope: how fast a realistic pipeline runs, and how much memory it actually uses, across file sizes from 10MB up to 1GB — a size chosen deliberately as a practical ceiling for this design, not a technical limit.

Methodology

• Pipeline: a 10-step pipeline — assert → default → normalize → dedupe → filter → redact → select → rename → sort → count — run identically across CSV, JSON, and JSON Lines input, so the numbers below isolate format overhead rather than step-selection differences. cast/convert/validate-json are JSON/JSONL-only steps and are intentionally excluded so the three formats are directly comparable. http-request/http-transform are excluded as I/O-bound — this measures pipeline throughput, not network latency. See benchmark/pipeline-csv.yaml (and the -json/-jsonl variants) in the repo for the exact definition.
• Data: synthetic customer-like records (name, email, country, plan, credit card, signup date, amount, status) generated with realistic value variety — mixed casing, ~5% missing country, ~2% duplicate ids, ~25% inactive status — so every step in the pipeline has real work to do. See benchmark/gen.
• Timing: pipectl run --timing, 5 repetitions per size/format, median reported.
• Memory: peak resident set size (RSS) of the pipectl process for each run, measured via the OS's own process accounting (rusage) — not estimated.
• Reproduce it yourself: go run ./benchmark/run from the repo root builds pipectl fresh, generates the data, runs the full matrix, and rewrites the results below in place. See benchmark/README.md.

Results

Last generated by go run ./benchmark/run on darwin/arm64, 10 cores, 2026-08-19.

CSV

Size	Records	Median duration	Throughput	Peak RSS	Peak RSS / input size
10MB	99339	520ms	19.2 MB/s	118.98MB	11.9x
100MB	984554	5.77s	17.3 MB/s	1.09GB	11.1x
250MB	2446084	14.32s	17.5 MB/s	2.60GB	10.6x
500MB	4882016	29.18s	17.1 MB/s	4.93GB	10.1x
1GB	9987425	62.66s	16.3 MB/s	9.28GB	9.3x

Per-step timing at 1GB (csv)

STEP                          DURATION            IN       OUT
1. assert                     <1ms           9987425   9987425
2. default                    72ms           9987425   9987425
3. normalize                  4671ms         9987425   9987425
4. dedupe                     8497ms         9987425   9787137
5. filter                     5382ms         9787137   7341478
6. redact                     399ms          7341478   7341478
7. select                     1548ms         7341478   7341478
8. rename                     <1ms           7341478   7341478
9. sort                       33881ms        7341478   7341478
10. count                     <1ms           7341478   7341478

JSON

Size	Records	Median duration	Throughput	Peak RSS	Peak RSS / input size
10MB	45844	470ms	21.2 MB/s	157.97MB	15.8x
100MB	456574	5.44s	18.4 MB/s	1.45GB	14.8x
250MB	1140111	13.90s	18.0 MB/s	3.47GB	14.2x
500MB	2275323	28.84s	17.3 MB/s	6.97GB	14.3x
1GB	4654919	61.60s	16.6 MB/s	11.71GB	11.7x

Per-step timing at 1GB (json)

STEP                          DURATION            IN       OUT
1. assert                     <1ms           4654919   4654919
2. default                    462ms          4654919   4654919
3. normalize                  2865ms         4654919   4654919
4. dedupe                     1551ms         4654919   4561883
5. filter                     790ms          4561883   3421916
6. redact                     530ms          3421916   3421916
7. select                     1258ms         3421916   3421916
8. rename                     1162ms         3421916   3421916
9. sort                       27117ms        3421916   3421916
10. count                     <1ms           3421916   3421916

JSONL

Size	Records	Median duration	Throughput	Peak RSS	Peak RSS / input size
10MB	46045	474ms	21.1 MB/s	129.44MB	12.9x
100MB	458571	5.48s	18.2 MB/s	1.02GB	10.4x
250MB	1145072	13.95s	17.9 MB/s	2.52GB	10.3x
500MB	2285220	29.04s	17.2 MB/s	4.83GB	9.9x
1GB	4675167	62.30s	16.4 MB/s	10.01GB	10.0x

Per-step timing at 1GB (jsonl)

STEP                          DURATION            IN       OUT
1. assert                     <1ms           4675167   4675167
2. default                    471ms          4675167   4675167
3. normalize                  2956ms         4675167   4675167
4. dedupe                     1596ms         4675167   4581746
5. filter                     856ms          4581746   3436777
6. redact                     570ms          3436777   3436777
7. select                     1564ms         3436777   3436777
8. rename                     1167ms         3436777   3436777
9. sort                       27974ms        3436777   3436777
10. count                     <1ms           3436777   3436777

Reading the memory numbers

"Peak RSS / input size" is the multiplier between a file's size on disk and the most memory pipectl used while processing it. It's consistently well above 1x because the whole file is read into a byte buffer and parsed into an in-memory record structure at the same time, and JSON/JSONL carry more per-record overhead than CSV's plain [][]string. Use this multiplier to sanity-check --max-input-size against the memory available on the machine actually running the pipeline — pipectl run also warns if a given --max-input-size looks large relative to total system RAM (see the CLI reference).