pipeflow: v0.4.0 on CRAN

Pipeline setup

{pipeflow} builds a pipeline by adding one R function per step withpip_add (see also the Getstarted article). In this post, we’ll work with the following toypipeline:

• step load produces a sequence
• step clean doubles it
• step fit sums it, and
• step report formats the result

pip <- pip_new("my-pip") |>    pip_add(        step = "load",        fun = \(n = 5) seq_len(n),        tags = c("io", "daily")    ) |>    pip_add(        step = "clean",        fun = \(x = ~load) x * 2,        tags = c("io", "core")    ) |>    pip_add(        step = "fit",        fun = \(x = ~clean) sum(x),        tags = c("model", "core")    ) |>    pip_add(        step = "report",        fun = \(x = ~fit) paste("result:", x),        tags = "report"    )

Each step must have a unique name, and function parameters can referto other steps — for example, x = ~load takes the output ofthe load step. We also assign tags to each step,which we will use to filter steps later.

Before we run it, let’s have a look:

pip<pipeflow> my-pip (4 steps)---------------------------     step params depends state       tags1:   load      n           new   io,daily2:  clean      x    load   new    io,core3:    fit      x   clean   new model,core4: report      x     fit   new     report---------------------------<ready> last run: never

We see one row per step, showing its parameters(params), the steps it depends on (depends),its current state (all new before the firstrun), and our tags. Once we run the pipeline, …

pip_run(pip)info [2026-10-03 09:36:44.793 UTC]: Starting run of pipeflow 'my-pip'info [2026-10-03 09:36:44.794 UTC]: Step 1/4 loadinfo [2026-10-03 09:36:44.795 UTC]: Step 2/4 cleaninfo [2026-10-03 09:36:44.797 UTC]: Step 3/4 fitinfo [2026-10-03 09:36:44.799 UTC]: Step 4/4 reportinfo [2026-10-03 09:36:44.803 UTC]: Finished run of pipeflow 'my-pip'pip<pipeflow> my-pip (4 steps)---------------------------     step params depends state            out       tags1:   load      n          done      1,2,3,4,5   io,daily2:  clean      x    load  done  2, 4, 6, 8,10    io,core3:    fit      x   clean  done             30 model,core4: report      x     fit  done     result: 30     report---------------------------<ready> last run: 2026-10-03 11:36:44

… there is a new out column with the results, which canalso be accessed directly:

pip[["report", "out"]][1] "result: 30"

Pipeline views in action

Real pipelines get long, and often you only care about a subset ofsteps: a topic, a stage, or the steps that produce the desired outputs.Views (new in v0.4.0) let you work on such a subset without copyinganything. They reference the underlying pipeline, so every operationapplied to it — running it, updating parameters, collecting output, … —writes through to the original pipeline, restricted to the steps theview covers.

Creating views

pip_view() returns a view containing only the steps thatmatch the given filters:

pip_view(pip, tags = "core")<pipeflow_view> my-pip view (2 of 4 steps)------------------------------------------    step params depends state            out       tags1: clean      x    load  done  2, 4, 6, 8,10    io,core2:   fit      x   clean  done             30 model,core------------------------------------------<ready> last run: 2026-10-03 11:36:44pip_view(pip, tags = "core", state = "done")<pipeflow_view> my-pip view (2 of 4 steps)------------------------------------------    step params depends state            out       tags1: clean      x    load  done  2, 4, 6, 8,10    io,core2:   fit      x   clean  done             30 model,core------------------------------------------<ready> last run: 2026-10-03 11:36:44pip_view(pip, step = c("clean", "fit"))<pipeflow_view> my-pip view (2 of 4 steps)------------------------------------------    step params depends state            out       tags1: clean      x    load  done  2, 4, 6, 8,10    io,core2:   fit      x   clean  done             30 model,core------------------------------------------<ready> last run: 2026-10-03 11:36:44

By default steps must match all filters (logical AND), whilethe values within one filter are alternatives (OR).join = "union" keeps steps that match any filter, andfixed = FALSE treats filter values as regularexpressions:

pip_view(pip, tags = "report", step = "clean", join = "union")<pipeflow_view> my-pip view (2 of 4 steps)------------------------------------------     step params depends state            out    tags1:  clean      x    load  done  2, 4, 6, 8,10 io,core2: report      x     fit  done     result: 30  report------------------------------------------<ready> last run: 2026-10-03 11:36:44pip_view(pip, step = "^f", fixed = FALSE)<pipeflow_view> my-pip view (1 of 4 steps)------------------------------------------   step params depends state out       tags1:  fit      x   clean  done  30 model,core------------------------------------------<ready> last run: 2026-10-03 11:36:44

Selecting steps with [

The extract operator (also new in v0.4.0) provides a data.table-likeway of selecting steps and returns a view by default:

pip[c("load", "fit")]<pipeflow_view> my-pip view (2 of 4 steps)------------------------------------------   step params depends state       out       tags1: load      n          done 1,2,3,4,5   io,daily2:  fit      x   clean  done        30 model,core------------------------------------------<ready> last run: 2026-10-03 11:36:44

Boolean filters are evaluated in the context of the step table.

pip[tags %like% "core"]<pipeflow_view> my-pip view (2 of 4 steps)------------------------------------------    step params depends state            out       tags1: clean      x    load  done  2, 4, 6, 8,10    io,core2:   fit      x   clean  done             30 model,core------------------------------------------<ready> last run: 2026-10-03 11:36:44pip[step %in% c("clean", "fit") & state == "done"]<pipeflow_view> my-pip view (2 of 4 steps)------------------------------------------    step params depends state            out       tags1: clean      x    load  done  2, 4, 6, 8,10    io,core2:   fit      x   clean  done             30 model,core------------------------------------------<ready> last run: 2026-10-03 11:36:44

Running views

Running a view executes the steps it covers together with anyupstream dependencies that are not up to date. The run log marks theview’s own steps as [view] and the dependencies pulled inas [upstream]:

pip_reset(pip)  # reset the pipeline to "new" state for demonstrationpip_run(pip_view(pip, step = "report"))info [2026-10-03 09:36:44.883 UTC]: Starting run of pipeflow 'my-pip view'info [2026-10-03 09:36:44.883 UTC]: Step 1/4 [upstream] loadinfo [2026-10-03 09:36:44.883 UTC]: Step 2/4 [upstream] cleaninfo [2026-10-03 09:36:44.884 UTC]: Step 3/4 [upstream] fitinfo [2026-10-03 09:36:44.885 UTC]: Step 4/4 [view] reportinfo [2026-10-03 09:36:44.886 UTC]: Finished run of pipeflow 'my-pip view'

Afterwards, the original pipeline is up to date for the coveredsteps. Let’s collect all results related to the “core” steps:

pip[tags %like% "core"] |> pip_collect()$clean[1]  2  4  6  8 10$fit[1] 30

For a complete guide on composing views, see the Pipelineviews vignette.

More new and updatedfeatures

Views are only one of the additions. Each of the following has itsown article on the documentation site:

• Functional and method-style API — the pipeablepip_*() functions, with every function also available as amethod (p$run(), p$view(), …): Getstarted.
• Extract and replace operators — data.table-style[ and [[ for reading and editing steps: reference.
• Collect and group output —pip_collect() with grouping by tags or dependencies: Collectand group output.
• Combining and modifying pipelines — combine severalpipelines with rbind(), or replace, rename, remove, andreset steps: Combinepipelines · Modifyexisting pipelines.
• Split, map, and reduce — nativeexec = "split", "auto", and"reduce" modes: Split,map, and reduce.
• Nested pipelines — use a pipeline as a reusablebuilding block inside a step: Nestedpipelines.
• Self-modifying pipelines — .self,run-time restructuring, and restart(): Self-modifyingpipelines.

Under the hood, significant performance gains have been made byimplementing the dependency graph in C++ (since v0.3.0) and by improvingthe step and parameter bookkeeping in the underlying {data.table}.Finally, by dropping lgr and jsonlite,external package dependencies have been reduced to {data.table} andRcpp.

{pipeflow} vs {targets}

{targets} remains R’s de-facto standard for heavy-duty, reproduciblepipelines: it persists results to disk, records provenance, and scalesto distributed infrastructure via crew. {pipeflow} does nottry to compete on that home turf but targets a different niche — theinteractive session, often as the backend of a Shiny app — where apipeline is built and modified on the fly and has to respond while auser waits. Since responsiveness is key in that use case, {pipeflow} hasbeen optimized for low latency.

The pipeflowvs targets article compares the in-session latency of the twopackages head to head. The figure below gives a flavour of theresults:

It shows the scenario where one parameter has changed¹ and the pipeline isrerun, with each step doing 5 ms of work. The x-axis represents thenumber of steps in the pipeline (its size), and the y-axis shows thetime required to re-run it.²

Across the tested sizes, {pipeflow} is consistently the faster of thetwo — see the article for the full benchmark and methodology.

Overall, the two are best seen as complementary: reach for {targets}for large batch jobs that must be reproducible end to end, and for{pipeflow} when the pipeline is part of an interactive application.

Wrapping up

{pipeflow} 0.4.0 is on CRAN:

install.packages("pipeflow")

The fullchangelog lists every change, and the documentation includes thearticles linked above. If you build Shiny apps or spend a lot of timeexploring parameters interactively, give it a try — feedback and issuesare welcome on GitHub.