TIL that Bash pipelines do not work like R pipelines

I was lowkey mindblown to learn how Bash pipelines actually work, and it's making me reconsider if R "pipelines" should really be called "pipelines" (I think it's more accurate to say that R has a nice syntax for function-chaining).

In R, each step of the pipeline finishes before the next step begins. In Bash, the OS actually wires up the all programs into a new program, like a big interconnected pipe, and each line of text travels all the way the down the pipe without waiting for the next line of text. This sort of stream-processing can be way faster for certain tasks.

R

As an example, this takes 2.1 seconds in R

read_csv("bigfile.csv", show_col_types = FALSE) |> filter(col == "somevalue") |> slice_head(n = 5) |> print())

read_csv reads the whole file. filter scans the whole file. And then I'm not exactly sure how slice_head works, but the entire df it receives is in memory...

Bash

The same command takes 0.01 seconds in Bash (200x faster)

cat bigfile.csv | grep somevalue | head -5

First, Bash runs cat, grep, and head all at once, connecting the output of cat to the input of grep. Then cat starts reading the file. As soon as cat "prints" a line, that line gets fed into grep. If the line matches grep's pattern, grep just forwards that line to stdout gets fed to head. Once head has seen 5 lines, it closes, which triggers a SIGPIPE to grep and the whole pipeline gets shut down. If the first 5 lines were matches, cat would only have to read 5 lines, whereas read_csv would read the whole file no matter what.

Note on comparison: The examples aren't perfect analogues because read_csv has to transform text into a df. Also it depends on the size of the file, the density of the matches, etc.

Exception to this rule: some bash commands (e.g. sort) have to process the entire file, so they effectively run in batch-mode, like R