Redirection and Pipes - Aiming and Composing Streams
Context
A command's three channels point somewhere by default, two of them at your screen. Redirection is how you re-aim a channel at a file, at a sink that discards it, or at another channel. A pipe is how you aim it at another command instead, so one tool's output becomes the next tool's input with no file in between.
This is where the terminal stops being a place you issue single orders and becomes a place you assemble behavior. Capturing a training run's full log, muting a noisy data loader's warnings while keeping its results, counting how many of a hundred log files mention an error, all of it is aiming streams and wiring commands together.
Without it, output that scrolls off the screen is gone, and a job that died at three in the morning leaves nothing to read the next day. Worse, one ordering mistake in a redirection writes a file that looks like a complete log and silently is not, with the errors, the part you needed, delivered to a terminal nobody was watching. The previous lesson gave you commands to run. This one names the channels every command was already using and hands you the steering wheel.
Goal
- Identify, for any running command, which of its three byte channels a given piece of text travels on.
- Send a command's stdout into a file with
>and>>, predicting which one preserves prior contents, and discard a chosen channel with/dev/null, stating when discarding is the right call. - Merge stderr into stdout with
2>&1, and explain precisely whycmd > file 2>&1andcmd 2>&1 > filediffer. - Connect commands with
|so one command's output becomes the next one's input, and filter text withgrep. - Select many files at once with the glob wildcards
*,?and[...], and say which program expands them. - Read an exit code with
$?and chain commands with&&,||and;so the next step runs on success, on failure, or unconditionally.
Prerequisites
Neither is required, the lesson stands on its own. They are here if you want a second angle on the same machinery.
- YouTube - "Linux Crash Course - Data Streams (stdin, stdout & stderr)" (Learn Linux TV) - the three channels shown in a live terminal and split into separate files, which the redirection operators here re-aim.
- GNU Bash Reference Manual - "Redirections" - the precise rules behind the order trap, including the
&>shorthand this lesson names.
Table of Contents
- The Three Streams (stdin, stdout, stderr)
- Output Redirection (
>and>>) - Discarding Output (
/dev/null) - Stream Merge and the Order Trap (
2>&1) - Pipes and Filters (
|andgrep) - Glob Patterns (Wildcards)
- Exit Codes and Command Chaining (
$?,&&,||,;)
Theory
SETUP - DO THIS FIRST (everything below depends on it):
Two log files stand in for the output of real training runs. Build them once, and every grep, glob and pipeline below has something real to read.
cd ~ # run every block in this lesson from your home directory
cat > training.log <<'EOF'
epoch=1 loss=0.42
epoch=2 loss=0.31
Warning: data loader stalled for 3s
epoch=3 loss=0.28
ERROR: batch 17 has a malformed label
epoch=4 loss=0.27
EOF
cat > run.log <<'EOF'
epoch=1 loss=0.51
WARNING: learning rate schedule ignored
epoch=2 loss=0.44
error: checkpoint write retried
epoch=3 loss=0.39
EOFcat > file <<'EOF' writes every line up to the closing EOF into the file, which is how you create a multi-line file without an editor. Neither log records a run that converged, which is what makes the failure branches below actually fire.
The Three Streams (stdin, stdout, stderr)
What. A stream is a continuous, one-directional channel of bytes moving into or out of a running process. Every process the operating system starts is handed exactly three of them before its first line of code runs, labeled with small integers called file descriptors.
| Descriptor | Stream name | Direction |
|---|---|---|
0 |
stdin | bytes in |
1 |
stdout | bytes out, results |
2 |
stderr | bytes out, diagnostics |
Why. A process needs a uniform way to talk to the outside world without knowing what is on the other end, and the stream is that uniform flow of bytes. Because the destination is separate from the stream itself, a program can send its results to your screen, to a file, or to another program while its code stays identical. The split between 1 and 2 is what makes debugging possible at all: if a program mixed its complaints into its results you could never separate the answer from what went wrong, and when a training run dies you want its traceback by itself rather than buried in a thousand lines of progress output.
When. Every command, always. You think about the streams explicitly the moment you need to separate a program's answer from its complaints, which is the moment you start redirecting them.
# The listing IS the result, so it goes to stdout (1).
ls /home
# There is no result to give, so 'ls' writes a diagnostic to stderr (2).
ls /no_such_directory_here
# ls: cannot access '/no_such_directory_here': No such file or directoryBoth messages appear in the same terminal window, which is exactly why the two channels are easy to confuse. Same window, two different channels, and by default both happen to be wired to your screen.
# One command, BOTH channels active at once. The found path is a result
# on stdout, the "Permission denied" notes are diagnostics on stderr.
find /etc -name "hosts"Right now they interleave on your screen. Everything that follows in this lesson is about sending each one somewhere different and watching them split apart cleanly.
DON'T:
- DON'T picture a stream as a file you can scroll back through, it flows once, forward. A byte already read is not waiting to be re-read.
- DON'T assume everything you see in the terminal came from stdout, warnings and errors share the window and ride a different channel.
- DON'T treat a message on stderr as proof the command failed. A program can write a diagnostic and still succeed, the channel signals "this is a diagnostic", not "the program is over".
Output Redirection (> and >>)
What. The > operator takes a command's stdout and points it at a file instead of the terminal. >> does the same but adds to the end of the file rather than wiping it first. If the file does not exist, both create it. If it does exist, > truncates it to empty and >> preserves what is there.
Why. Results on screen vanish when you scroll past them, results in a file persist and can be reopened, searched and processed later. This is how you capture a result, the metrics a script printed, the listing of a directory, the predictions a model wrote. The choice between the two operators is the difference between a snapshot and a history: with > every write destroys the last, with >> every write extends.
When. Use > when you want the latest run's output and are content to start fresh each time. Use >> when the file is meant to grow, a log accumulating across many runs, a metrics file gaining a line after each epoch.
# Without redirection: the listing prints to the terminal (stdout -> screen).
ls /home
# With '>': stdout is re-aimed at a file. Nothing prints, the listing
# lands inside listing.txt. The file is CREATED if absent, or WIPED
# and rewritten if it already exists.
ls /home > listing.txtAfter the second command you see nothing on screen, because the bytes that would have hit the terminal were routed into listing.txt. Only stdout was redirected, so had ls produced an error, that error would still have reached the screen. That single-channel behavior is the whole reason the merge operator later in this lesson exists.
# The contrast in one place. Watch what each operator keeps.
echo "first" > demo.txt # demo.txt now: first
echo "second" > demo.txt # '>' WIPED it -> demo.txt now: second only
echo "third" >> demo.txt # '>>' appended -> demo.txt now: second, then thirdThe middle line proves the danger of >, it threw away first without warning. The last line shows >> doing the opposite. Same target file, opposite intentions.
DON'T:
- DON'T use
>on a file whose contents you want to keep. It truncates the file to empty before the command runs, so even a command that then fails and writes nothing has already destroyed your data. - DON'T expect
>to capture errors, bare>only redirects stdout. A>-redirected command can still spray error messages onto your terminal. - DON'T assume
>>and>differ on a file that does not exist yet, both simply create it. The distinction only appears once the file has contents.
Discarding Output (/dev/null)
What. /dev/null is a special system file that acts as a sink. Anything written to it is silently thrown away the instant it arrives.
Why. Sometimes output is noise you want gone, not in a file and not on screen. A command may print harmless warnings on stderr that clutter your view, or you may have run it purely for its side effect. /dev/null is the disciplined way to silence exactly one channel while leaving the other intact, and it pairs naturally with the descriptor numbers, 2>/dev/null discards only errors and >/dev/null discards only results.
When. Use it when a specific channel is genuinely unwanted, suppressing known harmless warnings during a noisy data-loading step, or hiding Permission denied chatter from a broad search while keeping the answer.
# 'find' over /etc emits found paths (stdout) and "Permission denied"
# notes (stderr). Send ONLY stderr to the sink.
find /etc -name "hosts" 2>/dev/null # keeps the answer, drops the complaints
# The reverse: discard the result, keep only the errors on screen.
find /etc -name "hosts" >/dev/nullDON'T:
- DON'T discard stderr reflexively.
2>/dev/nullhides every error, including the real failure you needed to see. Silencing diagnostics out of habit is how a broken pipeline looks fine right up until it is not. - DON'T think
/dev/nullstores anything recoverable, it is a black hole and not a trash bin.
Stream Merge and the Order Trap (2>&1)
What. 2>&1 means: make descriptor 2, stderr, point to wherever descriptor 1, stdout, is pointing right now. It folds the error channel into the output channel so both flow to the same destination. The &1 part means "the current target of descriptor 1", not a file literally named 1.
Why. Often you want a command's complete output, results and errors interleaved in the order they actually happened, captured together in one file. Bare > only moves stdout and leaves stderr on the terminal, so 2>&1 is how you bring stderr along. This is the everyday way to grab a full log of a run, every metric and every warning in a single file you can read top to bottom.
When. Whenever "capture everything this command says" is the goal, or whenever you want to collapse both channels into a single sink.
Here is the heart of the lesson. These two commands look almost identical and do different things:
# CORRECT - capture BOTH stdout and stderr into the file.
# Step 1: '> capture.log' makes stdout (1) point at capture.log.
# Step 2: '2>&1' makes stderr (2) point where 1 points NOW = capture.log.
some_command > capture.log 2>&1# TRAP - this does NOT do the same thing.
# Step 1: '2>&1' makes stderr (2) point where 1 points NOW = the TERMINAL
# (stdout has not been moved yet).
# Step 2: '> capture.log' now moves stdout (1) to capture.log.
some_command 2>&1 > capture.log # errors STILL go to the terminalBash applies redirections left to right, and 2>&1 copies the destination of stdout at the moment it is read, not whatever stdout becomes later. In the correct version stdout is already aimed at capture.log by the time 2>&1 runs, so stderr follows it into the file. In the trap, 2>&1 runs first while stdout still points at the terminal, so stderr is pinned to the terminal and the later > capture.log moves only stdout. Same symbols, reversed order, opposite outcome.
# The rule made concrete: aim stdout FIRST, then tie stderr to it.
echo "metric: 0.9" > full.log 2>&1 # both channels -> full.log
echo "noisy thing" > /dev/null 2>&1 # silence BOTH, same ordering ruleEvery operator in this lesson reads as "(which descriptor)(arrow)(where to)". Once you read them that way the trap stops being mysterious, because 2>&1 can only copy the destination stdout has at that instant, so the instant matters.
Bash also accepts &> capture.log as one-token shorthand for > capture.log 2>&1, but sh on Ubuntu and WSL reads &> differently and leaves the file empty, so the long form is the one that works in every shell.
DON'T:
- DON'T write
2>&1before the stdout redirection when you mean to capture both.command 2>&1 > fileis the classic bug, errors leak to the terminal while you believe they are in the file. - DON'T read
2>&1as "redirect to a file named 1". The&is what makes it mean "the current destination of descriptor 1", and without it2>1creates a literal file called1.
Pipes and Filters (| and grep)
What. A pipe, written |, connects the stdout of the command on its left to the stdin of the command on its right. The left command's output never reaches the screen, it flows directly into the right command as input. A chain of commands joined by pipes is a pipeline. grep is the canonical thing to put on the right of one: it reads lines of text and prints only those matching a pattern.
Why. Redirection sends stdout into a file. A pipe sends it into another command instead, with no file in between, which is what lets tiny single-purpose tools combine into something no single tool provides. Output is often mostly noise with a few lines that matter, one error buried in a long log, the one config line you need, and grep is how you keep the signal. The pattern ... | grep PATTERN is among the most-typed phrases in a terminal.
When. Reach for a pipe whenever a command produces text and you want another tool to act on it. Any time you catch yourself thinking "and then do this to the output", a pipe is the answer. Reach for grep whenever the question is "which lines contain X".
# Without a pipe: 'ls' floods the terminal with filenames.
ls /usr/bin
# With a pipe: ls's stdout flows into 'wc -l' (line count) instead of
# the screen, so you see only how many entries there were.
ls /usr/bin | wc -l# Pipes chain. Each '|' wires one stdout to the next stdin.
ls /etc | grep ".conf" | wc -l # list -> filter -> countThree processes run at once, joined into a pipeline, and the data flows left to right through two pipes. This is composition in one line.
# grep reads a FILE directly, or PIPED input. Same result either way.
grep "loss" training.log
cat training.log | grep "loss"grep does not care whether its input is a file or a pipe, because both arrive as lines of text.
# Flags that reshape the same core job of selecting lines by pattern.
grep -i "warning" run.log # -i ignores case: Warning, WARNING, warning
grep -v "loss" run.log # -v INVERTS: every line NOT containing "loss"
grep -c "epoch" run.log # -c COUNTS matching lines instead of printing
grep -q "converged" run.log # -q is QUIET: no output, only an exit codeDON'T:
- DON'T expect a pipe to carry a command's errors.
|connects stdout to the next stdin, stderr is not piped and still goes to your terminal. Fold it in with2>&1first if you need it. - DON'T pipe into a command that does not read stdin.
ls | mkdirdoes nothing useful, becausemkdirtakes its target as an argument. - DON'T forget
grepis case-sensitive by default, sogrep "error"missesERRORandError. - DON'T add
catwhengrepcan read the file itself, the piped form earns its place only when the text comes from a command.
Glob Patterns (Wildcards)
What. A glob is a shorthand the shell expands into a list of matching paths before a command runs. * matches any run of characters including none, ? matches exactly one character, and [...] matches any one character from the set inside the brackets.
Why. Typing every filename by hand is slow and error-prone when you mean "all of them" or "all that end in .csv". Crucially the shell, not the command, does this expansion, so before ls *.csv runs the shell has already replaced the pattern with the actual filenames and ls only ever sees the finished list. That is why globs work with any command, the tool never knows a wildcard was involved.
When. Whenever you want a command to act on many paths chosen by a name pattern.
# Empty files for the patterns below to match.
touch data.csv train.csv test_2024.csv model1.csv model2.csv model10.csv run1.log run2.log run3.log
# '*' = any characters (including none).
ls *.csv # every .csv here, model10.csv included
# '?' = exactly ONE character.
ls model?.csv # model1.csv and model2.csv, but NOT model10.csv
# [...] = one character from the set.
ls run[123].log # run1.log, run2.log, run3.log# Globs work with ANY command, because the shell expands them first.
wc -l *.log # wc receives an already-expanded list of filenames# A glob that matches nothing is passed through unchanged.
ls *.parquet # if no .parquet files exist, the command receives the
# literal text '*.parquet' and reports it cannot find itThat last behavior is worth knowing so a "No such file" error naming a pattern does not confuse you.
DON'T:
- DON'T mix up globbing with
grep. A glob is expanded by the shell and selects file paths by name,grepruns as a command and filters line content. - DON'T assume
*matches hidden files. Names beginning with.are skipped on purpose, sols *will not list.bashrc.
Exit Codes and Command Chaining ($?, &&, ||, ;)
What. Every command leaves behind an exit code when it finishes, a single number reporting how it ended, where 0 means success and any nonzero value means some kind of failure. The variable $? holds the most recent one. The chaining operators read that code: A && B runs B only if A succeeded, A || B runs B only if A failed, and A ; B runs B unconditionally.
Why. "Did that work?" needs a machine-readable answer, not a glance at the screen. This extends the process lifecycle you already know: when a process terminates it does not just stop, it hands back a number summarizing the outcome, and that number is what lets other commands react automatically. This success-or-failure logic is the exact decision your own code will later make with if and else, met here in its shell form.
When. Use && to guard a step that must not run unless the previous one succeeded. Use || for a fallback or an error message. Use ; when you want several commands in order and do not care about their statuses.
# A command that SUCCEEDS leaves exit code 0.
ls /home
echo $? # 0
# A command that FAILS leaves a nonzero code.
ls /no_such_dir
echo $? # 2, which is ls's "couldn't access" code$? reflects only the most recent command, and it is overwritten by every command including the echo used to print it.
# grep encodes found / not found as its exit code, making it a yes/no test.
grep -q "converged" training.log
echo $? # 0 if "converged" appeared, 1 if it did not# && : run the second command ONLY if the first succeeded.
cd no_such_project && python3 train.py # the cd fails, so training never starts
# || : run the second command ONLY if the first FAILED.
grep -q "converged" training.log || echo "WARNING: run did not converge"
# ; : run both in order, no matter what.
# finished still prints
echo "starting"; ls /no_such_dir; echo "finished"# A stand-in training script that runs cleanly and exits 0.
echo "print('epoch=1 loss=0.42')" > train.py
# Combine them: a success path and a failure path in one line.
python3 train.py && echo "training OK" || echo "training FAILED"
# epoch=1 loss=0.42
# training OKDON'T:
- DON'T read
$?after running another command in between, capture it immediately. - DON'T assume nonzero means "crashed". It means "did not succeed", which includes ordinary outcomes like
grepfinding no match. - DON'T use
;when you mean&&.build ; deploydeploys a broken build, because;ignores exit codes entirely. - DON'T over-read
A && B || Cas a strict if/else. IfAsucceeds butBitself fails,Cstill runs.
Additional - the whole lesson in one line
The pieces interlock. grep -i "error" *.log | wc -l globs every .log file with shell expansion, filters their lines case-insensitively with grep, and pipes the survivors to a line count. Add > report.txt 2>&1 and the count, and anything wc complains about, is captured, add && echo "logs checked" and the exit code decides whether the confirmation prints. Aim, compose, select, check, chain, that is the entire lesson in a single executable thought.
Lab Exercises
Lab Exercise 1: Capture, Append, and Lose
Create a file with one line in it using >. Add a second line with >> and confirm both are present. Now write a third line using > and confirm what happened to the first two. Then run a command that fails, such as listing a directory that does not exist, redirecting only its stdout to a file, and state where the error message actually went and why the file is empty.
Lab Exercise 2: Compose a Report
Working in a directory containing several .log files, build one line that selects every log file with a glob, keeps only the lines mentioning error regardless of capitalization, and counts them. Then extend it so a confirmation is appended to a running audit.log only if the pipeline exited cleanly. In a comment, name which stage's exit code the guard is actually reading.
Solution 1: Capture, Append, and Lose
# '>' creates the file and writes one line.
echo "epoch=1 loss=0.42" > metrics.txt
# '>>' adds beneath it, keeping what was there.
echo "epoch=2 loss=0.31" >> metrics.txt
cat metrics.txt # both lines, oldest first
# '>' TRUNCATES before writing, destroying both previous lines.
echo "epoch=3 loss=0.28" > metrics.txt
cat metrics.txt # only the epoch=3 line survives
# A failing command with only stdout redirected.
ls /no_such_dir > result.txt
# ls: cannot access '/no_such_dir': No such file or directory <- on the TERMINAL
cat result.txt # emptyThe error printed on the terminal because ls wrote it to stderr, and > moves stdout alone. result.txt is empty because > truncated it to nothing before ls ran, and ls then produced no stdout at all. Both halves of that outcome were decided before the command did any work.
DON'T:
- DON'T conclude the command "wrote nothing" from an empty file, it wrote to a channel the redirection never touched.
- DON'T carry the first two lines into the final answer, the closing
>erased them without warning.
Solution 2: Compose a Report
# glob *.log (expanded by the shell) -> grep filters lines case-insensitively
# -> wc counts them -> && appends a marker only if the pipeline succeeded.
grep -i "error" *.log | wc -l && echo "report built" >> audit.log
# the && reads the exit code of 'wc', the pipeline's LAST stage, not grep'sThe shell expands *.log into the real filenames before anything runs, grep -i keeps matching lines regardless of case, and wc -l reports how many survived. The && then appends a line with >> so audit.log grows rather than being replaced.
The comment carries the point worth keeping. A pipeline runs several commands and reports only one exit status, the last stage's, so the guard is asking whether wc counted successfully. wc succeeds whether it counted a thousand errors or zero, which makes this a confirmation that confirms the pipeline ran and nothing about what it found.
DON'T:
- DON'T assume
&&reads the status of the stage that interests you, it reads the last one.grep's found-or-not status was consumed inside the pipeline and never reached the guard.
Theory Assessment
Conceptual reinforcement, not a coding section. All four questions with their hints first, then all four answers.
Q1: What > moves, and what it destroys
State which channel > redirects and what happens to a file that already has contents. Then predict: given echo "a" > f.txt, then echo "b" >> f.txt, then echo "c" > f.txt, what are the final contents of f.txt, and why?
Hint: Only one of the three channels is affected, and the file is not treated gently. Track the file after every command, because one of the three operators wipes it.
Q2: The order trap
Explain why cmd > log 2>&1 captures both channels while cmd 2>&1 > log does not, referencing how bash evaluates redirections and be precise about what &1 refers to. Then write the single redirection that throws away both a command's stdout and stderr, and explain why the order of its parts matters there for exactly the same reason.
Hint: The key is when 2>&1 reads stdout's destination relative to when > log changes it. For the silencing half, send stdout to the sink first, then merge, and ask what breaks if you swap them.
Q3: Who expands a glob, and what a pipe leaves behind
When you run ls *.csv, name which program turns *.csv into actual filenames and at what moment. Then distinguish what a glob selects from what grep selects. Finally, state which channel a pipe carries and which one it conspicuously does not.
Hint: It is not ls, and it happens before the command starts. For the last part, two of the three streams are involved and the third is left behind on your screen.
Q4: The check that checks nothing
Explain the danger of writing build ; deploy when you meant build && deploy. Then consider this line, which looks like a check and reads like a check:
grep -i "error" *.log | wc -l && echo "clean"Name which command's exit code the && is actually reading, say whether clean prints when errors were found, and explain what happened to the status that would have told you what you wanted to know.
Hint: One operator ignores failure entirely, so picture deploying something that never built. For the second part, a pipeline has several commands but only one exit status, and it belongs to a specific stage.
Answer 1:
> redirects only stdout, descriptor 1. If the target file already has contents, > truncates it to empty first, then writes the command's output from the start. That truncation happens even if the command produces no output or fails outright, so prior data is lost the moment the redirection is set up, before the command runs at all.
echo "a" > f.txt # f.txt: a
echo "b" >> f.txt # f.txt: a, b
echo "c" > f.txt # '>' WIPES -> f.txt: cFinal contents: c alone. The closing > truncated the file before writing, destroying both a and b. Only the append in step 2 preserved anything, and the final overwrite erased it.
DON'T assume > is safe on a populated file, it overwrites unconditionally with no warning and no undo.
Answer 2:
Bash applies redirections left to right, and 2>&1 copies the destination stdout has at the instant it is read, not the one it will have later. &1 means "the current destination of descriptor 1", and the & is what distinguishes that from a file literally named 1.
In cmd > log 2>&1, stdout moves to log first, so when 2>&1 runs it copies that destination and stderr goes to log too. Both captured. In cmd 2>&1 > log, 2>&1 runs first while stdout still points at the terminal, so stderr is pinned to the terminal, and the later > log moves stdout alone.
# Discard BOTH: send stdout to the sink, THEN merge stderr into it.
some_command > /dev/null 2>&1Silencing works by the identical rule. > /dev/null aims stdout at the sink, then 2>&1 points stderr at stdout's current destination, which is now the sink. Written the other way round as 2>&1 > /dev/null, stderr is tied to the terminal before stdout moves, so errors still print. The capture case and the silence case are one rule, not two, and if you can state why the file version breaks you already know why the silent version does.
DON'T assume 2>&1 looks ahead to where stdout will end up, it only sees the present.
Answer 3:
The shell expands *.csv into matching filenames, and it does so before ls runs, while parsing the command line. ls never sees the *, it receives an already-finished list of real names. That is why globbing works identically with every command, the expansion happens beneath the tool rather than inside it.
A glob selects file paths by name and is a shell feature resolved before the command starts. grep selects lines by content and runs as a command on the text it is handed. One chooses which files, the other chooses which lines, at different stages.
A pipe carries stdout, descriptor 1, into the next command's stdin, descriptor 0. It does not carry stderr, which still goes to your terminal unless you merged it first.
DON'T credit ls with understanding wildcards. DON'T assume the pipe forwards diagnostics along with results.
Answer 4:
build ; deploy runs deploy unconditionally, because ; ignores exit codes entirely, so a failed build still deploys. build && deploy runs deploy only if build exited 0. Confusing them lets failures slip silently down the chain.
Now the line that looks like a check. A pipeline runs several commands but reports only one exit status, and it is the last stage's. Here that is wc, not grep. So the && is asking "did wc finish cleanly", and wc succeeds at counting whether it counted a thousand errors or zero. The word clean prints either way, which makes it a confirmation that confirms nothing you care about.
That is worth sitting with, because the line reads like a test. grep's status, the one that would actually report whether any error was found, was consumed inside the pipeline and never reached the &&.
DON'T assume && reads the status of the command that interests you. DON'T treat ; and && as interchangeable.
Assignment
Assignment: Prove the Trap, Then Build the Report
Part one. Run one command that produces both stdout and stderr, such as searching /etc for a filename, in three forms: (a) the correct > out.log 2>&1, (b) the trap 2>&1 > out.log, and (c) a form that appends both channels onto the end of an existing out.log instead of overwriting it. For (a) and (b), report what ended up in the file and what still printed on your terminal, because the difference shows in both places.
Part two. Write a single line that globs every .log file, filters for error case-insensitively, sorts the matching lines, collapses duplicates while counting how often each occurred, and, only if the pipeline exited cleanly, appends a report built line to audit.log. In comments, identify which stage is the glob, which are the pipes, and which is the chain.
Hints:
- For (a) versus (b), watch two places at once, the file and the terminal. Neither alone tells the whole story.
uniq -conly collapses adjacent duplicates, so thesortmust come first or the counts are wrong whenever identical lines are scattered across files. With several filesgrepalso starts every matching line with its file name, so the same line from two files never counts as a duplicate until-hdrops the names.
Solution:
# (a) CORRECT: stdout to out.log first, THEN stderr merged in.
# out.log ends up with BOTH the found path and the Permission-denied lines.
find /etc -name "hosts" > out.log 2>&1
# (b) TRAP: 2>&1 ties stderr to the terminal (where stdout still points),
# THEN > out.log moves only stdout. out.log gets the result ONLY and the
# errors LEAK to your screen.
find /etc -name "hosts" 2>&1 > out.log
# (c) APPEND both channels: like (a), but '>>' keeps out.log's prior
# contents and adds beneath them instead of truncating first.
find /etc -name "hosts" >> out.log 2>&1
cat out.log # read it after each form and compare against what printedForm (a) captures everything because stdout is already aimed at out.log when 2>&1 runs. Form (b) writes only the result to the file while the Permission denied lines escape to the terminal, because 2>&1 bound stderr to the screen before > out.log moved stdout. Reading the file after each and watching what still printed makes the difference visible in both places at once.
# glob *.log (shell) -> grep -ih filters lines, -h dropping the file names
# -> sort groups equal lines so uniq can see them -> uniq -c collapses
# duplicates and counts each -> && appends a marker to audit.log only if
# the pipeline exited cleanly.
grep -ih "error" *.log | sort | uniq -c && echo "report built" >> audit.log
# glob: *.log, expanded by the shell before grep runs
# pipes: the two | joining grep, sort, and uniq
# chain: && guards the >> append on the pipeline's exit code (uniq's)DON'T treat (a) and (b) as the same command written two ways, the tokens are identical and only the order differs, and that order alone decides whether your errors are captured or lost. DON'T drop the sort before uniq -c, without it the counts are wrong whenever identical lines are scattered through the files. DON'T drop the -h either, without it every line carries its file name and the same error from two files never merges.
Additional Resources
Optional, and nothing later depends on either. They extend past this lesson rather than repeating it.
- GNU Grep Manual - "Regular Expressions" - where
greppatterns go beyond plain words, the natural next step after literal matching. - GNU Coreutils Manual -
uniqinvocation - the companion tool this lesson's assignment uses to tally repeated lines in a pipeline.
Summary
Redirection is nothing more than re-aiming a numbered channel, at a file, at a sink, or at another channel, and once you read every operator as "(which descriptor) goes (where)" the whole topic collapses into one calm idea. > and >> aim stdout at a file, the first wiping it and the second growing it, /dev/null aims a channel at oblivion, and 2>&1 aims stderr at wherever stdout currently goes.
The one place this asks for genuine care is the order trap, and it is worth dwelling on because it is the first time the sequence of symbols silently changes the outcome. command > log 2>&1 and command 2>&1 > log are built from identical pieces and read almost the same aloud, yet one captures everything and the other quietly lets your errors escape to a terminal nobody is reading. It is not a quirk to memorize, it falls directly out of the rule that bash reads redirections left to right.
The pipe then changes what a command is. It stops being a thing you run and watch and becomes a component with an input, an output and a status, designed to be wired to others. grep is one filter you can drop into the stream, wildcards let the shell hand a whole group of files to any tool, the exit code is the small report a finished command leaves behind, and the chaining operators read that report to decide what happens next. None of these is large alone, the power is entirely in the composition.
That composition outlives the specific tools. Small programs that each do one thing and connect cleanly through text is a way of building that scales far past the terminal, and the && you typed today is the seed of every if you will ever write. What you have gained in practical terms is durability: output on a screen is gone the moment it scrolls past, and a full log with results and errors merged in the order they happened is the raw material of every debugging session you will ever run.
Cheat Sheet
Part A - Reference Table
| Operator / Tool | Moves or does | Effect | When to use | When to avoid |
|---|---|---|---|---|
> file |
stdout (1) | overwrite, truncate then write | save a fresh snapshot | the file's contents matter |
>> file |
stdout (1) | append to the end | build a growing log | you want only the latest output |
2> file |
stderr (2) | overwrite with errors only | capture diagnostics separately | you also need results in it |
2>/dev/null |
stderr (2) | discard errors | mute known harmless warnings | an error might be the real failure |
> file 2>&1 |
both to file | results and errors together | a full run log | reversing the order |
2>&1 > file |
TRAP | stdout to file, stderr to terminal | almost never what you want | you meant to capture both |
&> file |
both to file | bash one-token capture-all | quick full capture in bash | non-bash shells, not portable |
| |
stdout to next stdin | wires two processes | feed output into another tool | the right side ignores stdin |
grep PATTERN |
keeps matching lines | filters by content | selecting lines | selecting files, use a glob |
grep -i -v -c -q |
ignore case, invert, count, quiet | tunes the filter | shaping the match | you need text and used -q |
* ? [...] |
shell expands to paths | selects files by name | acting on many files | matching hidden . files |
$? |
exit code of last command | 0 is success |
checking success | after another command ran |
A && B |
B only if A succeeded | guards a dependent step | build then deploy | B must run regardless |
A || B |
B only if A failed | fallback or error message | warn on failure | as a strict else |
A ; B |
B regardless of A | unconditional sequence | independent steps | B depends on A |
Part B - Compact Code Block
ls /home > out.txt # stdout -> file (OVERWRITE, wiped first)
echo "line" >> out.txt # stdout -> file (APPEND, keeps old)
cmd 2>/dev/null # drop errors, keep results on screen
cmd >/dev/null 2>&1 # silence BOTH (sink first, then merge)
# THE ORDER TRAP - same tokens, different result:
cmd > run.log 2>&1 # both captured (aim stdout, THEN merge)
cmd 2>&1 > run.log # errors leak (merge ran too early)
# Rule: redirect stdout FIRST, put 2>&1 LAST.ls /etc | grep conf | wc -l # pipe: list -> filter -> count
grep -i "error" *.log # glob + case-insensitive filter
cmd ; echo $? # run, then read exit status (0 = ok)
grep -q "ok" log && echo yes || echo no # branch on found / not found
# guarded chain
mkdir out && cd out && echo "done" > note.txt