Bash Pattern Scanning
awk treats each line of input as a record split into fields, letting you filter, transform, and calculate on structured text in a single compact command.
What Is awk?
awk is a small programming language built for scanning text and acting on patterns it finds. Every line it reads is called a record, and awk automatically splits that record into fields you can reference as $1, $2, $3, and so on, with $0 always meaning the whole line. By default fields are split on whitespace, so for a comma-separated file like team.csv you tell awk to split on commas instead with -F','.
Example
$ cat team.csv
name,department,age,salary
Alice Johnson,Engineering,29,85000
Bob Smith,Marketing,34,62000
carol white,engineering,41,95000
David Lee,Sales,26,54000
Eve Davis,Marketing,38,71000
Frank Miller,Engineering,45,110000
grace hopper,Sales,31,67000
$ awk -F',' '{print $1, $2}' team.csv
name department
Alice Johnson Engineering
Bob Smith Marketing
carol white engineering
David Lee Sales
Eve Davis Marketing
Frank Miller Engineering
grace hopper SalesFiltering Rows with Patterns
awk programs are really pairs of pattern and action: pattern { action }. When a line matches the pattern the action runs, and when a pattern is left out awk assumes true and runs the action on every line. NR holds the number of the current line, so NR>1 is a common way to skip a header row, while a comparison like $4>70000 filters by a field's value.
Example
$ awk -F',' 'NR>1 && $4>70000 {print $1, $4}' team.csv
Alice Johnson 85000
carol white 95000
Eve Davis 71000
Frank Miller 110000Built-in Variables, BEGIN, and END
Beyond NR, awk keeps NF, the number of fields on the current line, and lets you accumulate values across the whole file using ordinary variables. A BEGIN block runs once before any input is read, and an END block runs once after the last line, which makes them the natural place to set up counters and print totals.
Example
$ awk -F',' 'NR>1 {sum+=$4; count++} END {print "Total:", sum; print "Average:", sum/count}' team.csv
Total: 544000
Average: 77714.3- BEGIN and END blocks run once before the first line and once after the last, ideal for printing headers, totals, or setting FS.
- printf gives precise control over column widths and number formatting, letting you align output the way plain print cannot.
- A bare regex pattern works without an explicit field test: awk -F',' '/Sales/' team.csv prints any line containing Sales.
- OFS controls the separator awk uses when it rebuilds a line after you modify a field, which is how you turn a comma-separated file into a pipe-separated one.
- Combine conditions with && and || to filter on several columns at once, such as department and a minimum age together.
