This HN post got me to brush up my awk beyond the basic {print $2,$3} to extract fields from csv-like files. Here are some of my notes from The AWK Programming Language by Alfred Aho, Peter Weinberger and Brian Kernighan.

  • sequence of pattern-action statements pattern { action }
  • 2 types of data: numbers and strings.
  • fields in current input line: $1, $2… $0 is the entire line.
  • if there is no pattern, perform action on every line
  • if there is no action, print lines that match the pattern 
    $ echo -e 'alice 20\nbob 30\niyer 24' | awk '$2>20'  # all people aged >20
bob 30
iyer 24
  • NF - number of fields in current line 
    $ echo 'a b c d' | awk '{print NF, $1, $NF}'  # print field count, the 1st and last field
4 a d
  • NR - line number 
    $ echo -e 'alice 20\nbob 30' | awk '{print NR, $0}'  # prefix with line number
1 alice 20
2 bob 30
  • simple arithmetic 
    $ echo 'alice 5.50 22' | awk '{print $1, $2 * $3}'
alice 121
  • use printf like in C 
    $ echo 'alice 5.50 22' | awk '{printf("%s $%.2f\n", $1, $2 * $3)}'
alice $121.00
  • string/regex matching 
    $ echo -e 'susan 20\nbob 24\nsusie 12' | awk '$1=="susie"'
susie 12
$ echo -e 'susan 20\nbob 24\nsusie 12' | awk '$1 ~ /^su/'  # all lines where the 1st field starts with "su"
susan 20
susie 12

  • use ||, && and ! as in C

  • BEGIN,END - special patterns that matches before first line is read and after last line has been processed. 
    $ echo -e 'susan,20\nbob,24\nsusie,12' | awk 'BEGIN {print "name,age"} {print}'  # add header to csv file
name,age
susan,20
bob,24
susie,12
$ echo -e 'susan 20\nbob 24\nsusie 12' | awk '{sum=sum+$2} END{print sum/NR}' # average age
18.6667
  • if-else, while and for loops similar to C 
    $ cat /tmp/tmp
name,age
Alice,20
Bob,30
$ awk -F, '{for(i=1;i<=NF;i+=1)if(NR==1)a[i]=$i;else a[i]=a[i] FS $i}END{for(i=1;i<=NF;i+=1)print a[i]}' /tmp/tmp
name,Alice,Bob
age,20,30

    The one-liner above transposes the input csv file, expanded here for readability:

    {  # since this action block does NOT have a pattern, it will execute for all lines
  for (i = 1; i <= NF; i += 1)
    if (NR == 1)  # if this is the first line
      a[i] = $i
    else
      a[i] = a[i] FS $i  # concatenate current cell to the ith "column" string separated by the Field Separator
}
  
END { # this action block gets executed after all lines have been processed
  for (i = 1; i <= NF; i += 1)
    print a[i]
}
  • use length for finding string length 
    $ echo -e 'name age\nalice 20\nbob 30' | awk '{print length($0)}'  # print length of each line
8
8
6