Zum Inhalt

awk command

In 1977, a programming language-level tool for processing text, named' awk', was born at Bell Labs. The name comes from the first letters of the last names of three famous people:

  • Alfred Aho
  • Peter Weinberger
  • Brian Kernighan

Similar to shell (bash, csh, zsh, and ksh), awk has derivatives with the development of history:

  • awk: Born in 1977 Bell Labs.
  • nawk (new awk): It was born in 1985 and is an updated and enhanced version of awk. It was widely used with Unix System V Release 3.1 (1987). The old version of awk is called oawk (old awk).
  • gawk (GNU awk): It was written by Paul Rubin in 1986. The GNU Project was born in 1984.
  • mawk: was written in 1996 by Mike Brennan, the interpreter of the awk programming language.
  • jawk: Implementation of awk in JAVA

In the GNU/Linux operating system, the usual awk refers to gawk. However, some distributions, such as Ubuntu or Debian, use mawk as their default awk.

In the Rocky Linux 8.8, awk refers to gawk.

Shell > whereis awk
awk: /usr/bin/awk /usr/libexec/awk /usr/share/awk /usr/share/man/man1/awk.1.gz

Shell > ls -l /usr/bin/awk
lrwxrwxrwx. 1 root root 4 4月  16 2022 /usr/bin/awk -> gawk

Shell > rpm -qf /usr/bin/awk
gawk-4.2.1-4.el8.x86_64

For information not covered, see the gawk manual.

Although awk is a tool for processing text, it has some programming language features:

  • variable
  • process control (loop)
  • data type
  • logical operation
  • function
  • array
  • ...

The working principle of awk: Similar to relational databases, it supports processing fields (columns) and records (rows). By default, awk treats each line of a file as a record and places these records in memory for line-by-line processing, with a portion of each line treated as a field in the record. By default, delimiters to separate different fields use spaces and tabs, while numbers represent different fields in the row record. To reference multiple fields, separate them with commas or tabs.

A simple example that is easy to understand:

Shell > df -hT
| 1             |     2        |  3    |  4   |  5    |   6   |   7            | 8       |
|Filesystem     |    Type      | Size  | Used | Avail | Use%  | Mounted        | on      |←← 1 (first line)
|devtmpfs       |    devtmpfs  | 1.8G  |   0  | 1.8G  |  0%   | /dev           |         |←← 2
|tmpfs          |    tmpfs     | 1.8G  |    0 | 1.8G  |  0%   | /dev/shm       |         |←← 3
|tmpfs          |    tmpfs     | 1.8G  | 8.9M | 1.8G  |  1%   | /run           |         |←← 4
|tmpfs          |    tmpfs     | 1.8G  |   0  | 1.8G  |  0%   | /sys/fs/cgroup |         |←← 5
|/dev/nvme0n1p2 |    ext4      | 47G   | 2.6G |  42G  |  6%   | /              |         |←← 6
|/dev/nvme0n1p1 |    xfs       | 1014M | 182M | 833M  |  18%  | /boot          |         |←← 7
|tmpfs          |    tmpfs     | 364M  |   0  | 364M  |  0%   | /run/user/0    |         |←← 8  (end line)

Shell > df -hT | awk '{print $1,$2}'
Filesystem  Type
devtmpfs devtmpfs
tmpfs tmpfs
tmpfs tmpfs
tmpfs tmpfs
/dev/nvme0n1p2 ext4
/dev/nvme0n1p1 xfs
tmpfs tmpfs

# $0: Reference the entire text content.
Shell > df -hT | awk '{print $0}'
Filesystem     Type      Size   Used  Avail Use% Mounted on
devtmpfs       devtmpfs  1.8G     0  1.8G    0%  /dev
tmpfs          tmpfs     1.8G     0  1.8G    0%  /dev/shm
tmpfs          tmpfs     1.8G  8.9M  1.8G    1%  /run
tmpfs          tmpfs     1.8G     0  1.8G    0%  /sys/fs/cgroup
/dev/nvme0n1p2 ext4       47G  2.6G   42G    6%  /
/dev/nvme0n1p1 xfs      1014M  182M  833M   18%  /boot
tmpfs          tmpfs     364M     0  364M    0%  /run/user/0

Instructions for using awk

The usage of awk is - awk option 'pattern {action}' FileName

pattern: Find specific content in the text action: Action instruction { }: Group some instructions according to specific patterns

option description
-f program-file
--file program-file
Reading awk program source files from files
-F FS Specify the separator for separating fields. The 'FS' here is a built-in variable in awk, with default values of spaces or tabs
-v var=value variable assignment
--posix Turn on compatibility mode
--dump-variables=[file] Write global variables in awk to a file. If no file is specified, the default file is awkvars.out
--profile=[file] Write performance analysis data to a specific file. If no file is specified, the default file is awkprof.out
pattern description
BEGIN{ } An action that is performed before all row records are read
END{ } An action that is performed after all row records are read
/regular expression/ Match the regular expression for each input line record
pattern && pattern Logic and operation
pattern || pattern Logic or operation
!pattern Logical negation operation
pattern1,pattern2 Specify the pattern range to match all row records within that range

awk is powerful and involves a lot of knowledge, so some of the content will be explained later.

printf commands

Before formally learning awk, beginners need to understand the command printf.

printf:format and print data. Its usage is -printf FORMAT [ARGUMENT]...

FORMAT:Used to control the content of the output. The following common interpretation sequences are supported:

  • \a - alert (BEL)
  • \b - backspace
  • \f - form feed
  • \n - new line
  • \r - carriage return
  • \t - horizontal tab
  • \v - vertical tab
  • %Ns - The output string. The N represents the number of strings, for example: %s %s %s
  • %Ni - Output integers. The N represents the number of integers of the output, for example: %i %i
  • %m.nf - Output Floating Point Number. The m represents the total number of digits output, and the n represents the number of digits after the decimal point. For example: %8.5f

ARGUMENT: If it is a file, you need to do some preprocessing to output correctly.

Shell > cat /tmp/printf.txt
ID      Name    Age     Class
1       Frank   20      3
2       Jack    25      5
3       Django  16      6
4       Tom     19      7

# Example of incorrect syntax:
Shell > printf '%s %s $s\n' /tmp/printf.txt
/tmp/printf.txt

# Change the format of the text
Shell > printf '%s' $(cat /tmp/printf.txt)
IDNameAgeClass1Frank2032Jack2553Django1664Tom197
# Change the format of the text
Shell > printf '%s\t%s\t%s\n' $(cat /tmp/printf.txt)
ID      Name    Age
Class   1       Frank
20      3       2
Jack    25      5
3       Django  16
6       4       Tom
19      7

Shell > printf "%s\t%s\t%s\t%s\n" a b c d 1 2 3 4
a       b       c       d
1       2       3       4

No print command exists in RockyLinux OS. You can only use print in awk, and its difference from printf is that it automatically adds a newline at the end of each line. For example:

Shell > awk '{printf $1 "\t" $2"\n"}' /tmp/printf.txt
ID      Name
1       Frank
2       Jack
3       Django
4       Tom

Shell > awk '{print $1 "\t" $2}' /tmp/printf.txt
ID      Name
1       Frank
2       Jack
3       Django
4       Tom

Basic usage example

  1. Reading awk program source files from files

    Shell > vim /tmp/read-print.awk
    #!/bin/awk
    {print $6}
    
    Shell > df -hT | awk -f /tmp/read-print.awk
    Use%
    0%
    0%
    1%
    0%
    6%
    18%
    0%
    
  2. Specify delimiter

    Shell > awk -F ":" '{print $1}' /etc/passwd
    root
    bin
    daemon
    adm
    lp
    sync
    ...
    
    Shell > tail -n 5 /etc/services | awk -F "\/" '{print $2}'
    awk: warning: escape sequence `\/' treated as plain `/'
    axio-disc       35100
    pmwebapi        44323
    cloudcheck-ping 45514
    cloudcheck      45514
    spremotetablet  46998
    

    You can also use words as delimiters. Parentheses indicate this is an overall delimiter, and "|" means or.

    Shell > tail -n 5 /etc/services | awk -F "(tcp)|(udp)" '{print $1}'
    axio-disc       35100/
    pmwebapi        44323/
    cloudcheck-ping 45514/
    cloudcheck      45514/
    spremotetablet  46998/
    
  3. Variable assignment

    Shell > tail -n 5 /etc/services | awk -v a=123 'BEGIN{print a}{print $1}'
    123
    axio-disc
    pmwebapi
    cloudcheck-ping
    cloudcheck
    spremotetablet
    

    Assign the value of user-defined variables in bash to awk's variables.

    Shell > ab=123
    Shell > echo ${ab}
    123
    Shell > tail -n 5 /etc/services | awk -v a=${ab} 'BEGIN{print a}{print $1}'
    123
    axio-disc
    pmwebapi
    cloudcheck-ping
    cloudcheck
    spremotetablet
    
  4. Write awk's global variables to a file

    Shell > seq 1 6 | awk --dump-variables '{print $0}'
    1
    2
    3
    4
    5
    6
    
    Shell > cat /root/awkvars.out
    ARGC: 1
    ARGIND: 0
    ARGV: array, 1 elements
    BINMODE: 0
    CONVFMT: "%.6g"
    ENVIRON: array, 27 elements
    ERRNO: ""
    FIELDWIDTHS: ""
    FILENAME: "-"
    FNR: 6
    FPAT: "[^[:space:]]+"
    FS: " "
    FUNCTAB: array, 41 elements
    IGNORECASE: 0
    LINT: 0
    NF: 1
    NR: 6
    OFMT: "%.6g"
    OFS: " "
    ORS: "\n"
    PREC: 53
    PROCINFO: array, 20 elements
    RLENGTH: 0
    ROUNDMODE: "N"
    RS: "\n"
    RSTART: 0
    RT: "\n"
    SUBSEP: "\034"
    SYMTAB: array, 28 elements
    TEXTDOMAIN: "messages"
    

    Later, we will introduce what these variables mean. To review them now, jump to variables.

  5. BEGIN{ } and END{ }

    Shell > head -n 5 /etc/passwd | awk 'BEGIN{print "UserName:PasswordIdentification:UID:InitGID"}{print $0}END{print "one\ntwo"}'
    UserName:PasswordIdentification:UID:InitGID
    root:x:0:0:root:/root:/bin/bash
    bin:x:1:1:bin:/bin:/sbin/nologin
    daemon:x:2:2:daemon:/sbin:/sbin/nologin
    adm:x:3:4:adm:/var/adm:/sbin/nologin
    lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
    one
    two
    
  6. --profile option

    Shell > df -hT | awk --profile 'BEGIN{print "start line"}{print $0}END{print "end line"}'
    start line
    Filesystem     Type      Size  Used Avail Use% Mounted on
    devtmpfs       devtmpfs  1.8G     0  1.8G   0% /dev
    tmpfs          tmpfs     1.8G     0  1.8G   0% /dev/shm
    tmpfs          tmpfs     1.8G  8.9M  1.8G   1% /run
    tmpfs          tmpfs     1.8G     0  1.8G   0% /sys/fs/cgroup
    /dev/nvme0n1p2 ext4       47G  2.7G   42G   6% /
    /dev/nvme0n1p1 xfs      1014M  181M  834M  18% /boot
    tmpfs          tmpfs     363M     0  363M   0% /run/user/0
    end line
    
    Shell > cat /root/awkprof.out
        # gawk profile, created Fri Dec  8 15:12:56 2023
    
        # BEGIN rule(s)
    
        BEGIN {
     1          print "start line"
        }
    
        # Rule(s)
    
     8  {
     8          print $0
        }
    
        # END rule(s)
    
        END {
     1          print "end line"
        }
    

    Modify the awkprof.out file.

    Shell > vim /root/awkprof.out
    BEGIN {
        print "start line"
    }
    
    {
        print $0
    }
    
    END {
        print "end line"
    }
    
    Shell > df -hT | awk -f /root/awkprof.out
    start line
    Filesystem     Type      Size  Used Avail Use% Mounted on
    devtmpfs       devtmpfs  1.8G     0  1.8G   0% /dev
    tmpfs          tmpfs     1.8G     0  1.8G   0% /dev/shm
    tmpfs          tmpfs     1.8G  8.9M  1.8G   1% /run
    tmpfs          tmpfs     1.8G     0  1.8G   0% /sys/fs/cgroup
    /dev/nvme0n1p2 ext4       47G  2.7G   42G   6% /
    /dev/nvme0n1p1 xfs      1014M  181M  834M  18% /boot
    tmpfs          tmpfs     363M     0  363M   0% /run/user/0
    end line
    
  7. Match rows (records) through regular expressions

    Shell > cat /etc/services | awk '/[^0-9a-zA-Z]1[1-9]{2}\/tcp/ {print $0}'
    sunrpc          111/tcp         portmapper rpcbind      # RPC 4.0 portmapper TCP
    auth            113/tcp         authentication tap ident
    sftp            115/tcp
    uucp-path       117/tcp
    nntp            119/tcp         readnews untp   # USENET News Transfer Protocol
    ntp             123/tcp
    netbios-ns      137/tcp                         # NETBIOS Name Service
    netbios-dgm     138/tcp                         # NETBIOS Datagram Service
    netbios-ssn     139/tcp                         # NETBIOS session service
    ...
    
  8. Logical operations (logical and, logical OR, reverse)

    logical and: && logical OR: || reverse: !

    Shell > cat /etc/services | awk '/[^0-9a-zA-Z]1[1-9]{2}\/tcp/ && /175/ {print $0}'
    vmnet           175/tcp                 # VMNET
    
    Shell > cat /etc/services | awk '/[^0-9a-zA-Z]9[1-9]{2}\/tcp/ || /91{2}\/tcp/ {print $0}'
    telnets         992/tcp
    imaps           993/tcp                         # IMAP over SSL
    pop3s           995/tcp                         # POP-3 over SSL
    mtp             1911/tcp                        #
    rndc            953/tcp                         # rndc control sockets (BIND 9)
    xact-backup     911/tcp                 # xact-backup
    apex-mesh       912/tcp                 # APEX relay-relay service
    apex-edge       913/tcp                 # APEX endpoint-relay service
    ftps-data       989/tcp                 # ftp protocol, data, over TLS/SSL
    nas             991/tcp                 # Netnews Administration System
    vsinet          996/tcp                 # vsinet
    maitrd          997/tcp                 #
    busboy          998/tcp                 #
    garcon          999/tcp                 #
    #puprouter      999/tcp                 #
    blockade        2911/tcp                # Blockade
    prnstatus       3911/tcp                # Printer Status Port
    cpdlc           5911/tcp                # Controller Pilot Data Link Communication
    manyone-xml     8911/tcp                # manyone-xml
    sype-transport  9911/tcp                # SYPECom Transport Protocol
    
    Shell > cat /etc/services | awk '!/(tcp)|(udp)/ {print $0}'
    discard         9/sctp                  # Discard
    discard         9/dccp                  # Discard SC:DISC
    ftp-data        20/sctp                 # FTP
    ftp             21/sctp                 # FTP
    ssh             22/sctp                 # SSH
    exp1            1021/sctp                # RFC3692-style Experiment 1 (*)                [RFC4727]
    exp1            1021/dccp                # RFC3692-style Experiment 1 (*)                [RFC4727]
    exp2            1022/sctp                # RFC3692-style Experiment 2 (*)                [RFC4727]
    exp2            1022/dccp                # RFC3692-style Experiment 2 (*)                [RFC4727]
    ltp-deepspace   1113/dccp               # Licklider Transmission Protocol
    cisco-ipsla     1167/sctp               # Cisco IP SLAs Control Protocol
    rcip-itu        2225/sctp               # Resource Connection Initiation Protocol
    m2ua            2904/sctp               # M2UA
    m3ua            2905/sctp               # M3UA
    megaco-h248     2944/sctp               # Megaco-H.248 text
    ...
    
  9. Locates consecutive lines by string and prints them

    Shell > cat /etc/services | awk '/^ntp/,/^netbios/ {print $0}'
    ntp             123/tcp
    ntp             123/udp                         # Network Time Protocol
    netbios-ns      137/tcp                         # NETBIOS Name Service
    

    Info

    Start range: stop matching when the first match is encountered. End range: stop matching when the first match is encountered.

Built-in variable

Variable name Description
FS The delimiter of the input field. The default is space or tab
OFS The delimiter of the output field. The default is space
RS The delimiter of the input row record. The default is a newline character (\n)
ORS The delimiter of output row record. The default is a newline character (\n)
NF Count the number of fields in the current row record
NR Count the number of row records. After each line of text is processed, the value of this variable will be +1
FNR Count the number of row records. When the second file is processed, the NR variable continues to add up, but the FNR variable is recounted
ARGC The number of command line arguments
ARGV An array of command line arguments, with subscript starting at 0 and ARGV[0] representing awk
ARGIND The index value of the file currently being processed. The first file is 1, the second file is 2, and so on
ENVIRON Environment variables of the current system
FILENAME Output the currently processed file name
IGNORECASE Ignore case
SUBSEP The delimiter of the subscript in the array, which defaults to "\034"
  1. FS and OFS

    Shell > cat /etc/passwd | awk 'BEGIN{FS=":"}{print $1}'
    root
    bin
    daemon
    adm
    lp
    sync
    

    You can also use the -v option to assign values to variables.

    Shell > cat /etc/passwd | awk -v FS=":" '{print $1}'
    root
    bin
    daemon
    adm
    lp
    sync
    

    The default output delimiter is a space when using commas to reference multiple fields. You can, however, specify the output delimiter separately.

    Shell > cat /etc/passwd | awk 'BEGIN{FS=":"}{print $1,$2}'
    root x
    bin x
    daemon x
    adm x
    lp x
    
    Shell > cat /etc/passwd | awk 'BEGIN{FS=":";OFS="\t"}{print $1,$2}'
    # or
    Shell > cat /etc/passwd | awk -v FS=":" -v OFS="\t" '{print $1,$2}'
    root    x
    bin     x
    daemon  x
    adm     x
    lp      x
    
  2. RS and ORS

    By default, awk uses newline characters to distinguish each line record

    Shell > echo -e "https://example.com/books/index.html\ntitle//tcp"
    https://example.com/books/index.html
    title//tcp
    
    Shell > echo -e "https://example.com/books/index.html\ntitle//tcp" | awk 'BEGIN{RS="\/\/";ORS="%%"}{print $0}'
    awk: cmd. line:1: warning: escape sequence `\/' treated as plain `/'
    https:%%example.com/books/index.html
    title%%tcp
    %%              Why? Because "print"
    
  3. NF

    Count the number of fields per line in the current text

    Shell > head -n 5 /etc/passwd | awk -F ":" 'BEGIN{RS="\n";ORS="\n"} {print NF}'
    7
    7
    7
    7
    7
    

    Print the fifth field

    Shell > head -n 5 /etc/passwd | awk -F ":" 'BEGIN{RS="\n";ORS="\n"} {print $(NF-2)}'
    root
    bin
    daemon
    adm
    lp
    

    Print the last field

    Shell > head -n 5 /etc/passwd | awk -F ":" 'BEGIN{RS="\n";ORS="\n"} {print $NF}'
    /bin/bash
    /sbin/nologin
    /sbin/nologin
    /sbin/nologin
    /sbin/nologin
    

    Exclude the last two fields

    Shell > head -n 5 /etc/passwd | awk -F ":" 'BEGIN{RS="\n";ORS="\n"} {$NF=" ";$(NF-1)=" ";print $0}'
    root x 0 0 root
    bin x 1 1 bin
    daemon x 2 2 daemon
    adm x 3 4 adm
    lp x 4 7 lp
    

    Exclude the first field

    Shell > head -n 5 /etc/passwd | awk -F ":" 'BEGIN{RS="\n";ORS="\n"} {$1=" ";print $0}' | sed -r 's/(^  )//g'
    x 0 0 root /root /bin/bash
    x 1 1 bin /bin /sbin/nologin
    x 2 2 daemon /sbin /sbin/nologin
    x 3 4 adm /var/adm /sbin/nologin
    x 4 7 lp /var/spool/lpd /sbin/nologin
    
  4. NR and FNR

    Shell > tail -n 5 /etc/services | awk '{print NR,$0}'
    1 axio-disc       35100/udp               # Axiomatic discovery protocol
    2 pmwebapi        44323/tcp               # Performance Co-Pilot client HTTP API
    3 cloudcheck-ping 45514/udp               # ASSIA CloudCheck WiFi Management keepalive
    4 cloudcheck      45514/tcp               # ASSIA CloudCheck WiFi Management System
    5 spremotetablet  46998/tcp               # Capture handwritten signatures
    

    Print the total number of lines in the file content

    Shell > cat /etc/services | awk 'END{print NR}'
    11473
    

    Print the content of line 200

    Shell > cat /etc/services | awk 'NR==200'
    microsoft-ds    445/tcp
    

    Print the second field on line 200

    Shell > cat /etc/services | awk 'BEGIN{RS="\n";ORS="\n"} NR==200 {print $2}'
    445/tcp
    

    Print content within a specific range

    Shell > cat /etc/services | awk 'BEGIN{RS="\n";ORS="\n"} NR<=10 {print NR,$0}'
    1 # /etc/services:
    2 # $Id: services,v 1.49 2017/08/18 12:43:23 ovasik Exp $
    3 #
    4 # Network services, Internet style
    5 # IANA services version: last updated 2016-07-08
    6 #
    7 # Note that it is presently the policy of IANA to assign a single well-known
    8 # port number for both TCP and UDP; hence, most entries here have two entries
    9 # even if the protocol doesn't support UDP operations.
    10 # Updated from RFC 1700, ``Assigned Numbers'' (October 1994).  Not all ports
    

    Comparison between NR and FNR

    Shell > head -n 3 /etc/services > /tmp/a.txt
    
    Shell > cat /tmp/a.txt
    # /etc/services:
    # $Id: services,v 1.49 2017/08/18 12:43:23 ovasik Exp $
    #
    
    Shell > cat /etc/resolv.conf
    # Generated by NetworkManager
    nameserver 8.8.8.8
    nameserver 114.114.114.114
    
    Shell > awk '{print NR,$0}' /tmp/a.txt /etc/resolv.conf
    1 # /etc/services:
    2 # $Id: services,v 1.49 2017/08/18 12:43:23 ovasik Exp $
    3 #
    4 # Generated by NetworkManager
    5 nameserver 8.8.8.8
    6 nameserver 114.114.114.114
    
    Shell > awk '{print FNR,$0}' /tmp/a.txt /etc/resolv.conf
    1 # /etc/services:
    2 # $Id: services,v 1.49 2017/08/18 12:43:23 ovasik Exp $
    3 #
    1 # Generated by NetworkManager
    2 nameserver 8.8.8.8
    3 nameserver 114.114.114.114
    
  5. ARGC and ARGV

    Shell > awk 'BEGIN{print ARGC}' log dump long
    4
    Shell > awk 'BEGIN{print ARGV[0]}' log dump long
    awk
    Shell > awk 'BEGIN{print ARGV[1]}' log dump long
    log
    Shell > awk 'BEGIN{print ARGV[2]}' log dump long
    dump
    
  6. ARGIND

    This variable is mainly used to determine the file the awk program is working on.

    Shell > awk '{print ARGIND,$0}' /etc/hostname /etc/resolv.conf
    1 Master
    2 # Generated by NetworkManager
    2 nameserver 8.8.8.8
    2 nameserver 114.114.114.114
    
  7. ENVIRON

    You can reference operating systems or user-defined variables in awk programs.

    Shell > echo ${SSH_CLIENT}
    192.168.100.2 6969 22
    
    Shell > awk 'BEGIN{print ENVIRON["SSH_CLIENT"]}'
    192.168.100.2 6969 22
    
    Shell > export a=123
    Shell > env | grep -w a
    a=123
    Shell > awk 'BEGIN{print ENVIRON["a"]}'
    123
    Shell > unset a
    
  8. FILENAME

    Shell > awk 'BEGIN{RS="\n";ORS="\n"} NR=FNR {print ARGIND,FILENAME"---"$0}' /etc/hostname /etc/resolv.conf /etc/rocky-release
    1 /etc/hostname---Master
    2 /etc/resolv.conf---# Generated by NetworkManager
    2 /etc/resolv.conf---nameserver 8.8.8.8
    2 /etc/resolv.conf---nameserver 114.114.114.114
    3 /etc/rocky-release---Rocky Linux release 8.9 (Green Obsidian)
    
  9. IGNORECASE

    This variable is useful if you want to use regular expressions in awk and ignore case.

    Shell > awk 'BEGIN{IGNORECASE=1;RS="\n";ORS="\n"} /^(SSH)|^(ftp)/ {print $0}' /etc/services
    ftp-data        20/tcp
    ftp-data        20/udp
    ftp             21/tcp
    ftp             21/udp          fsp fspd
    ssh             22/tcp                          # The Secure Shell (SSH) Protocol
    ssh             22/udp                          # The Secure Shell (SSH) Protocol
    ftp-data        20/sctp                 # FTP
    ftp             21/sctp                 # FTP
    ssh             22/sctp                 # SSH
    ftp-agent       574/tcp                 # FTP Software Agent System
    ftp-agent       574/udp                 # FTP Software Agent System
    sshell          614/tcp                 # SSLshell
    sshell          614/udp                 #       SSLshell
    ftps-data       989/tcp                 # ftp protocol, data, over TLS/SSL
    ftps-data       989/udp                 # ftp protocol, data, over TLS/SSL
    ftps            990/tcp                 # ftp protocol, control, over TLS/SSL
    ftps            990/udp                 # ftp protocol, control, over TLS/SSL
    ssh-mgmt        17235/tcp               # SSH Tectia Manager
    ssh-mgmt        17235/udp               # SSH Tectia Manager
    
    Shell > awk 'BEGIN{IGNORECASE=1;RS="\n";ORS="\n"} /^(SMTP)\s/,/^(TFTP)\s/ {print $0}' /etc/services
    smtp            25/tcp          mail
    smtp            25/udp          mail
    time            37/tcp          timserver
    time            37/udp          timserver
    rlp             39/tcp          resource        # resource location
    rlp             39/udp          resource        # resource location
    nameserver      42/tcp          name            # IEN 116
    nameserver      42/udp          name            # IEN 116
    nicname         43/tcp          whois
    nicname         43/udp          whois
    tacacs          49/tcp                          # Login Host Protocol (TACACS)
    tacacs          49/udp                          # Login Host Protocol (TACACS)
    re-mail-ck      50/tcp                          # Remote Mail Checking Protocol
    re-mail-ck      50/udp                          # Remote Mail Checking Protocol
    domain          53/tcp                          # name-domain server
    domain          53/udp
    whois++         63/tcp          whoispp
    whois++         63/udp          whoispp
    bootps          67/tcp                          # BOOTP server
    bootps          67/udp
    bootpc          68/tcp          dhcpc           # BOOTP client
    bootpc          68/udp          dhcpc
    tftp            69/tcp
    

Operator

Operator Description
(...) Grouping
$n Field reference
++
--
Incremental
Decreasing
+
-
!
Mathematical plus sign
Mathematical minus sign
Negation
*
/
%
Mathematical multiplication sign
Mathematical division sign
Modulo operation
in Elements in an array
&&
||
Logic and Operations
Logical OR operation
?: Abbreviation of conditional expressions
~ Another representation of regular expressions
!~ Reverse Regular Expression

Note

In the awk program, the following expressions will be judged as false:

  • The number is 0;
  • Empty string;
  • Undefined value.
Shell > awk 'BEGIN{n=0;if(n) print "Ture";else print "False"}'
False
Shell > awk 'BEGIN{s="";if(s) print "True";else print "False"}'
False
Shell > awk 'BEGIN{if(t) print "True";else print "Flase"}'
False
  1. Exclamation point

    Print odd rows:

    Shell > seq 1 10 | awk 'i=!i {print $0}'
    1
    3
    5
    7
    9
    

    Question

    Why? Read the first line: Because "i" is not assigned a value, so "i=!i" indicates TRUE. Read the second line: At this point, "i=!i" indicates FALSE. And so on, the final printed line is an odd number.

    Print even rows:

    Shell > seq 1 10 | awk '!(i=!i)'
    # or
    Shell > seq 1 10 | awk '!(i=!i) {print $0}'
    2
    4
    6
    8
    10
    

    Note

    As you can see, sometimes you can ignore the syntax for the "action" part, which by default is equivalent to "{print $0}".

  2. Reversal

    Shell > cat /etc/services | awk '!/(tcp)|(udp)|(^#)|(^$)/ {print $0}'
    http            80/sctp                         # HyperText Transfer Protocol
    bgp             179/sctp
    https           443/sctp                        # http protocol over TLS/SSL
    h323hostcall    1720/sctp                       # H.323 Call Control
    nfs             2049/sctp       nfsd shilp      # Network File System
    rtmp            1/ddp                           # Routing Table Maintenance Protocol
    nbp             2/ddp                           # Name Binding Protocol
    echo            4/ddp                           # AppleTalk Echo Protocol
    zip             6/ddp                           # Zone Information Protocol
    discard         9/sctp                  # Discard
    discard         9/dccp                  # Discard SC:DISC
    ...
    
  3. Basic operations in mathematics

    Shell > echo -e "36\n40\n50" | awk '{print $0+1}'
    37
    41
    
    Shell > echo -e "30\t5\t8\n11\t20\t34"
    30      5       8
    11      20      34
    Shell > echo -e "30\t5\t8\n11\t20\t34" | awk '{print $2*2+1}'
    11
    41
    

    It can also be used in the "pattern":

    Shell > cat -n /etc/services | awk  '/^[1-9]*/ && $1%2==0 {print $0}'
    ...
    24  tcpmux          1/udp                           # TCP port service multiplexer
    26  rje             5/udp                           # Remote Job Entry
    28  echo            7/udp
    30  discard         9/udp           sink null
    32  systat          11/udp          users
    34  daytime         13/udp
    36  qotd            17/udp          quote
    ...
    
    Shell > cat -n /etc/services | awk  '/^[1-9]*/ && $1%2!=0 {print $0}'
    ...
    23  tcpmux          1/tcp                           # TCP port service multiplexer
    25  rje             5/tcp                           # Remote Job Entry
    27  echo            7/tcp
    29  discard         9/tcp           sink null
    31  systat          11/tcp          users
    ...
    
  4. Pipe symbol

    You can use the bash command in the awk program, for example:

    Shell > echo -e "6\n3\n9\n8" | awk '{print $0 | "sort"}'
    3
    6
    8
    9
    

    Info

    Please pay attention! You must use double quotes to include the command.

  5. Regular expression

    Here, we cover basic examples of regular expressions. You can use regular expressions on row records.

    Shell > cat /etc/services | awk '/[^0-9a-zA-Z]1[1-9]{2}\/tcp/ {print $0}'
    
    # Be equivalent to:
    
    Shell > cat /etc/services | awk '$0~/[^0-9a-zA-Z]1[1-9]{2}\/tcp/ {print $0}'
    

    If the file has a large amount of text, regular expressions can also be used for fields, which will help improve processing efficiency. The usage example is as follows:

    Shell > cat /etc/services | awk '$0~/^(ssh)/ && $2~/tcp/ {print $0}'
    ssh             22/tcp                          # The Secure Shell (SSH) Protocol
    sshell          614/tcp                 # SSLshell
    ssh-mgmt        17235/tcp               # SSH Tectia Manager
    
    Shell > cat /etc/services | grep -v -E "(^#)|(^$)" | awk '$2!~/(tcp)|(udp)/ {print $0}'
    http            80/sctp                         # HyperText Transfer Protocol
    bgp             179/sctp
    https           443/sctp                        # http protocol over TLS/SSL
    h323hostcall    1720/sctp                       # H.323 Call Control
    nfs             2049/sctp       nfsd shilp      # Network File System
    rtmp            1/ddp                           # Routing Table Maintenance Protocol
    nbp             2/ddp                           # Name Binding Protocol
    ...
    

Flow control

  1. if statement

    The basic syntax format is - if (condition) statement [ else statement ]

    Example of a single branch use of an if statement:

    Shell > cat /etc/services | awk '{if(NR==110) print $0}'
    pop3            110/udp         pop-3
    

    The condition is determined as a regular expression:

    Shell > cat /etc/services | awk '{if(/^(ftp)\s|^(ssh)\s/) print $0}'
    ftp             21/tcp
    ftp             21/udp          fsp fspd
    ssh             22/tcp                          # The Secure Shell (SSH) Protocol
    ssh             22/udp                          # The Secure Shell (SSH) Protocol
    ftp             21/sctp                 # FTP
    ssh             22/sctp                 # SSH
    

    Double branch:

    Shell > seq 1 10 | awk '{if($0==10) print $0 ; else print "False"}'
    False
    False
    False
    False
    False
    False
    False
    False
    False
    10
    

    Multiple branches:

    Shell > cat /etc/services | awk '{ \
    if($1~/netbios/)
        {print $0}
    else if($2~/175/)
        {print "175"}
    else if($2~/137/)
        {print "137"}
    else {print "no"}
    }'
    
  2. while statement

    The basic syntax format is - while (condition) statement

    Traverse and print out the fields of all row records.

    Shell > tail -n 2 /etc/services
    cloudcheck      45514/tcp               # ASSIA CloudCheck WiFi Management System
    spremotetablet  46998/tcp               # Capture handwritten signatures
    
    Shell > tail -n 2 /etc/services | awk '{ \
    i=1;
    while(i<=NF){print $i;i++}
    }'
    
    cloudcheck
    45514/tcp
    #
    ASSIA
    CloudCheck
    WiFi
    Management
    System
    spremotetablet
    46998/tcp
    #
    Capture
    handwritten
    signatures
    
  3. for statement

    The basic syntax format is - for (expr1; expr2; expr3) statement

    Traverse and print out the fields of all row records.

    Shell > tail -n 2 /etc/services | awk '{ \
    for(i=1;i<=NF;i++) print $i
    }'
    

    Print the fields for each row of records in reverse order.

    Shell > tail -n 2 /etc/services | awk '{ \
    for(i=NF;i>=1;i--) print $i
    }'
    
    System
    Management
    WiFi
    CloudCheck
    ASSIA
    #
    45514/tcp
    cloudcheck
    signatures
    handwritten
    Capture
    #
    46998/tcp
    spremotetablet
    

    Print each line of records in the opposite direction.

    Shell > tail -n 2 /etc/services | awk  '{ \
    for(i=NF;i>=1;i--) {printf $i" "};
    print ""
    }'
    
    System Management WiFi CloudCheck ASSIA # 45514/tcp cloudcheck
    signatures handwritten Capture # 46998/tcp spremotetablet
    
  4. break statement and continue statement

    The comparison between the two is as follows:

    Shell > awk 'BEGIN{  \
    for(i=1;i<=10;i++)
      {
        if(i==3) {break};
        print i
      }
    }'
    
    1
    2
    
    Shell > awk 'BEGIN{  \
    for(i=1;i<=10;i++)
      {
        if(i==3) {continue};
        print i
      }
    }'
    
    1
    2
    4
    5
    6
    7
    8
    9
    10
    
  5. exit statement

    You can specify a return value in the range of [0,255]

    The basic syntax format is - exit [expression]

    Shell > seq 1 10 | awk '{
      if($0~/5/) exit "135"
    }'
    
    Shell > echo $?
    135
    

Array

array: A collection of data with the same data type arranged in a certain order. Each data in an array is called an element.

Like most programming languages, awk also supports arrays, which are divided into indexed arrays (with numbers as subscripts) and associative arrays (with strings as subscripts).

awk has a lot of functions, and the functions related to arrays are:

  • length(Array_Name) - Get the length of the array.

  • Custom array

    Format - Array_Name[Index]=Value

    Shell > awk 'BEGIN{a1[0]="test0" ; a1[1]="s1"; print a1[0]}'
    test0
    

    Get the length of the array:

    Shell > awk 'BEGIN{name[-1]="jimcat8" ; name[3]="jack" ; print length(name)}'
    2
    

    Store all GNU/Linux users in an array:

    Shell > cat /etc/passwd | awk -F ":" '{username[NR]=$1}END{print username[2]}'
    bin
    Shell > cat /etc/passwd | awk -F ":" '{username[NR]=$1}END{print username[1]}'
    root
    

    Info

    The numeric subscript of an awk array can be a positive integer, a negative integer, a string, or 0, so the numeric subscript of an awk array has no concept of an initial value. This is not the same as arrays in bash.

    Shell > arr1=(2 10 30 string1)
    Shell > echo "${arr1[0]}"
    2
    Shell > unset arr1
    
  • Delete array

    Format - delete Array_Name

  • Delete an element from an array

    Format - delete Array_Name[Index]

  • Traversal array

    You can use the for statement, which is suitable for cases where the array subscript is unknown:

    Shell > head -n 5 /etc/passwd | awk -F ":" ' \
    {
      username[NR]=$1
    }
    END {
      for(i in username)
      print username[i],i
    }
    '
    
    root 1
    bin 2
    daemon 3
    adm 4
    lp 5
    

    If the subscript of an array is regular, you can use this form of the for statement:

    Shell > cat /etc/passwd | awk -F ":" ' \
    {
      username[NR]=$1
    }
    END{
      for(i=1;i<=NR;i++)
      print username[i],i
    }
    '
    
    root 1
    bin 2
    daemon 3
    adm 4
    lp 5
    sync 6
    shutdown 7
    halt 8
    ...
    
  • Use "++" as the subscript of the array

    Shell > tail -n 5 /etc/group | awk -F ":" '\
    {
      a[x++]=$1
    }
    END{
      for(i in a)
      print a[i],i
    }
    '
    
    slocate 0
    unbound 1
    docker 2
    cgred 3
    redis 4
    
  • Use a field as the subscript of an array

    Shell > tail -n 5 /etc/group | awk -F ":" '\
    {
      a[$1]=$3
    }
    END{
      for(i in a)
      print a[i],i
    }
    '
    
    991 docker
    21 slocate
    989 redis
    992 unbound
    990 cgred
    
  • Count the number of occurrences of the same field

    Count the number of occurrences of the same IPv4 address. Basic idea:

    • First use the grep command to filter out all IPv4 addresses
    • Then hand it over to the awk program for processing
    Shell > cat /var/log/secure | egrep -o "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" | awk ' \
    {
      a[$1]++
    }
    END{
      for(v in a) print a[v],v
    }
    '
    
    4 0.0.0.0
    4 192.168.100.2
    

    Info

    a[$1]++ is equivalent to a[$1]+=1

    Count the number of occurrences of words regardless of case. Basic idea:

    • Split all fields into multiple rows of records
    • Then hand it over to the awk program for processing
    Shell > cat /etc/services | awk -F " " '{for(i=1;i<=NF;i++) print $i}'
    
    Shell > cat /etc/services | awk -F " " '{for(i=1;i<=NF;i++) print $i}' | awk '\
    BEGIN{IGNORECASE=1;OFS="\t"} /^netbios$/  ||  /^ftp$/  {a[$1]++}  END{for(v in a) print a[v],v}
    '
    
    3       NETBIOS
    18      FTP
    7       ftp
    
    Shell > cat /etc/services | awk -F " " '{ for(i=1;i<=NF;i++) print $i }' | awk '\
    BEGIN{IGNORECASE=1;OFS="\t"}  /^netbios$/  ||  /^ftp$/   {a[$1]++}  END{for(v in a)  \
    if(a[v]>=5) print a[v],v}
    '
    
    18      FTP
    7       ftp
    

    You can first filter specific row records and then perform statistics, such as:

    Shell > ss -tulnp | awk -F " "  '/tcp/ {a[$2]++} END{for(i in a) print a[i],i}'
    2 LISTEN
    
  • Print lines based on the number of occurrences of a specific field

    Shell > tail /etc/services
    aigairserver    21221/tcp               # Services for Air Server
    ka-kdp          31016/udp               # Kollective Agent Kollective Delivery
    ka-sddp         31016/tcp               # Kollective Agent Secure Distributed Delivery
    edi_service     34567/udp               # dhanalakshmi.org EDI Service
    axio-disc       35100/tcp               # Axiomatic discovery protocol
    axio-disc       35100/udp               # Axiomatic discovery protocol
    pmwebapi        44323/tcp               # Performance Co-Pilot client HTTP API
    cloudcheck-ping 45514/udp               # ASSIA CloudCheck WiFi Management keepalive
    cloudcheck      45514/tcp               # ASSIA CloudCheck WiFi Management System
    spremotetablet  46998/tcp               # Capture handwritten signatures
    
    Shell > tail /etc/services | awk 'a[$1]++ {print $0}'
    axio-disc       35100/udp               # Axiomatic discovery protocol
    

    Reverse:

    Shell > tail /etc/services | awk '!a[$1]++ {print $0}'
    aigairserver    21221/tcp               # Services for Air Server
    ka-kdp          31016/udp               # Kollective Agent Kollective Delivery
    ka-sddp         31016/tcp               # Kollective Agent Secure Distributed Delivery
    edi_service     34567/udp               # dhanalakshmi.org EDI Service
    axio-disc       35100/tcp               # Axiomatic discovery protocol
    pmwebapi        44323/tcp               # Performance Co-Pilot client HTTP API
    cloudcheck-ping 45514/udp               # ASSIA CloudCheck WiFi Management keepalive
    cloudcheck      45514/tcp               # ASSIA CloudCheck WiFi Management System
    spremotetablet  46998/tcp               # Capture handwritten signatures
    
  • Multidimensional array

    The awk program does not support multi-dimensional arrays, but support for multi-dimensional arrays is achievable through simulation. By default, "\034" is the delimiter for the subscript of a multidimensional array.

    Please note the following differences when using multidimensional arrays:

    Shell > awk 'BEGIN{ a["1,0"]=100 ; a[2,0]=200 ; a["3","0"]=300 ; for(i in a) print a[i],i }'
    200 20
    300 30
    100 1,0
    

    Redefine the delimiter:

    Shell > awk 'BEGIN{ SUBSEP="----" ; a["1,0"]=100 ; a[2,0]=200 ; a["3","0"]=300 ; for(i in a) print a[i],i }'
    300 3----0
    200 2----0
    100 1,0
    

    Reorder:

    Shell > awk 'BEGIN{ SUBSEP="----" ; a["1,0"]=100 ; a[2,0]=200 ; a["3","0"]=300 ; for(i in a) print a[i],i | "sort" }'
    100 1,0
    200 2----0
    300 3----0
    

    Count the number of times the field appears:

    Shell > cat c.txt
    A 192.168.1.1 HTTP
    B 192.168.1.2 HTTP
    B 192.168.1.2 MYSQL
    C 192.168.1.1 MYSQL
    C 192.168.1.1 MQ
    D 192.168.1.4 NGINX
    
    Shell > cat c.txt | awk 'BEGIN{SUBSEP="----"} {a[$1,$2]++} END{for(i in a) print a[i],i}'
    1 A----192.168.1.1
    2 B----192.168.1.2
    2 C----192.168.1.1
    1 D----192.168.1.4
    

Built-in function

Function name Description
int(expr) Truncate as an integer
sqrt(expr) Square root
rand() Returns a random number N with a range of (0,1). The result is not that every run is a random number, but that it remains the same.
srand([expr]) Use "expr" to generate random numbers. If "expr" is not specified, the current time is used as the seed by default, and if there is a seed, the generated random number is used.
asort(a,b) The elements of the array "a" are reordered (lexicographically) and stored in the new array "b", with the subscript in the array "b" starting at 1. This function returns the number of elements in the array.
asorti(a,b) Reorder the subscript of the array "a" and store the sorted subscript in the new array "b" as an element, with the subscript of the array "b" starting at 1.
sub(r,s[,t]) Use the "r" regular expression to match the input records, and replace the matching result with "s". "t" is optional, indicating a replacement for a certain field. The function returns the number of replacements - 0 or 1. Similar to sed s//
gsub(r,s[,t]) Global replacement. "t" is optional, indicating the replacement of a certain field. If "t" is ignored, it indicates global replacement. Similar to sed s///g
gensub(r,s,h[,t]) The "r" regular expression matches the input records and replaces the matching result with "s". "t" is optional, indicating a replacement for a certain field. "h" represents replacing the specified index position
index(s,t) Returns the index position of the string "t" in the string "s" (the string index starts from 1). If the function returns 0, it means it does not exist
length([s]) Returns the length of "s"
match(s,r[,a]) Test whether the string "s" contains the string "r". If included, return the index position of "r" within it (string index starting from 1). If not, return 0
split(s,a[,r[,seps]]) Split string "s" into an array "a" based on the delimiter "seps". The subscript of the array starts with 1.
substr(s,i[,n]) Intercept the string. "s" represents the string to be processed; "i" indicates the index position of the string; "n" is the length. If you do not specify "n", it means to intercept all remaining parts
tolower(str) Converts all strings to lowercase
toupper(str) Converts all strings to uppercase
systime() Current timestamp
strftime([format[,timestamp[,utc-flag]]]) Format the output time. Converts the timestamp to a string
  1. int function

    Shell > echo -e "qwer123\n123\nabc\n123abc123\n100.55\n-155.27"
    qwer123
    123
    abc
    123abc123
    100.55
    -155.27
    
    Shell > echo -e "qwer123\n123\nabc\n123abc123\n100.55\n-155.27" | awk '{print int($1)}'
    0
    123
    0
    123
    100
    -155
    

    As you can see, the int function only works for numbers, and when encountering a string, converts it to 0. When encountering a string starting with a number, truncates it.

  2. sqrt function

    Shell > awk 'BEGIN{print sqrt(9)}'
    3
    
  3. rand function and srand function

    The example of using the rand function is as follows:

    Shell > awk 'BEGIN{print rand()}'
    0.924046
    Shell > awk 'BEGIN{print rand()}'
    0.924046
    Shell > awk 'BEGIN{print rand()}'
    0.924046
    

    The example of using the srand function is as follows:

    Shell > awk 'BEGIN{srand() ; print rand()}'
    0.975495
    Shell > awk 'BEGIN{srand() ; print rand()}'
    0.99187
    Shell > awk 'BEGIN{srand() ; print rand()}'
    0.069002
    

    Generate an integer within the range of (0,100):

    Shell > awk 'BEGIN{srand() ; print int(rand()*100)}'
    56
    Shell > awk 'BEGIN{srand() ; print int(rand()*100)}'
    33
    Shell > awk 'BEGIN{srand() ; print int(rand()*100)}'
    42
    
  4. asort function and asorti function

    Shell > cat /etc/passwd | awk -F ":" '{a[NR]=$1} END{anu=asort(a,b) ; for(i=1;i<=anu;i++) print i,b[i]}'
    1 adm
    2 bin
    3 chrony
    4 daemon
    5 dbus
    6 ftp
    7 games
    8 halt
    9 lp
    10 mail
    11 nobody
    12 operator
    13 polkitd
    14 redis
    15 root
    16 shutdown
    17 sshd
    18 sssd
    19 sync
    20 systemd-coredump
    21 systemd-resolve
    22 tss
    23 unbound
    
    Shell > awk 'BEGIN{a[1]=1000 ; a[2]=200 ; a[3]=30 ; a[4]="admin" ; a[5]="Admin" ; \
    a[6]="12string" ; a[7]=-1 ; a[8]=-10 ; a[9]=-20 ; a[10]=-21 ;nu=asort(a,b) ; for(i=1;i<=nu;i++) print i,b[i]}'
    1 -21
    2 -20
    3 -10
    4 -1
    5 30
    6 200
    7 1000
    8 12string
    9 Admin
    10 admin
    

    Info

    Sorting rules:

    • Numbers have higher priority than strings and are arranged in ascending order.
    • Arrange strings in ascending dictionary order

    If you are using the asorti function, the example is as follows:

    Shell > awk 'BEGIN{ a[-11]=1000 ; a[-2]=200 ; a[-10]=30 ; a[-21]="admin" ; a[41]="Admin" ; \
    a[30]="12string" ; a["root"]="rootstr" ; a["Root"]="r1" ; nu=asorti(a,b) ; for(i in b) print i,b[i] }'
    1 -10
    2 -11
    3 -2
    4 -21
    5 30
    6 41
    7 Root
    8 root
    

    Info

    Sorting rules:

    • Numbers have priority over strings
    • If a negative number is encountered, the first digit from the left will be compared. If it is the same, the second digit will be compared, and so on
    • If a positive number is encountered, it will be arranged in ascending order
    • Arrange strings in ascending dictionary order
  5. sub function and gsub function

    Shell > cat /etc/services | awk '/netbios/ {sub(/tcp/,"test") ; print $0 }'
    netbios-ns      137/test                         # NETBIOS Name Service
    netbios-ns      137/udp
    netbios-dgm     138/test                         # NETBIOS Datagram Service
    netbios-dgm     138/udp
    netbios-ssn     139/test                         # NETBIOS session service
    netbios-ssn     139/udp
    
    Shell > cat /etc/services |  awk '/^ftp/ && /21\/tcp/  {print $0}'
    ftp             21/tcp
                        ↑
    Shell > cat /etc/services |  awk 'BEGIN{OFS="\t"}  /^ftp/ && /21\/tcp/   {gsub(/p/,"P",$2) ; print $0}'
    ftp     21/tcP
                 ↑
    Shell > cat /etc/services |  awk 'BEGIN{OFS="\t"}  /^ftp/ && /21\/tcp/   {gsub(/p/,"P") ; print $0}'
    ftP             21/tcP
                        

    Just like the sed command, you can also use the "&" symbol to reference already matched strings.

    Shell > vim /tmp/tmp-file1.txt
    A 192.168.1.1 HTTP
    B 192.168.1.2 HTTP
    B 192.168.1.2 MYSQL
    C 192.168.1.1 MYSQL
    C 192.168.1.1 MQ
    D 192.168.1.4 NGINX
    
    # Add a line of text before the second line
    Shell > cat /tmp/tmp-file1.txt | awk 'NR==2 {gsub(/.*/,"add a line\n&")} {print $0}'
    A 192.168.1.1 HTTP
    add a line
    B 192.168.1.2 HTTP
    B 192.168.1.2 MYSQL
    C 192.168.1.1 MYSQL
    C 192.168.1.1 MQ
    D 192.168.1.4 NGINX
    
    # Add a string after the IP address in the second line
    Shell > cat /tmp/tmp-file1.txt | awk 'NR==2 {gsub(/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/,"&\tSTRING")} {print $0}'
    A 192.168.1.1 HTTP
    B 192.168.1.2   STRING HTTP
    B 192.168.1.2 MYSQL
    C 192.168.1.1 MYSQL
    C 192.168.1.1 MQ
    D 192.168.1.4 NGINX
    
  6. index function

    Shell > tail -n 5 /etc/services
    axio-disc       35100/udp               # Axiomatic discovery protocol
    pmwebapi        44323/tcp               # Performance Co-Pilot client HTTP API
    cloudcheck-ping 45514/udp               # ASSIA CloudCheck WiFi Management keepalive
    cloudcheck      45514/tcp               # ASSIA CloudCheck WiFi Management System
    spremotetablet  46998/tcp               # Capture handwritten signatures
    
    Shell > tail -n 5 /etc/services | awk '{print index($2,"tcp")}'
    0
    7
    0
    7
    7
    
  7. length function

    # The length of the output field
    Shell > tail -n 5 /etc/services | awk '{print length($1)}'
    9
    8
    15
    10
    14
    
    # The length of the output array
    Shell > cat /etc/passwd | awk -F ":" 'a[NR]=$1 END{print length(a)}'
    22
    
  8. match function

    Shell > echo -e "1592abc144qszd\n144bc\nbn"
    1592abc144qszd
    144bc
    bn
    
    Shell > echo -e "1592abc144qszd\n144bc\nbn" | awk '{print match($1,144)}'
    8
    1
    0
    
  9. split function

    Shell > echo "365%tmp%dir%number" | awk '{split($1,a1,"%") ; for(i in a1) print i,a1[i]}'
    1 365
    2 tmp
    3 dir
    4 number
    
  10. substr function

    Shell > head -n 5 /etc/passwd
    root:x:0:0:root:/root:/bin/bash
    bin:x:1:1:bin:/bin:/sbin/nologin
    daemon:x:2:2:daemon:/sbin:/sbin/nologin
    adm:x:3:4:adm:/var/adm:/sbin/nologin
    lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
    
    # I need this part of the content - "emon:/sbin:/sbin/nologin"
    Shell > head -n 5 /etc/passwd | awk '/daemon/ {print substr($0,16)}'
    emon:/sbin:/sbin/nologin
    
    Shell > tail -n 5 /etc/services
    axio-disc       35100/udp               # Axiomatic discovery protocol
    pmwebapi        44323/tcp               # Performance Co-Pilot client HTTP API
    cloudcheck-ping 45514/udp               # ASSIA CloudCheck WiFi Management keepalive
    cloudcheck      45514/tcp               # ASSIA CloudCheck WiFi Management System
    spremotetablet  46998/tcp               # Capture handwritten signatures
    
    # I need this part of the content - "tablet"
    Shell > tail  -n 5 /etc/services | awk '/^sp/ {print substr($1,9)}'
    tablet
    
  11. tolower function and toupper function

    Shell > echo -e "AbcD123\nqweR" | awk '{print tolower($0)}'
    abcd123
    qwer
    
    Shell > tail -n 5 /etc/services | awk '{print toupper($0)}'
    AXIO-DISC       35100/UDP               # AXIOMATIC DISCOVERY PROTOCOL
    PMWEBAPI        44323/TCP               # PERFORMANCE CO-PILOT CLIENT HTTP API
    CLOUDCHECK-PING 45514/UDP               # ASSIA CLOUDCHECK WIFI MANAGEMENT KEEPALIVE
    CLOUDCHECK      45514/TCP               # ASSIA CLOUDCHECK WIFI MANAGEMENT SYSTEM
    SPREMOTETABLET  46998/TCP               # CAPTURE HANDWRITTEN SIGNATURES
    
  12. Functions that deal with time and date

    What is a UNIX timestamp? According to the development history of GNU/Linux, UNIX V1 was born in 1971, and the book "UNIX Programmer's Manual" was published on November 3 of the same year, which defines 1970-01-01 as the reference date of the start of UNIX.

    The conversion between a timestamp and a natural date time in days:

    Shell > echo "$(( $(date --date="2024/01/06" +%s)/86400 + 1 ))"
    19728
    
    Shell > date -d "1970-01-01 19728days"
    Sat Jan  6 00:00:00 CST 2024
    

    The conversion between a timestamp and a natural date time in seconds:

    Shell > echo "$(date --date="2024/01/06 17:12:00" +%s)"
    1704532320
    
    Shell > echo "$(date --date='@1704532320')"
    Sat Jan  6 17:12:00 CST 2024
    

    The conversion between natural date time and UNIX timestamp in awk program:

    Shell > awk 'BEGIN{print systime()}'
    1704532597
    
    Shell > echo "1704532597" | awk '{print strftime("%Y-%m-%d %H:%M:%S",$0)}'
    2024-01-06 17:16:37
    

I/O statement

Statement Description
getline Read the next matching row record and assign it to "$0".
The return value is 1: Indicates that relevant row records have been read.
The return value is 0: Indicates that the last line has been read
The return value is negative: Indicates encountering an error
getline var Read the next matching row record and assign it to the variable "var"
command | getline [var] Assign the result to "$0" or the variable "var"
next Stop the current input record and perform the following actions
print Print the result
printf See here
system(cmd-line) Execute the command and return the status code. 0 indicates that the command was executed successfully; non-0 indicates that the execution failed
print ... >> file Output redirection
print ... | command Print the output and use it as input to the command
  1. getline

    Shell > seq 1 10 | awk '/3/ || /6/ {getline ; print $0}'
    4
    7
    
    Shell > seq 1 10 | awk '/3/ || /6/ {print $0 ; getline ; print $0}'
    3
    4
    6
    7
    

    Using the functions we learned earlier and the "&" symbol, we can:

    Shell > tail -n 5 /etc/services | awk '/45514\/tcp/ {getline ; gsub(/.*/ , "&\tSTRING1") ; print $0}'
    spremotetablet  46998/tcp               # Capture handwritten signatures        STRING1
    
    Shell > tail -n 5 /etc/services | awk '/45514\/tcp/ {print $0 ; getline; gsub(/.*/,"&\tSTRING2") } {print $0}'
    axio-disc       35100/udp               # Axiomatic discovery protocol
    pmwebapi        44323/tcp               # Performance Co-Pilot client HTTP API
    cloudcheck-ping 45514/udp               # ASSIA CloudCheck WiFi Management keepalive
    cloudcheck      45514/tcp               # ASSIA CloudCheck WiFi Management System
    spremotetablet  46998/tcp               # Capture handwritten signatures        STRING2
    

    Print even and odd lines:

    Shell > tail -n 10 /etc/services | cat -n | awk '{ if( (getline) <= 1) print $0}'
    2  ka-kdp          31016/udp               # Kollective Agent Kollective Delivery
    4  edi_service     34567/udp               # dhanalakshmi.org EDI Service
    6  axio-disc       35100/udp               # Axiomatic discovery protocol
    8  cloudcheck-ping 45514/udp               # ASSIA CloudCheck WiFi Management keepalive
    10  spremotetablet  46998/tcp               # Capture handwritten signatures
    
    Shell > tail -n 10 /etc/services | cat -n | awk '{if(NR==1) print $0} { if(NR%2==0) {if(getline > 0) print $0} }'
    1  aigairserver    21221/tcp               # Services for Air Server
    3  ka-sddp         31016/tcp               # Kollective Agent Secure Distributed Delivery
    5  axio-disc       35100/tcp               # Axiomatic discovery protocol
    7  pmwebapi        44323/tcp               # Performance Co-Pilot client HTTP API
    9  cloudcheck      45514/tcp               # ASSIA CloudCheck WiFi Management System
    
  2. getline var

    Add each line of the b file to the end of each line of the C file:

    Shell > cat /tmp/b.txt
    b1
    b2
    b3
    b4
    b5
    b6
    
    Shell > cat /tmp/c.txt
    A 192.168.1.1 HTTP
    B 192.168.1.2 HTTP
    B 192.168.1.2 MYSQL
    C 192.168.1.1 MYSQL
    C 192.168.1.1 MQ
    D 192.168.1.4 NGINX
    
    Shell > awk '{getline var1 <"/tmp/b.txt" ; print $0 , var1}' /tmp/c.txt
    A 192.168.1.1 HTTP b1
    B 192.168.1.2 HTTP b2
    B 192.168.1.2 MYSQL b3
    C 192.168.1.1 MYSQL b4
    C 192.168.1.1 MQ b5
    D 192.168.1.4 NGINX b6
    

    Replace the specified field of the c file with the content line of the b file:

    Shell > awk '{ getline var2 < "/tmp/b.txt" ; gsub($2 , var2 , $2) ; print $0 }' /tmp/c.txt
    A b1 HTTP
    B b2 HTTP
    B b3 MYSQL
    C b4 MYSQL
    C b5 MQ
    D b6 NGINX
    
  3. command | getline [var]

    Shell > awk 'BEGIN{ "date +%Y%m%d" | getline datenow ; print datenow}'
    20240107
    

    Tip

    Use double quotes to include Shell command.

  4. next

    Earlier, we introduced the break statement and the continue statement, the former used to terminate the loop, and the latter used to jump out of the current loop. See here. For next, when the conditions are met, it will stop the input recording that meets the conditions and continue with subsequent actions.

    Shell > seq 1 5 | awk '{if(NR==3) {next} print $0}'
    1
    2
    4
    5
    
    # equivalent to
    Shell > seq 1 5 | awk '{if($1!=3) print $0}'
    

    Skip eligible line records:

    Shell > cat /etc/passwd | awk -F ":" 'NR>5 {next} {print $0}'
    root:x:0:0:root:/root:/bin/bash
    bin:x:1:1:bin:/bin:/sbin/nologin
    daemon:x:2:2:daemon:/sbin:/sbin/nologin
    adm:x:3:4:adm:/var/adm:/sbin/nologin
    lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
    
    # equivalent to
    Shell > cat /etc/passwd | awk -F ":" 'NR>=1 && NR<=5 {print $0}'
    

    Tip

    "next" cannot be used in "BEGIN{}" and "END{}".

  5. system function

    You can use this function to call commands in the Shell, such as:

    Shell > awk 'BEGIN{ system("echo nginx http") }'
    nginx http
    

    Tip

    Please note to add double quotes when using the system function. If not added, the awk program will consider it a variable of the awk program.

    Shell > awk 'BEGIN{ cmd1="date +%Y" ; system(cmd1)}'
    2024
    

    What if the Shell command itself contains double quotes? Using escape characters - "\", such as:

    Shell > egrep "^root|^nobody" /etc/passwd
    Shell > awk 'BEGIN{ system("egrep \"^root|^nobody\" /etc/passwd") }'
    root:x:0:0:root:/root:/bin/bash
    nobody:x:65534:65534:Kernel Overflow User:/:/sbin/nologin
    

    Another example:

    Shell > awk 'BEGIN{ if ( system("xmind &> /dev/null") == 0 ) print "True"; else print "False" }'
    False
    
  6. Write the output of the awk program to a file

    Shell > head -n 5 /etc/passwd | awk -F ":" 'BEGIN{OFS="\t"} {print $1,$2 > "/tmp/user.txt"}'
    Shell > cat /tmp/user.txt
    root    x
    bin     x
    daemon  x
    adm     x
    lp      x
    

    Tip

    ">" indicates writing to the file as an overlay. If you want to write to the file as an append, please use ">>". Reminder again, you should use double quotation marks to include the file path.

  7. pipe character

  8. Custom functions

    syntax - function NAME(parameter list) { function body }. Such as:

    Shell > awk 'function mysum(a,b) {return a+b} BEGIN{print mysum(1,6)}'
    7
    

Concluding remarks

If you have specialized programming language skills, awk is relatively easy to learn. However, for most sysadmins with weak programming language skills (including the author), awk can be very complicated to learn. For information not covered, please refer to here.

Thank you again for reading.

Author: tianci li