Linux Three Musketeers

Introduction:

1) AWK is a language for processing text files and is a powerful text analysis tool.
The reason why it is called AWK is because it takes the first character of the Family Name of the three founders Alfred Aho, Peter Weinberger, and Brian Kernighan
https://www.runoob.com/linux/linux-comm-awk.html
2) Linux The sed command uses scripts to process text files.
sed can process and edit text files according to the script's instructions.
Sed is mainly used to automatically edit one or more files, simplify repeated operations on files, and write conversion programs.
https://www.runoob.com/linux/linux-comm-sed.html
3) The Linux grep command is used to find qualified strings in the file.
The grep command is used to find files whose content contains the specified template style. If the content of a file is found to conform to the specified template style, the default grep command will display the column containing the template style. If no filename is specified, or - is given, the grep command reads data from the standard input device.
https://www.runoob.com/linux/linux-comm-grep.html

awk data slice

grammar

awk [option parameter] 'script' var=value file(s)
or awk [option parameter] -f scriptfile var=value file(s)
option parameter description:

-F fs or --field-separator fs
specifies the input file fold separator, fs is a string or a regular expression, such as -F:.
-v var=value or --asign var=value
Assign a user-defined variable.
-f scripfile or --file scriptfile
Read awk commands from a script file.
-mf nnn and -mr nnn
set intrinsic limits on nnn values, the -mf option limits the maximum number of blocks allocated to nnn; the -mr option limits the maximum number of records. These two functions are extensions of the Bell Labs version of awk and do not apply in standard awk.
-W compact or --compat, -W traditional or --traditional
Run awk in compatibility mode. So gawk behaves exactly like standard awk, all awk extensions are ignored.
-W copyleft or --copyleft, -W copyright or --copyright
Print short copyright information.
-W help or --help, -W usage or --usage
Print all awk options and a short description of each option.
-W lint or --lint
Print warnings for constructs that are not portable to legacy unix platforms.
-W lint-old or --lint-old
Print warnings about constructs that are not portable to legacy unix platforms.
-W posix
turns on compatibility mode. However, the following restrictions are not recognized: /x, function keywords, func, escape sequences, and when fs is a space, use a newline as a field separator; operators and = cannot replace ^and =; fflush is invalid.
-W re-interval or --re-inerval
allows the use of interval regular expressions, refer to (Posix character classes in grep), such as bracket expressions [[:alpha:]].
-W source program-text or --source program-text
Use program-text as source code, can be mixed with -f command.
-W version or --version
Print version of bug report information.

test1：

[root@VM-0-8-centos ~]# echo "123|456|789" | awk -F '|' '{ print $0 }'
123|456|789
[root@VM-0-8-centos ~]# echo "123|456|789" | awk -F '|' '{ print $1 }'
123
[root@VM-0-8-centos ~]# echo "123|456|789" | awk -F '|' '{ print $3 }'
789

test2: The 5 most recently logged in users

# 每行按空格或TAB分割
[root@VM-0-8-centos ~]# last -n 5 | awk '{print $1}'
root
root
root
root
root

wtmp

Awk can theoretically replace grep and sed

awk ‘pattern{action}’

awk ‘NR>1’

awk 'NR>2’

awk 'NR == 2'

awk ‘NR>2’

NR is from top to bottom, with NR you can skip the lines you don't want to see

[root@VM-0-8-centos ~]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 177052 199820 1742500    0    0     2    11    2    0  1  0 99  0  0
[root@VM-0-8-centos ~]# vmstat | awk 'NR>1'
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 175364 199820 1742820    0    0     2    11    2    0  1  0 99  0  0
[root@VM-0-8-centos ~]# vmstat | awk 'NR>2'
 0  0      0 176152 199820 1742836    0    0     2    11    2    0  1  0 99  0  0
[root@VM-0-8-centos ~]# vmstat | awk 'NR==2'
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

awk ‘@$2~/xxx/’

~add regular expression

tail -10 /tmp/nginx.log
[[email protected] ~]$ awk '$9~/500/' /tmp/nginx.log
139.180.132.174 - - [05/Dec/2018:00:09:12 +0000] "GET /__zep__/js.zip HTTP/1.1" 500 2183 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.018 0.018 .

Regarding awk scripts, we need to pay attention to two keywords BEGIN and END.

BEGIN{ here is the statement before execution }
END {This is the statement to be executed after all lines are processed }
{This is the statement to be executed when processing each line}

[root@VM-0-8-centos ~]# echo -e "1|2|3\n4|5|6\n7|8|9"
1|2|3
4|5|6
7|8|9
[root@VM-0-8-centos ~]# echo -e "1|2|3\n4|5|6\n7|8|9" | awk -F '|' 'BEGIN{a=0}{a=a+$2}END{print a}'
15

[root@VM-0-8-centos ~]# echo -e "1|2|3\n4|5|6\n7|8|9" | awk -F '|' 'BEGIN{a=0}{a=a+$2;print $2}END{;print a;}'
2
5
8
15

test3: Practice taking the first column under /etc/passwd and counting

[root@VM-0-8-centos ~]# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
libstoragemgmt:x:998:997:daemon account for libstoragemgmt:/var/run/lsm:/sbin/nologin
rpc:x:32:32:Rpcbind Daemon:/var/lib/rpcbind:/sbin/nologin
ntp:x:38:38::/etc/ntp:/sbin/nologin
abrt:x:173:173::/etc/abrt:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
chrony:x:997:995::/var/lib/chrony:/sbin/nologin
tcpdump:x:72:72::/:/sbin/nologin
syslog:x:996:994::/home/syslog:/bin/false
mysql:x:27:27:MySQL Server:/var/lib/mysql:/bin/false
[root@VM-0-8-centos ~]# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
libstoragemgmt:x:998:997:daemon account for libstoragemgmt:/var/run/lsm:/sbin/nologin
rpc:x:32:32:Rpcbind Daemon:/var/lib/rpcbind:/sbin/nologin
ntp:x:38:38::/etc/ntp:/sbin/nologin
abrt:x:173:173::/etc/abrt:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
chrony:x:997:995::/var/lib/chrony:/sbin/nologin
tcpdump:x:72:72::/:/sbin/nologin
syslog:x:996:994::/home/syslog:/bin/false
mysql:x:27:27:MySQL Server:/var/lib/mysql:/bin/false
[root@VM-0-8-centos ~]# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
libstoragemgmt:x:998:997:daemon account for libstoragemgmt:/var/run/lsm:/sbin/nologin
rpc:x:32:32:Rpcbind Daemon:/var/lib/rpcbind:/sbin/nologin
ntp:x:38:38::/etc/ntp:/sbin/nologin
abrt:x:173:173::/etc/abrt:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
chrony:x:997:995::/var/lib/chrony:/sbin/nologin
tcpdump:x:72:72::/:/sbin/nologin
syslog:x:996:994::/home/syslog:/bin/false
mysql:x:27:27:MySQL Server:/var/lib/mysql:/bin/false

[root@VM-0-8-centos ~]# cat /etc/passwd | awk -F ':' 'BEGIN{count=0}{name[count]=$1;count++}END{for(i=0;i<NR;i++)print i,name[i]}'
0 root
1 bin
2 daemon
3 adm
4 lp
5 sync
6 shutdown
7 halt
8 mail
9 operator
10 games
11 ftp
12 nobody
13 systemd-network
14 dbus
15 polkitd
16 libstoragemgmt
17 rpc
18 ntp
19 abrt
20 sshd
21 postfix
22 chrony
23 tcpdump
24 syslog
25 mysql

sed data modification

grammar

sed [-hnV][-e] [ -f <script file> ] [ text file ] <br />Parameter Description：

-e <script> or –expression= <script> Process the input text file with the script specified in the option.
-f <script file> or --file=<script file> Process the input text file with the script file specified in the options.
-h or --help Display help.
-n or –quiet or –silent only display the result after script processing.
-V or –version Display version information.

Action description：

a : new, a can be followed by strings, and these strings will appear on a new line (the current next line)~
c : Replace, c can be followed by strings, and these strings can replace the lines between n1 and n2!
d : delete, because it is deleted, so there is usually nothing after d;
i : Insert, i can be followed by strings, and these strings will appear on a new line (the current previous line);
p : Print, that is, print out a selected data. Usually p will be run with the parameter sed -n~
s : Replace, you can directly carry out the replacement work! Usually the action of this s can be paired with regular notation! For example 1,20s/old/new/g and that's it!

test1：

[root@VM-0-8-centos etc]# echo “cat dog fish cat” | sed ‘s/cat/wulala/’

wulala dog fish cat

[root@VM-0-8-centos etc]# echo “cat dog fish cat” | sed ‘s/cat/wulala/g’

wulala dog fish wulala

[root@VM-0-8-centos etc]# echo “123|456|789” | sed ‘s/|/+/g’

123+456+789

test2 ：

[root@VM-0-8-centos ~]# cat test 

aaaaa 

bbbbb 

ccccc 

ddddd 

[root@VM-0-8-centos ~]# sed 's/aaaaa/bbbbbb/' test opens up a virtual space and does not change Original document 

bbbbbb 

bbbbb 

ccccc 

ddddd 

[root@VM-0-8-centos ~]# cat test 

aaaaa 

bbbbb 

ccccc 

ddddd 

[root@VM-0-8-centos ~]# sed -i 's/aaaaa/bbbbbb/' test Change the original document after adding i 

[root@VM-0-8-centos ~]# cat test 

bbbbbb 

bbbbb 

ccccc 

ddddd

grep data to find and locate

grammar

grep [ -abcEFGhHilLnqrsvVwxy ] [ -A<display lines> ] [ -B<display columns> ] [ -C<display columns> ] [ -d <action> ] [ -e<template style> ] [ - f <template file> ] [ --help ] [ template style ] [ file or directory... ] <br />parameter：

-a or --text: Do not ignore binary data.
-A<display lines> or --after-context=<display lines>: In addition to showing the column that conforms to the template style, and show the content after the line.
-b or --byte-offset: Before displaying the line that matches the style, mark the number of the first character of the line.
-B<display lines> or --before-context=<display lines>: In addition to displaying the line that matches the style, and displaying the content before that line.
-c or --count: Count the number of columns matching the style.
-C<display lines> or --context=<display lines> or -<display lines>: In addition to displaying the line that matches the style, it also displays the content before and after the line.
-d <action> or --directories=<action>: This parameter must be used when specifying that a directory is to be searched instead of a file, otherwise the grep command will report the information and stop the action.
-e<template style> or --regexp=<template style>: Specify a string as the style for finding file contents.
-E or --extended-regexp: Use the style as an extended regular expression.
-f <rule file> or --file=<rule file>: Specify a rule file, the content of which contains one or more rule styles, let grep find the file content that meets the rule conditions, and the format is one rule style per line.
-F or --fixed-regexp: Treat styles as a list of fixed strings.
-G or --basic-regexp: Use styles as normal notation.
-h or --no-filename: Do not indicate the name of the file to which the line belongs before displaying the line that matches the style.
-H 或 --with-filename: Indicates the name of the file to which the line belongs before the line that matches the style is displayed.
-i 或 --ignore-case: Ignore differences in character case.
-l 或 --file-with-matches: List the names of files whose contents conform to the specified style.
-L 或 --files-without-match: List file names whose contents do not conform to the specified style.
-n or --line-number: Before displaying the line that matches the style, mark the column number of the line.
-o or --only-matching: Show only the part that matches the PATTERN.
-q or --quiet or --silent: No information is displayed.
-r or --recursive: The effect of this parameter is the same as specifying the "-d recurse" parameter.
-s or --no-messages: No error message is displayed.
-v or --invert-match: Display all lines that do not contain matching text.
-V or --version: Display version information.
-w or --word-regexp: Show only the columns that match the whole word.
-x --line-regexp: Only show the columns that all columns match.
-and: The effect of this parameter is the same as specifying the "-i" parameter.

If there is any infringement, please contact by email, I am sorry.
This is only for learning personal notes, and if there is any reprint, please indicate the source.
Contact email: [email protected] Let’s
learn to test and open the penguin group together (chat, water group, advertising do not disturb): 826471103