CentOS 7 operation and maintenance optimization

December 25, 2017 18:05:27

Readings: 644

CentOS 7 operation and maintenance optimization

Generally, after we install CentOS mini and other corresponding services, it can work normally. However, after working for a period of time, the server will be unstable, invaded, or even directly paralyzed in the event of sudden high concurrency. Most of these problems are due to our consideration of its actual pressure resistance and safety. Therefore, here are some suggestions for operation and maintenance optimization.

1. Turn off unnecessary services

As we all know, the fewer services, the less resources the system will occupy, so you should turn off unnecessary services. It is recommended to turn off unnecessary services. The advantage of doing so is to reduce memory and CPU resource usage. First of all, you can see which services have been started in the system.

// 安装ntsysv
yum install -y ntsysv
// 设置启动的服务
ntsysv

Listed below are the services that need to be started, and services that are not listed will be shut down.

crond : Automatically schedule tasks.

network: The network service of the Linux system is very important. If this service is not enabled, the server cannot be connected to the Internet.

sshd: OpenSSH server daemon.

rsyslog: Linux log system service (the service name is syslog under CentOS5.8), it must be started.

2. Close unwanted TTYs

Open the file with vim editor

vim /etc/init/start-ttys.conf
// 内容如下：
start on stopped rc RUNLEVEL=[2345]
env ACTIVE_CONSOLES=/dev/tty[1-6]
env X_TTY=/dev/tty1
task
script
    . /etc/sysconfig/init
    for tty in $(echo $ACTIVE_CONSOLES) ; do
        [ "$RUNLEVEL" = "5" -a "$tty" = "$X_TTY" ] && continue
        initctl start tty TTY=$tty
    done
end script

This code makes init open 6 consoles, which can be divided into ALT + F1 to ALT + F6 consoles are all resident in memory by default. You can see it with the ps aux command, the command is as follows:

ps aux | grep tty | grpe -v grep

The command display results are as follows:

root         1211  0.0  0.2 115520  2048 tty1
root         1213  0.0  0.2 115520  2048 tty2
root         1214  0.0  0.2 115520  2048 tty3
root         1217  0.0  0.2 115520  2048 tty4
root         1219  0.0  0.2 115520  2048 tty5

In fact, there is no need to use so much, so how to close the process that does not need babies?
Usually it's enough to keep two consoles.

vim /etc/init/start-ttys.conf

3. Adjust TCP/IP network parameters

Adjusting the TCPⅡP network parameters can strengthen the ability to fight against SYN Flood. The commands are as follows:

echo 'net.ipv4.tcp_syncookies = 1' >> /etc/sysctl.conf
sysctl -p

4. Modify the number of history records of the SHELL command

// 用Vim编辑器打开
vim /etc/profile
// 找到HISTSIZE=1000 并改为 100；
HISTSIZE=100
// 立即生效
source /etc/profile

5. Regularly correct the time of the server

yum install -y ntp
crontab -e
// 加入一行
*/5 * * * * /usr/sbin/ntpdate ntp.api.bz

ntp.api.bz is a group of NTP server clusters. It used to be 6 servers, located in Shanghai Telecom; now it is 3 servers, scattered in Shanghai and Zhejiang Telecom, which can be viewed with the dig command

dig ntp.api.bz

6. Stop the IPV6 network service

By default on CentOS64, IPv6 is enabled.

// 可用如下命令查看：
lsmod | grep ipv6

Some networks and applications do not yet support IPv6, so disabling IPv6 can be a very good choice: to strengthen the security of the system and improve the overall performance of the system. However, first of all, we need to confirm whether IPv6 is active or not. The command is as follows:

// 列出全部网络接口信息
ifconfig -a

// 修改相应的配置文件，停止 IPv6 ，命令如下：
echo "install ipv6 /bin/true" > /etc/modprobe.d/disable-ipv6.conf
# 每当系统需要加载IPv6时，强制执行 /bin/true 来替代实际加载的模块
echo "IPV6INIT=no" >> /etc/sysconfig/network-scripts/ifcfg-eth0
# 禁用基于IPv6网络，使之不会被触发启动

7. Adjust the maximum number of open files for Linux

It is necessary to adjust the maximum number of open files in Linux, otherwise the performance of the machine running the Squid curse service will be poor under high load; in addition, when deploying applications under Linux, sometimes encounter "Too many open files" such as Problem, this value also affects the server's maximum concurrency. In fact, Linux has a file handle limit. However, the default value is very high, usually 1024. The production server can easily reach this value, so this value needs to be changed.

// 打开配置  
vim /etc/security/limit.conf
// 在最后一行添加如下
* soft nofile 65535
* hard nofile 65535
// 再打开配置
vim /etc/rc.local
// 添加如下内容
ulimit -SHn 65535

Also, the ulimit -n command doesn't really see the file's maximum number of open files. You can view it with the following script:

#!/bin/bash
for pid in `ps aux |grep nginx |grep -v grep|awk '{print $2}'`
do
cat /proc/${pid}/limits |grep 'Max open files'
done

8. Start the network card

When configuring the IP address of the CentOS 7 network card, one of the easily overlooked items is that Linux does not start the network card at boot time, and the consequence is obvious, that the Linux machine will never have an IP address.

// 查看以太网代号(也可用ifconfig命令)
ip address
// 修改网卡配置文件
vim /etc/sysconfig/network-scripts/ifcfg-enp1s0
// 修改如下内容（如果没有，请自行添加）
# 系统启动时就启动网卡设备
ONBOOT=yes
# 允许用从DHCP处获取的DNS覆盖本地的DNS
PEERDNS=yes
# 不允许普通用户修改网卡
USERCTL=no

9. Disable write disk I/O

Linux files have 3 times by default, as shown below.

atime: The access time to this file.
ctime: The time when the inOde of this file changed.
mtime: The modification time of this file.

If there are multiple small files (for example, there are multiple small pictures on the page of the Web server), it is usually unnecessary to record the access time of the file, so that the I/O of writing to the disk can be reduced, but how to configure this?

// 修改文件系统的配置文件
vim /etc/fstab
// 然后，在包含大量小文件的分区中使用 noatime 和 nodiratime 这两个命令。例如：  
/dev/sda5 /data/pics ext3 noatime,nodiratime 0 0 
这样文件被访问时就不会再产生写磁盘的 I/O 了。

10. Modify SSH login configuration

SSH service configuration optimization, please keep at least one user with sudo privileges in the machine, the following configuration prohibits root remote login, the code content is as follows:

# 禁止root远程登录
sed -i 's@#PermitRootLogin yes@PermitRootLogin no@' /etc/ssh/sshd_config
# 禁止空密码登录
sed -i 's@PermitEmptyPasswords no@PermitEmptyPasswords no@' /etc/ssh/sshd_config
# 关闭SSH反向查询，以加快SSH的访问速度
sed -i 's@UseDNS yes@UseDNS no@' /etc/ssh/sshd_config /etc/ssh/sshd_config

11. Increase users with SUdO privileges

The steps and process of adding users are relatively simple and omitted here. Since the system has prohibited root remote login, an admin user with sudo privileges is required, and the privileges are equivalent to those of root.

vim /etc/sudoers
## Allow root to run any commands anywhere
root    ALL=(ALL)   ALL
# 然后添加如下内容：
admin   ALL=(ALL)   ALL
# 如果在进行sudo切换时不想输入密码，可以做如下更改：
admin   ALL=(ALL)   NOPASSWD:ALL

12. Optimize the kernel TCP parameters under Linux to improve system performance

The optimization of the kernel, like the optimization of the server, should be based on the principle of stability and security. The following takes the Squid server as an example to illustrate. After the client and the server establish a TCP/IP connection, the Socket will be closed, and the port status of the server connection will become TIME_WAIT. Does that mean that all SOCkets that perform active shutdown will enter the TIME_WAIT state? Is there any situation that can make the Socket that is actively closed enter the CLOSED state directly? The answer is that the party that is actively closing will enter the TIME_WAIT state after sending the last ACK, and stay for 2MSL (maximum packet survival) time, which is essential for TCP/IP, which means that this cannot be "solved". of.

There are two main reasons why the TCP/IP designers do this:

Prevent the packets in the previous connection from reappearing after getting lost, affecting the new connection after 2MSL time, all the duplicate packets in the previous connection will disappear.
To reliably close TCP connections. The last ACKFN sent by the active closing party may be lost. If lost, the passive party will resend Fm. At this time, if the active party is in the CLOSED state, it will respond to RST instead of ACK. Therefore, the active party must be in the TIM and IT state, not in the CLOSED state. In addition, TIME_WAIT does not take up a lot of resources unless attacked.

// 在Squid服务器中可输入如下命令查看当前连接统计数： 
netstat -n | awk '/^tcp/ {++S[$NF]} END{for(a in S)} print a, S[a]}'

The command display results are as follows:

LAST_ACK 14
SYN_RECV 348
ESTABISHED 70
FIN_WAIT1 229
FIN_WAIT2 30
CLOSING 33
TIME_WAIT 18122

The meanings in the commands are as follows.

CLOSED: No active or ongoing connection.
LISTEN: The server is waiting for an incoming call.
SYN_RECV: A connection request has arrived, waiting for confirmation.
SYN_SENT: The application has started, opening a connection.
ESTABLISHED; normal data transfer status.
FIN_WAT1: The application said it was done.
FIN_WAT2: The other side has agreed to release.
ITMED_WAIT: Wait for all packets to die.
CLOSING; both try to close at the same time.
TIME_WAIT: The other side has initiated a release.
LAST_ACK: Wait for all packets to die.
That is to say, this command can classify and summarize the network connection status of the current system.
In a high-concurrency Squid server under Linux, the number of TCP TIME_WAIT sockets can often reach 20,000 or 30,000, and the server can easily be dragged to death. However, the number of TIME_WAIT sockets for the Squid server can be reduced by modifying the Linux kernel parameters, as follows:

vim /etc/sysctl.conf
// 然后，增加以下参数
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 5000

The following will briefly explain the meaning of each of the above parameters:

net.ipv4.tcp_syncookies = 1 means enable SYN Cookies. When the SYN waiting queue overflows, enable cookie rotation to handle, which can prevent a small number of SYN attacks. This parameter defaults to 0, which means off.
net.ipv4.tcp_tw_reuse = 1 means to enable reuse, that is, to allow TCP connections to be reused for TIME-WAIT sockets. This parameter defaults to 0, which means off.
net.ipv4.tcp_tw_recycle = 1 means to enable fast recycling of TIME-WAIT sockets in TCP connections. The default value of this parameter is 0, which means it is closed.
net.ipv4.tcp_fin_timeout = 30 means that if the socket is requested to be closed by the local end, then this parameter will determine the time to remain in the FlN-WAIT-2 state.
net.ipv4.tcp_keepalive_time = 1200 means that when Keepalived is enabled, the frequency of TCP sending Keepalived messages is changed to 20 minutes, and the default value is 2 hours.
net.ipv4.ip_local_port_range = 10000 65000 indicates the port range for outgoing connections of the CentOS system. Its default value is very small, here it is changed from 10000 to 65000. It is recommended not to set the minimum value here too low, otherwise normal ports may be occupied.
net.ipv4.tcp_max_syn_backlog = 8192 indicates the length of the SYN queue. The default value is 1024. Here, the queue length is increased to 8192, which can accommodate more network connections waiting to be connected.
net.ipv4.tcp_max_tw_buckets = 5000 means that the system keeps the maximum number of TIME_WAIT sockets at the same time. If this number is exceeded, the TlME_WAIT socket will be cleared immediately and a warning message will be printed. The default value is 180000, which is changed to 5000 here. For Apache, Nginx and other servers, the parameters introduced above can reduce the number of TIME_WAIT sockets very well, but for Squid, the effect is not great. With this parameter, the maximum number of TME_WAIT sockets can be controlled. Amount to avoid Squid logging the server to be dragged down by a large number of TIME_WAIT sockets.

Execute the following command to make the kernel configuration take effect immediately:

/sbin/sysctl -p

If it's for a web server like Apache or Nginx, you only need to change the following items.

net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_local_port_range = 10000 65000

// 执行以下命令使内核配置立马生效
/sbin/sysctl -p

If it is a Post6x mail server, the recommended kernel optimization scheme is as follows:

net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_local_port_range = 10000 65000
kernel.shmmax = 134217728

// 执行以下命令使内核配置立马生效
/sbin/sysctl -p

Of course, these are only the most basic changes. You can also change the kernel settings according to your own needs. For example, in the case of high concurrency of our online machines, the error of ''TCP: too many orpharned sockets' will often appear as much as possible. Also follow the highest principle of server stability. If the server is unstable, all the work and efforts will be in vain.
If the above optimizations still cannot meet the working requirements, you may need to customize your server core or upgrade the server hardware.