CentOS 7 System Optimization and Deployment Guide

Wed, 23 Jun 2021 00:00:00 +0000

CentOS7 Kernel Upgrade

Download kernel source:

`1`	`rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm`

Install the latest kernel version:

`1`	`yum --enablerepo=elrepo-kernel install -y kernel-lt`

Check entries:

`1`	`cat /boot/grub2/grub.cfg \| grep menuentry`

Set default boot kernel:

`1`	`grub2-set-default "CentOS Linux (4.4.221-1.el7.elrepo.x86_64) 7 (Core)"`

Disable Firewall

1
2

systemctl stop firewalld  
systemctl disable firewalld

Install Common Tools

`1`	`yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget vim git net-tools dos2unix lsof tcpdump lrzsz telnet bash-completion.noarch conntrack-tools`

Linux completion:

`1`	`yum install libvirt-bash-completion bash-completion gedit-plugin-bracketcompletion gedit-plugin-wordcompletion libguestfs-bash-completion -y`

Configure SELinux

1
2

setenforce 0  
sed -i '/^SELINUX=/ s/enforcing/disabled/' /etc/selinux/config

Update History and Shell Timeout Settings

Edit /etc/profile:

1
2

export HISTSIZE=100  
export TMOUT=300

Disable swap partition

swapoff -a
# To permanently disable swap partition, comment out the swap line in the following file
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
echo "vm.swappiness = 0">> /etc/sysctl.conf
sysctl -p

Disable mail service

1
2

systemctl stop postfix.service
systemctl disable postfix.service

Log optimization

mkdir /var/log/journal
mkdir /etc/systemd/journald.conf.d
cat > /etc/systemd/journald.conf.d/99-prophet.conf << EOF
[journal]
# Persistent storage to disk
Storage=persistent
# Compress historical logs
Compress=yes
 
SyncIntervalSec=5m
RateLimitInterval=30s
RateLimitBurst=1000
# Maximum disk space 10G
SystemMaxUse=10G
# Single log file maximum size 200M
SystemMaxFileSize=200M
# Log retention time 2 weeks
MaxRetentionSec=2week
# Do not forward logs to syslog
ForwardToSyslog=no
 
EOF
systemctl restart systemd-journald

Load ipvs modules

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
 
chmod 755 /etc/sysconfig/modules/ipvs.modules
bash /etc/sysconfig/modules/ipvs.modules

File Optimization

echo ‘* - nofile 65535 ’ »/etc/security/limits.conf echo ‘vm.max_map_count=262144 ’ »/etc/security/limits.conf

sysctl vm.overcommit_memory=1

tail -1 /etc/security/limits.conf sysctl -p

Kernel Optimization

cat >>/etc/sysctl.conf<<EOF
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30

net.ipv4.tcp_keepalive_time = 1200
net.ipv4.ip_local_port_range = 4000 65000
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_max_tw_buckets = 36000

net.ipv4.route.gc_timeout = 100
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_synack_retries = 1
net.core.somaxconn = 262144
net.core.netdev_max_backlog = 262144
net.ipv4.tcp_max_orphans = 16384

net.ipv4.tcp_mem = 94500000 915000000 927000000
EOF

sysctl -p

Explanation of Network Parameters

net.ipv4.tcp_syncookies = 1: Enables SYN Cookies. When the SYN backlog overflows, cookies are used to handle connections, mitigating minor SYN flooding attacks. Default: 0 (disabled).
net.ipv4.tcp_tw_reuse = 1: Allows reusing TIME-WAIT sockets for new TCP connections. Default: 0 (disabled).
net.ipv4.tcp_tw_recycle = 1: Enables fast recycling of TIME-WAIT sockets. Default: 0 (disabled).
net.ipv4.tcp_fin_timeout = 30: Defines the time (in seconds) a connection remains in FIN-WAIT-2 state if closed locally.
net.ipv4.tcp_keepalive_time = 1200: Sets the frequency (in seconds) for TCP keepalive probes. Default: 7200 (2 hours), modified to 1200 (20 minutes).
net.ipv4.ip_local_port_range = 1024 65000: Specifies the port range for outgoing connections. Default: 32768-61000, expanded to 1024-65000.
net.ipv4.tcp_max_syn_backlog = 8192: Sets the maximum length of the SYN queue to accommodate more pending connections. Default: 1024.
net.ipv4.tcp_max_tw_buckets = 5000: Limits the maximum number of TIME-WAIT sockets. Exceeding this threshold triggers immediate cleanup. Default: 180000, adjusted for servers like Apache/Nginx to reduce TIME-WAIT sockets. Squid may require additional tuning.

SSH Service Optimization

# Edit SSH configuration
vim /etc/ssh/sshd_config

# Disable GSSAPI authentication
GSSAPIAuthentication no

# Disable DNS resolution checks
UseDNS no  # (Remove the '#' to uncomment; default is disabled)

# Restart SSH service
systemctl restart sshd

1. CPU Core Count, Model, and Clock Speed

`1`	`cat /proc/cpuinfo \| grep name \| cut -f2 -d: \| uniq -c`

2. Testing Disk I/O Performance

1). hdparm Command

The hdparm command provides a CLI interface for reading and setting parameters of IDE/SCSI hard drives. Note: This command only tests disk read speed.

[root@server-68.2.stage.polex.io var ]$ hdparm -Tt /dev/polex_pv/varvol

/dev/polex_pv/varvol:
 Timing cached reads:    MB in  2.00 seconds = 7803.05 MB/sec
 Timing buffered disk reads:  MB in  3.01 seconds = 374.90 MB/sec

[Additional translation of subsequent sections would continue here following the same pattern]

2). The dd Command

The Linux dd command is used to read, convert, and output data. dd can read data from standard input or files, transform it according to specified formats, and then output it to files, devices, or standard output.

We can use the copy function of the dd command to test the IO performance of a disk. Note that dd provides only a rough measurement of disk IO performance and is not highly accurate.

[root@server-68.2.stage.polex.io var ]$ time dd if=/dev/zero of=test.file bs=1G count= oflag=direct
+ records in
+ records out
 bytes (2.1 GB) copied, 13.5487 s,  MB/s
 
real    0m13.556s
user    0m0.000s
sys    0m0.888s

??? note “Parameter Explanation” As shown, the disk write speed for this partition is 159 MB/s. Key parameters include:

- `/dev/zero`: A pseudo-device that generates empty character streams; no IO is incurred.
- `if`: Specifies the input file for `dd` to read from.
- `of`: Specifies the output file for `dd` to write to.
- `bs`: Defines the block size for each write operation.
- `count`: Sets the number of blocks to write.
- `oflag=direct`: Required for IO testing, ensures direct writes to disk (bypassing cache).

3). FIO Testing Disk IO Performance

The fio command is specifically used to test IOPS and is more accurate than the dd command. The fio command has many parameters. Here are some examples for reference:

`1`	`yum install fio`

# Random read:  
fio -filename=/dev/sda1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=4k -size=60G -numjobs=64 -runtime=10 -group_reporting -name=file  
# Sequential read:  
fio -filename=/dev/sda1 -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=4k -size=60G -numjobs=64 -runtime=10 -group_reporting -name=file  
# Random write:  
fio -filename=/dev/sda1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=4k -size=60G -numjobs=64 -runtime=10 -group_reporting -name=file  
# Sequential write:  
fio -filename=/dev/sda1 -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=4k -size=60G -numjobs=64 -runtime=10 -group_reporting -name=file  
# Mixed random read/write:  
fio -filename=/dev/sda1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=30 -ioengine=psync -bs=4k -size=60G -numjobs=64 -runtime=10 -group_reporting -name=file -ioscheduler=noop

In the results, bw=1532.2KB/s, iops=383 indicates the measured IOPS.

??? note “Parameter Explanation”

filename=/dev/sda1: Test file name, typically selecting the data directory of the disk to be tested
direct=1: Bypasses system buffers during testing for more authentic results
rw=randwrite: Tests random write I/O
rw=randrw: Tests mixed random read/write I/O
rw=randread: Tests random read I/O
bs=4k: Block size per I/O operation is 4KB
bsrange=512-2048: Specifies data block size range
size=60g: Test file size set to 60GB with 4KB I/O operations
numjobs=64: Test runs with 64 concurrent threads
runtime=10: Test duration limited to 10 seconds
ioengine=psync: I/O engine uses psync mode
rwmixwrite=30: 30% write ratio in mixed read/write mode
group_reporting: Aggregates results per-process
Additional parameters:
- lockmem=1g: Limits memory usage to 1GB for testing
- zero_buffers: Initialize buffers with zeros
- nrfiles=8: Number of files generated per process

4). iostat Command

First use iostat to check if disk I/O has high read/write loads
If %util approaches 100%, it indicates too many I/O requests and the I/O system is saturated. The disk may be a bottleneck. Generally, if %util exceeds 70%, the I/O pressure is significant with considerable read wait time. Then check other parameters.

1
2
3

yum install sysstat

iostat -x 1 10

??? note “Explanation”

    rrqm/s: Number of read operations merged per second. Calculated as delta(rmerge)/s
    wrqm/s: Number of write operations merged per second. Calculated as delta(wmerge)/s
    r/s: Read I/O operations completed per second. Calculated as delta(rio)/s
    w/s: Write I/O operations completed per second. Calculated as delta(wio)/s
    rsec/s: Sectors read per second. Calculated as delta(rsect)/s
    wsec/s: Sectors written per second. Calculated as delta(wsect)/s
    rKB/s: Kilobytes read per second. Half of rsec/s (since sector size is 512 bytes)
    
    wKB/s: Kilobytes written per second. Half of wsec/s
    avgrq-sz: Average data size per I/O operation (sectors). Calculated as delta(rsect+wsect)/delta(rio+wio)
    avgqu-sz: Average I/O queue length. Calculated as delta(aveq)/s/1000 (since aveq is in milliseconds)
    await: Average wait time per I/O operation (milliseconds). Calculated as delta(ruse+wuse)/delta(rio+wio)
    svctm: Average service time per I/O operation (milliseconds). Calculated as delta(use)/delta(rio+wio)
    %util: Percentage of time with I/O operations active, or time when I/O queue was non-empty

5). iotop Command

A tool to identify processes with high I/O usage. Simply execute the iotop command:

`1`	`yum install iotop -y`

3. sar Command

The sar -u 1 1 command checks CPU utilization, sampling the data once every 1 second for 1 iteration.
The sar command is an essential tool for analyzing system bottlenecks, used to monitor performance metrics including CPU, memory, disk, and network.

[root@server-68.2.stage.polex.io var ]$ sar -d -p Linux 3.10.0-693.5.2.el7.x86_64 (server-) // x86_64 ( CPU)

:: PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util :: PM sda 1.00 0.00 3.00 3.00 0.01 9.00 9.00 0.90 :: PM sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 :: PM polex_pv-rootvol 1.00 0.00 3.00 3.00 0.01 9.00 9.00 0.90 :: PM polex_pv-varvol 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 :: PM polex_pv-homevol 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

:: PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util :: PM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 :: PM sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 :: PM polex_pv-rootvol 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 :: PM polex_pv-varvol 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 :: PM polex_pv-homevol 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util Average: sda 0.50 0.00 1.50 3.00 0.00 9.00 9.00 0.45 Average: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: polex_pv-rootvol 0.50 0.00 1.50 3.00 0.00 9.00 9.00 0.45 Average: polex_pv-varvol 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: polex_pv-homevol 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

In the command, the “-d” parameter represents viewing disk performance, the “-p” parameter displays dev devices by names like sda, sdb, etc., “1” indicates sampling values every 1 second, and “2” specifies collecting data a total of 2 times.

??? note “Parameter Explanation”
await: The average waiting time per device I/O operation (in milliseconds).

**svctm**: The average service time per device I/O operation (in milliseconds).  

**%util**: The percentage of time spent on I/O operations each second.  

For disk I/O performance, the following criteria generally apply:  

- Normally, **svctm** should be smaller than **await**. The value of **svctm** depends on disk performance, but CPU and memory load can also affect it. Excessive I/O requests may indirectly increase the **svctm** value.  

- The **await** value is typically influenced by **svctm**, the I/O queue length, and the I/O request pattern. If **svctm** is close to **await**, it indicates minimal I/O waiting, implying excellent disk performance. If **await** is significantly higher than **svctm**, it suggests a long I/O queue wait, which slows down applications. This can often be resolved by using a faster disk.  

- **%util** is another critical metric. If **%util** approaches 100%, the disk is handling too many I/O requests and operating at full capacity, indicating a potential bottleneck. Prolonged high utilization will degrade system performance. Solutions include optimizing programs or upgrading to a faster/higher-capacity disk.

4. vmstat Command

1
2
3

[root@server-68.2.stage.polex.io var ]$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

In the output, the bi and bo values reflect current disk performance:

bi: Blocks received per second from block devices. Block devices include all disks and other block devices on the system. The default block size is 1024 bytes.
bo: Blocks sent per second to block devices. For example, reading files increases bo.
Generally, both bi and bo should be close to 0. Consistently high values indicate excessive I/O activity, requiring system adjustments.

5. uptime Command

uptime

The output displays:

Current system time
System uptime (duration since last reboot)
Number of logged-in users
Load averages for the last 1 minute, 5 minutes, and 15 minutes.

If the load average values consistently exceed the number of CPUs in the system, it indicates high CPU load, which may degrade performance.

1) netstat Command

`1`	`netstat -an \|grep tcp # View all active TCP connection details`

2). Socket Statistics Command

Previously using the netstat command was found to be inefficient on busy servers, sometimes consuming over 90% of CPU.

The Socket Statistics (ss) command, however, operates at a lower level using the tcp_diag module in the TCP protocol stack for statistical analysis, making it faster and more efficient.

Common ss Commands:

ss -t: Displays all current TCP connections.

??? note “Details”
- -t: Show TCP connection information only
- -a: Display all connection information
- -u: Show UDP connection information only

While nearly all Linux systems include `netstat` by default, `ss` may not be pre-installed (CentOS includes it by default).  

The `ss` command is part of the `iproute` toolkit, a suite of tools for managing TCP/UDP/IP networks with IPv4/IPv6 support.  

If the `ss` command is missing, install the toolkit with:

`1`	`yum install iproute iproute-doc`

7. Disk I/O, Throughput, and Storage IOPS

Disk I/O, Throughput, and Storage IOPS Performance Metrics

Cloud server disk storage performance metrics include Disk I/O, IOPS, and Throughput. Below is a detailed explanation of these terms and their relationships:

Storage IOPS (Input/Output Operations Per Second): The number of read/write operations a disk can perform per second.
Disk I/O: Refers to input (writing data to disk) and output (reading data from disk). The data volume per I/O request is measured in KiB (e.g., 4KiB, 256KiB, 1024KiB).
Throughput: The total data transfer rate per second, combining read and write operations.

Formula: Relationship Between IOPS, I/O Size, and Throughput

The relationship is defined as:
Throughput = IOPS × I/O Size

In other words, higher IOPS and larger I/O sizes result in greater throughput. While higher IOPS and throughput values are generally desirable, they are constrained by hardware limits.

For further details on disk I/O performance for cloud servers, refer to Alibaba Cloud’s documentation on ECS storage performance at ecs6.com.

Common Linux Monitoring Commands

free
df
top / htop
uptime
iftop
iostat
iotop
vmstat
netstat
nethogs (shows bandwidth used by each process)

System Tuning on Kang's Blog