nginx.service start-pre operation timed out. Terminating.

问题
服务器重启后Nginx服务不会自动启动,手动启动Nginx服务没有出现任何问题。

错误状态
#systemctl status nginx
nginx.service - nginx - high performance web server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
Active: failed (Result: timeout) since Fri 2020-02-28 19:13:13 CST; 6min ago
Docs: http://nginx.org/en/docs/
Process: 806 ExecStartPre=/usr/local/nginx/sbin/nginx -t -c /usr/local/nginx/conf/nginx.conf (code=killed, signal=TERM)

Feb 28 19:11:43 izj6cdtn3gxvjysis55g0hz systemd[1]: Starting nginx - high performance web server...
Feb 28 19:13:13 izj6cdtn3gxvjysis55g0hz systemd[1]: nginx.service start-pre operation timed out. Terminating.
Feb 28 19:13:13 izj6cdtn3gxvjysis55g0hz systemd[1]: Failed to start nginx - high performance web server.
Feb 28 19:13:13 izj6cdtn3gxvjysis55g0hz systemd[1]: Unit nginx.service entered failed state.
Feb 28 19:13:13 izj6cdtn3gxvjysis55g0hz systemd[1]: nginx.service failed.

#journalctl -u nginx.service
-- Logs begin at Sat 2020-02-29 03:11:38 CST, end at Fri 2020-02-28 19:20:01 CST. --
Feb 28 19:11:43 izj6cdtn3gxvjysis55g0hz systemd[1]: Starting nginx - high performance web server...
Feb 28 19:13:13 izj6cdtn3gxvjysis55g0hz systemd[1]: nginx.service start-pre operation timed out. Terminating.
Feb 28 19:13:13 izj6cdtn3gxvjysis55g0hz systemd[1]: Failed to start nginx - high performance web server.
Feb 28 19:13:13 izj6cdtn3gxvjysis55g0hz systemd[1]: Unit nginx.service entered failed state.
Feb 28 19:13:13 izj6cdtn3gxvjysis55g0hz systemd[1]: nginx.service failed.

原因
没有足够的超时值来启动Nginx服务

解决办法
vim /usr/lib/systemd/system/nginx.service
增加TimeoutStartSec参数值:
[Service]
...
TimeoutStartSec=600
-------------------------------------------
例子
cat /usr/lib/systemd/system/nginx.service
[Unit]
Description=nginx - high performance web server
Documentation=http://nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStartPost=/bin/sleep 0.1
ExecStartPre=/usr/local/nginx/sbin/nginx -t -c /usr/local/nginx/conf/nginx.conf
ExecStart=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
ExecReload=/usr/local/nginx/sbin/nginx -s reload
ExecStop=/usr/local/nginx/sbin/nginx -s stop
PrivateTmp=true
LimitNOFILE=51200
LimitNPROC=51200
LimitCORE=51200
TimeoutStartSec=600

[Install]
WantedBy=multi-user.target
--------------------------------------------

这个情况也可能出现在其他服务上面,没有足够的超时值来启动Nginx或Apache或者其他服务

CentOS7
使用任何文本编辑器(例如vi)打开初始化init脚本:
对于Apache: /usr/lib/systemd/system/httpd.service
对于nginx: /usr/lib/systemd/system/nginx.service

增加TimeoutStartSec参数值
[Service]
...
TimeoutStartSec=600

Debian9或者Ubuntu18
可以使用以下命令找到确切初始化脚本
systemctl status apache2.service | grep loaded
systemctl status nginx.service | grep loaded

使用任何文本编辑器(例如vi)打开初始化init脚本:
对于Apache:/lib/systemd/system/apache2.service
对于nginx:/lib/systemd/system/nginx.service

增加TimeoutStartSec参数值
[Service]
...
TimeoutStartSec=600
https://support.plesk.com/hc/en-us/articles/213387149-Nginx-Apache-are-not-running-after-server-reboot-on-a-Plesk-server-service-start-operation-timed-out-Terminating

Baidu Sitemap Generator 报错,WordPress发现了您的插件造成的错误:Baidu Sitemap Generator

升级PHP7.3之后,发现WordPress的Baidu Sitemap Generator在生成百度 XMLSitemap 生成器插件出现如下错误
错误详情
============
错误类别E_ERROR发生在文件www.zhangfangzhou.cn/wp-content/plugins/baidu-sitemap-generator/sitemap-function.php的439行。错误信息:Uncaught Error: Call to undefined function split() in www.zhangfangzhou.cn/wp-content/plugins/baidu-sitemap-generator/sitemap-function.php:439
Stack trace:
#0 www.zhangfangzhou.cn/wp-content/plugins/baidu-sitemap-generator/baidu_sitemap.php(278): xml_annotate()
#1 www.zhangfangzhou.cn/wp-content/plugins/baidu-sitemap-generator/baidu_sitemap.php(264): build_baidu_sitemap_xml('https...', 1)
#2 www.zhangfangzhou.cn/wp-content/plugins/baidu-sitemap-generator/baidu_sitemap.php(110): build_baidu_sitemap(1)
......
......
因为这款插件一直没有更新,PHP7版本废弃了一些PHP函数,split这个函数在PHP7版本已经不支持了,因此必须替换成PHP7版本的preg_split函数。

修复Baidu Sitemap Generator 报错需要修改2个地方

1、修改Baidu Sitemap Generator插件目录下的sitemap-function.php文件的439行和baidu_sitemap.php文件的308行
修改成

list( $today_year, $today_month, $today_day, $hour, $minute, $second ) = preg_split( '([^0-9])', $blogtime );

就是用preg_split替换掉split

2、修改Baidu Sitemap Generator插件目录下的baidu_sitemap.php


function baidu_sitemap_menu() {
/** Add a page to the options section of the website **/
if (current_user_can('manage_options'))
add_options_page("Baidu-Sitemap","Baidu-Sitemap", 8, __FILE__, 'baidu_sitemap_optionpage')
}
修改成
function baidu_sitemap_menu() {
/** Add a page to the options section of the website **/
if (current_user_can('manage_options'))
add_options_page("Baidu-Sitemap","Baidu-Sitemap", 'manage_options', __FILE__, 'baidu_sitemap_optionpage');
}

就是把8换成'manage_options'
修改完成后重启PHP,即可更新百度 XMLSitemap

百度云磁盘CDS Linux数据盘扩展分区

百度云磁盘CDS Linux数据盘扩展分区
随着Web业务的持续增加,原先400G的Web服务器数据磁盘已经不能满足使用,为满足Web业务的需要,决定扩容数据磁盘到1000G。
在百度云磁盘CDS扩容只是将磁盘的存储容量扩大,不会扩展分区和文件系统。

实战记录百度云磁盘CDS Linux数据盘扩展分区,Web 服务器为CentOS release 6.10 (Final)系统,数据磁盘的格式为EXT4。

1、查看Web服务器数据磁盘挂载信息
#mount | grep "/dev/vdb"
/dev/vdb1 on /data type ext4 (rw)

2、查看是否有程序在Web服务器数据磁盘正在运行
fuser -m /dev/vdb1
/dev/vdb1: 5115

如果有的话,查找并结束掉进程
ps aux | grep 5115
#ps aux | grep 5115
apache 5115 0.2 0.1 449060 29048 ? S 10:56 0:03 /usr/sbin/httpd

3、关闭正在使用Web服务器数据磁盘的服务或者进程
service httpd stop 或者 kill -9 5115

4、编辑fstab,取消Web服务器数据磁盘自动挂载
vi /etc/fstab
#/dev/vdb1 /data ext4 defaults 0 0

5、卸载Web服务器数据磁盘
#umount /dev/vdb1

#umount /dev/vdb1 如果有程序正在运行,则会这样提示
umount: /data: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
取消正在运行的程序 比如webserver mysql

6、调整分区大小
(1)查看分区表
fdisk /dev/vdb

p print the partition table
p
Command (m for help): p
Disk /dev/vdb: 1073.7 GB, 1073741824000 bytes
16 heads, 63 sectors/track, 2080507 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x2c594bf0

Device Boot Start End Blocks Id System
/dev/vdb1 1 2080507 1048575496+ 83 Linux

(2) 删除当前分区。当屏幕显示"已选择分区 1
d delete a partition
d

(3)新建分区
n add a new partition
n

(4)保存修改并退出
w write table to disk and exit
w
The partition table has been altered!

----------------------------
如果fstab有自动挂载则会出现这样的提示,需要提前执行第4步。
Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
---------------------------
8、执行命令 lsblk /dev/vdb 查看当前文件系统是否已增加分区表
lsblk /dev/vdb
#lsblk /dev/vdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vdb 252:16 0 1000G 0 disk
└─vdb1 252:17 0 1000G 0 part /data

9、执行命令 e2fsck -n /dev/vdb1 查看当前文件系统的状态是否为 clean
e2fsck -n /dev/vdb1
如果有异常则会提示
/dev/vdb1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vdb1: 5641753/26214400 files (2.5% non-contiguous), 97404495/104857570 blocks

修复
fsck.ext4 -C0 /dev/vdb1

10、通知内核更新分区表 通知内核需同步数据盘的分区表信息。
yum -y install parted
partprobe /dev/vdb

11、完成文件系统扩展
resize2fs /dev/vdb1
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/vdb1 to 262142637 (4k) blocks.
The filesystem on /dev/vdb1 is now 262142637 blocks long.

Web服务器数据磁盘采用 ext4 文件系统,可以执行 resize2fs /dev/vdb1 命令以扩展文件系统大小。
对于 xfs 文件系统,需要先运行 mount /dev/vdb1 /data/ 命令,再运行xfs_growfs /dev/vdb1 以完成文件系统拓展。

12、编辑fstab,启用Web服务器数据磁盘自动挂载
vi /etc/fstab
/dev/vdb1 /data ext4 defaults 0 0

13、挂载全部
mount -a

14、确认文件系统扩展完毕
df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda1 ext4 20G 8.3G 11G 45% /
tmpfs tmpfs 12G 0 12G 0% /dev/shm
/dev/vdb1 ext4 985G 366G 569G 40% /data
/dev/vdc1 ext4 394G 270G 105G 73% /backup

---------------------------
https://cloud.baidu.com/doc/CDS/s/Lk0629a17

CentOS7.7安装 devtoolset-8(使用高版本gcc (GCC) 8.3.1 20190311)编译安装aria2-1.35.0

Aria2是一个支持HTTP/HTTPS, FTP, SFTP, BitTorrent和Metalink等协议的轻量级命令行下载工具。
编译安装aria2-1.35.0的时候出现错误,提示需要高版本gcc支持。
......
checking whether g++ supports C++11 features with -std=c++11 ... no
checking whether g++ supports C++11 features with -std=c++11 -stdlib=libc++... no
checking whether g++ supports C++11 features with -std=c++0x ... no
checking whether g++ supports C++11 features with -std=c++0x -stdlib=libc++... no
configure: error: *** A compiler with support for C++11 language features is required.
......

CentOS7.7默认的GCC版本为gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
#gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
#cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)

解决方法有两种
1、手动编译gcc>4.8的版本
自行编译
2、安装devtoolset-8(使用高版本gcc (GCC) 8.3.1 20190311
yum install centos-release-scl -y
yum install devtoolset-8 -y
启用devtoolset-8
#scl enable devtoolset-8 -- bash
#source /opt/rh/devtoolset-8/enable

临时编译前使用高版本gcc (GCC) 8.3.1 20190311 (推荐使用这个方法)
export CC=/opt/rh/devtoolset-8/root/usr/bin/gcc
export CPP=/opt/rh/devtoolset-8/root/usr/bin/cpp
export CXX=/opt/rh/devtoolset-8/root/usr/bin/c++
继续编译安装aria2-1.35.0
......

CentOS6安装devtoolset(使用高版本gcc)GCC 4.8 GCC 4.9 GCC 5.2

CentOS6安装devtoolset(使用高版本gcc)GCC 4.8 GCC 4.9 GCC 5.2

利用.user.ini设置open_basedir加强php系统安全性

利用.user.ini设置open_basedir加强php系统安全性,从而限制PHP脚本动作只发生在指定目录里面,从而一定程度加强系统安全性。

1、.user.ini配置
要使.user.ini生效,需要修改php.ini 中的这两个参数#http://php.net/manual/zh/configuration.file.per-user.php
user_ini.filename = ".user.ini" #取消前面的;
user_ini.cache_ttl = 300 #取消前面的;

在网站项目根目录/data/html/www.zhangfangzhou.cn/创建 .user.ini文件,写入如下内容
vi .user.ini
open_basedir=/data/html/www.zhangfangzhou.cn/:/tmp/:/proc/

open_basedir中用冒号分割,分割成了3个路径。
open_basedir一定要以斜线结尾,如果没有用斜线结尾的话,则代表的是一系列以对应名称开头的目录。比如/user 代表的就是 /user(*)/ ,所以一定要注意斜线结尾。

2、设置权限防止被恶意删除导致open_basedir限制失效
chattr +i /data/html/www.zhangfangzhou.cn/.user.ini #通过chattr命令修改.user.ini的Immutable属性

chattr - 更改Linux文件系统上的文件属性
a Append Only,系统只允许在这个文件之后追加数据,不允许任何进程覆盖或截断这个文件。如果目录具有这个属性,系统将只允许在这个目录下建立和修改文件,而不允许删除任何文件。
i Immutable,系统不允许对这个文件进行任何的修改。如果目录具有这个属性,那么任何的进程只能修改目录之下的文件,不允许建立和删除文件。

3、取消.user.ini的Immutable属性(可以删除.user.ini)
chattr -i /data/html/www.zhangfangzhou.cn/.user.ini

echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.

tail /var/log/messages
echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 19 05:27:57 web kernel: Call Trace:
May 19 05:27:57 web kernel: [<ffffffff811d10a0>] ? sync_buffer+0x0/0x50
May 19 05:27:57 web kernel: [<ffffffff81549183>] io_schedule+0x73/0xc0
May 19 05:27:57 web kernel: [<ffffffff811d10e0>] sync_buffer+0x40/0x50
May 19 05:27:57 web kernel: [<ffffffff81549c6f>] __wait_on_bit+0x5f/0x90
May 19 05:27:57 web kernel: [<ffffffff811d10a0>] ? sync_buffer+0x0/0x50
May 19 05:27:57 web kernel: [<ffffffff81549d18>] out_of_line_wait_on_bit+0x78/0x90
May 19 05:27:57 web kernel: [<ffffffff810a6920>] ? wake_bit_function+0x0/0x50
May 19 05:27:57 web kernel: [<ffffffff811d1096>] __wait_on_buffer+0x26/0x30
May 19 05:27:57 web kernel: [<ffffffffa00707ef>] jbd2_journal_commit_transaction+0x117f/0x14f0 [jbd2]
May 19 05:27:57 web kernel: [<ffffffff8108fb2b>] ? try_to_del_timer_sync+0x7b/0xe0
May 19 05:27:57 web kernel: [<ffffffffa0075a38>] kjournald2+0xb8/0x220 [jbd2]
May 19 05:27:57 web kernel: [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
May 19 05:27:57 web kernel: [<ffffffffa0075980>] ? kjournald2+0x0/0x220 [jbd2]
May 19 05:27:57 web kernel: [<ffffffff810a640e>] kthread+0x9e/0xc0
May 19 05:27:57 web kernel: [<ffffffff8100c28a>] child_rip+0xa/0x20
May 19 05:27:57 web kernel: [<ffffffff810a6370>] ? kthread+0x0/0xc0
May 19 05:27:57 web kernel: [<ffffffff8100c280>] ? child_rip+0x0/0x20
May 19 05:28:40 web kernel: end_request: I/O error, dev vdb, sector 564522839
May 19 05:33:42 web kernel: end_request: I/O error, dev vdb, sector 564522839
May 19 05:38:44 web kernel: end_request: I/O error, dev vdb, sector 564522839

进程等待IO时,经常处于D状态,即TASK_UNINTERRUPTIBLE状态,处于这种状态的进程不处理信号,所以kill不掉,如果进程长期处于D状态,那么肯定不正常,
原因可能有二
1)IO路径上的硬件出问题了,比如硬盘坏了(只有少数情况会导致长期D,通常会返回错误)
2)内核自己出问题了
这种问题不好定位,而且一旦出现就通常不可恢复,kill不掉,通常只能重启恢复了。
内核针对这种开发了一种hung task的检测机制。
基本原理是:定时检测系统中处于D状态的进程,如果其处于D状态的时间超过了指定时间(默认120s,可以配置),则打印相关堆栈信息,也可以通过proc参数配置使其直接panic。

1、查看是否存在坏块
/sbin/badblocks -v /dev/sdc

2、问题分析
May 19 05:27:57 web kernel: [<ffffffff811d10a0>] ? sync_buffer+0x0/0x50
May 19 05:27:57 web kernel: [<ffffffff81549183>] io_schedule+0x73/0xc0
May 19 05:27:57 web kernel: [<ffffffff811d10e0>] sync_buffer+0x40/0x50
May 19 05:27:57 web kernel: [<ffffffff81549c6f>] __wait_on_bit+0x5f/0x90

3、临时方案
根据应用程序情况,对vm.dirty_ratio,vm.dirty_background_ratio两个参数进行调优设置。
# sysctl -w vm.dirty_ratio=10
# sysctl -w vm.dirty_background_ratio=5
# sysctl -p

如果系统永久生效,修改/etc/sysctl.conf文件。
#vi /etc/sysctl.conf

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

重启系统生效
http://www.361way.com/kernel-hung-task-analysis/4326.html

/dev/vda1 Inodes that were part of a corrupted orphan linked list found.

/dev/vda1 contains a file system with errors, check forced.
/dev/vda1 Inodes that were part of a corrupted orphan linked list found.
/dev/vda1 UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

#检查文件系统
1、ext4文件系统
fsck -y /dev/vda1
或者
fsck.ext4 -a /dev/vda1

2、xfs文件系统
xfs_repair -n /dev/vda1 #检查文件系统是否损坏,只检查文件系统是否有损坏

fsck.ext4 [-panyrcdfvtDFV] [-b superblock] [-B blocksize]
[-I inode_buffer_blocks] [-P process_inode_size]
[-l|-L bad_blocks_file] [-C fd] [-j external_journal]
[-E extended-options] device

Emergency help:
-p Automatic repair (no questions)
-n Make no changes to the filesystem
-y Assume "yes" to all questions
-c Check for bad blocks and add them to the badblock list
-f Force checking even if filesystem is marked clean
-v Be verbose
-b superblock Use alternative superblock
-B blocksize Force blocksize when looking for superblock
-j external_journal Set location of the external journal
-l bad_blocks_file Add to badblocks list
-L bad_blocks_file Set badblocks list

Usage: xfs_repair [options] device
Options:
-f The device is a file
-L Force log zeroing. Do this as a last resort.
-l logdev Specifies the device where the external log resides.
-m maxmem Maximum amount of memory to be used in megabytes.
-n No modify mode, just checks the filesystem for damage.
-P Disables prefetching.
-r rtdev Specifies the device where the realtime section resides.
-v Verbose output.
-c subopts Change filesystem parameters - use xfs_admin.
-o subopts Override default behaviour, refer to man page.
-t interval Reporting interval in seconds.
-d Repair dangerously.
-V Reports version and exits.
https://support.microsoft.com/en-in/help/3213321/linux-recovery-cannot-ssh-to-linux-vm-due-to-file-system-errors-fsck

Apache 开启SSI配置使shtml支持 include()和SSI Shell漏洞问题

Apache 开启SSI配置使shtml支持 include()和SSI Shell漏洞问题
SSI (Server Side Includes)

1、编辑Apache的配置文件httpd.conf添加.shtml支持
# To parse .shtml files for server-side includes (SSI):
# (You will also need to add "Includes" to the "Options" directive.)
#
#AddType text/html .shtml #取消该行前的注释符#
#AddOutputFilter INCLUDES .shtml #取消该行前的注释符#
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml

2、编辑www.zhangfangzhou.cn.conf网站配置文件中添加Includes
Options FollowSymLinks Includes

3、重启Apache
service httpd restart
这样Apache2 开启SSI配置使shtml支持 include(),但是留下了一个SSI Shell漏洞问题,此时可以程序执行(exec)

4、只允许SSI但是禁止程序执行(exec)
编辑www.zhangfangzhou.cn.conf网站配置文件
Options FollowSymLinks Includes修改成Options FollowSymLinks IncludesNoExec
或者在.htaccess文件中添加Options +IncludesNOEXEC

5、补充
Includes #启用SSI
IncludesNoExec #启用SSI,但使EXEC指令无效

(1)、Includes
Options FollowSymLinks Includes以下规范有效
<!--#exec cmd="..."-->
<!--#exec cgi="..."-->
<!--#include file="..."-->

(2)、IncludesNoExec
Options FollowSymLinks IncludesNoExec
<!--#include file="..."-->
规范有效

<!--#exec cmd="..."-->
<!--#exec cgi="..."-->
exec等程序执行将受到限制不能执行