Apache起不来,报No space left on device: Couldn't create accept lock
今天早上,罗明在搭建测试环境的时候,发现启动MA应用后,访问总是出错。报错特别快,怀疑是apache没有起来,telnet apache端口,发现果然没有起来。
forrest@ubuntu:~$ telnet 10.20.142.12 2080
Trying 10.20.142.12...
telnet: Unable to connect to remote host: Connection refused
但是启动脚本显示是已经成功起来,改成访问jboss端口就可以了。
forrest@ubuntu:~$ telnet 10.20.142.12 2087
Trying 10.20.142.12...
Connected to 10.20.142.12.
Escape character is '^]'.
但是问题总是要解决。我以前遇到类似的问题,就是访问jboss端口是可以的,但是访问apache端口则总是报错,但是是报404的错误,说明apache已经起来,但是是mod_jk出错了。具体可以参见笔者以前写的一篇文章:后翌虚拟机环境搭建日志(简言之,就是添加JkMountCopy All配置项)。
一边单独启动apache,一边查看log日志:
scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/bin]./apachectl restart
./apachectl restart: httpd not running, trying to start
scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/logs]tailf error_log
[Fri Oct 08 18:41:35 2010] [warn] pid file /home/scrm/work/intl-myalibaba/deploy/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Fri Oct 08 18:41:35 2010] [emerg] (28)No space left on device: Couldn't create accept lock
[Fri Oct 08 18:42:44 2010] [warn] pid file /home/scrm/work/intl-myalibaba/deploy/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Fri Oct 08 18:42:44 2010] [emerg] (28)No space left on device: Couldn't create accept lock
发现报了这么一个奇怪的错误:
[emerg] (28)No space left on device: Couldn't create accept lock。
以为是空间不足,造成无法创建锁文件,但是df -h察看,发觉不是这个问题。
scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/logs]df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda3 4.9G 178M 4.4G 4% /
/dev/xvda1 99M 11M 83M 12% /boot
none 2.6G 0 2.6G 0% /dev/shm
/dev/xvda8 68G 48G 17G 74% /home
/dev/xvda5 2.9G 42M 2.7G 2% /tmp
/dev/xvda2 9.7G 6.1G 3.1G 67% /usr
/dev/xvda6 2.9G 166M 2.6G 6% /var
10.20.131.137:/home/efs/www
34G 23G 8.8G 73% /mnt/efs/www
10.20.134.136:/home/efs/www/escrow
34G 3.2G 29G 10% /mnt/escrow
google了一下,发现早已经有人遇到同样的问题了。
scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/logs]ipcs -s | grep scrm
0x00000000 1212419 scrm 600 1
0x00000000 1245188 scrm 600 1
0x00000000 1277957 scrm 600 1
0x00000000 18579474 scrm 600 1
0x00000000 18612243 scrm 600 1
0x00000000 18645012 scrm 600 1
0x00000000 53444632 scrm 600 1
0x00000000 53477401 scrm 600 1
0x00000000 53510170 scrm 600 1
0x00000000 19890215 scrm 600 1
0x00000000 19922984 scrm 600 1
0x00000000 19955753 scrm 600 1
0x00000000 21692458 scrm 600 1
0x00000000 21725227 scrm 600 1
0x00000000 21757996 scrm 600 1
0x00000000 22347824 scrm 600 1
0x00000000 22380593 scrm 600 1
0x00000000 22413362 scrm 600 1
0x00000000 25296950 scrm 600 1
0x00000000 25329719 scrm 600 1
0x00000000 25362488 scrm 600 1
0x00000000 38895685 scrm 600 1
0x00000000 38928454 scrm 600 1
0x00000000 38961223 scrm 600 1
0x00000000 36438091 scrm 600 1
0x00000000 36470860 scrm 600 1
0x00000000 36503629 scrm 600 1
0x00000000 29720660 scrm 600 1
0x00000000 29753429 scrm 600 1
0x00000000 29786198 scrm 600 1
0x00000000 29884503 scrm 600 1
0x00000000 29917272 scrm 600 1
0x00000000 29950041 scrm 600 1
0x00000000 30539866 scrm 600 1
0x00000000 30572635 scrm 600 1
0x00000000 30605404 scrm 600 1
0x00000000 39059564 scrm 600 1
0x00000000 39092333 scrm 600 1
0x00000000 39125102 scrm 600 1
0x00000000 40697970 scrm 600 1
0x00000000 40730739 scrm 600 1
0x00000000 40763508 scrm 600 1
察看发觉有一大堆的ipc使用,需要干掉。
![](http://b2b-doc.alibaba-inc.com/images/icons/emoticons/check.gif)
不加-s参数,将打印所有IPC信息。
scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/logs]ipcs
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
------ Semaphore Arrays --------
key semid owner perms nsems
------ Message Queues --------
key msqid owner perms used-bytes messages
清除命令:
scrm@intlqa142012x:[/home/scrm]for semid in `ipcs -s | grep scrm | cut -f2 -d" "`;
> do ipcrm -s $semid; done
再次grep一下,发现已经没有了。
scrm@intlqa142012x:[/home/scrm]ipcs -s | grep scrm
然后启动Apache即可。
![](http://b2b-doc.alibaba-inc.com/images/icons/emoticons/check.gif)
查看the number of semaphores that are available on the system
admin@intlqa142012x:[/home/admin]ipcs -l
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
------ Messages: Limits --------
max queues system wide = 16
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384
如果有权限的话,可以修改这些限制:
To change these parameters, modify the file /etc/sysctl.conf and add the following lines:
kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024
Then load these settings with the command:sysctl -p
-p Load in sysctl settings from the file specified or /etc/sysctl.conf if none given.
admin@intlqa142012x:[/home/admin]sysctl -p
error: permission denied on key 'net.ipv4.ip_forward'
error: permission denied on key 'net.ipv4.conf.default.rp_filter'
error: permission denied on key 'net.ipv4.conf.default.accept_source_route'
error: permission denied on key 'kernel.sysrq'
error: permission denied on key 'kernel.core_uses_pid'
Your Apache process should now be able to create the needed semaphores and run properly.
说明:kernel.msgmni
该文件指定消息队列标识的最大数目,即系统范围内最大多少个消息队列。缺省设置:16。
admin@intlqa142012x:[/home/admin]cat /proc/sys/kernel/msgmni
16
在root下可用sysctl kernel.msgmni检查该参数:
admin@intlqa142012x:[/home/admin]sysctl kernel.msgmni
kernel.msgmni = 16
也可以在命令行下,使用sysctl -w kernel.msgmni=XXX重新设定。
admin@intlqa142012x:[/home/admin]sysctl -w kernel.msgmni=32
error: permission denied on key 'kernel.msgmni'
forrest@ubuntu:~$ telnet 10.20.142.12 2080 Trying 10.20.142.12... telnet: Unable to connect to remote host: Connection refused
forrest@ubuntu:~$ telnet 10.20.142.12 2087 Trying 10.20.142.12... Connected to 10.20.142.12. Escape character is '^]'.
scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/bin]./apachectl restart ./apachectl restart: httpd not running, trying to start
scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/logs]tailf error_log [Fri Oct 08 18:41:35 2010] [warn] pid file /home/scrm/work/intl-myalibaba/deploy/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run? [Fri Oct 08 18:41:35 2010] [emerg] (28)No space left on device: Couldn't create accept lock [Fri Oct 08 18:42:44 2010] [warn] pid file /home/scrm/work/intl-myalibaba/deploy/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run? [Fri Oct 08 18:42:44 2010] [emerg] (28)No space left on device: Couldn't create accept lock
[emerg] (28)No space left on device: Couldn't create accept lock。
scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/logs]df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda3 4.9G 178M 4.4G 4% /
/dev/xvda1 99M 11M 83M 12% /boot
none 2.6G 0 2.6G 0% /dev/shm
/dev/xvda8 68G 48G 17G 74% /home
/dev/xvda5 2.9G 42M 2.7G 2% /tmp
/dev/xvda2 9.7G 6.1G 3.1G 67% /usr
/dev/xvda6 2.9G 166M 2.6G 6% /var
10.20.131.137:/home/efs/www
34G 23G 8.8G 73% /mnt/efs/www
10.20.134.136:/home/efs/www/escrow
34G 3.2G 29G 10% /mnt/escrow
scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/logs]ipcs -s | grep scrm 0x00000000 1212419 scrm 600 1 0x00000000 1245188 scrm 600 1 0x00000000 1277957 scrm 600 1 0x00000000 18579474 scrm 600 1 0x00000000 18612243 scrm 600 1 0x00000000 18645012 scrm 600 1 0x00000000 53444632 scrm 600 1 0x00000000 53477401 scrm 600 1 0x00000000 53510170 scrm 600 1 0x00000000 19890215 scrm 600 1 0x00000000 19922984 scrm 600 1 0x00000000 19955753 scrm 600 1 0x00000000 21692458 scrm 600 1 0x00000000 21725227 scrm 600 1 0x00000000 21757996 scrm 600 1 0x00000000 22347824 scrm 600 1 0x00000000 22380593 scrm 600 1 0x00000000 22413362 scrm 600 1 0x00000000 25296950 scrm 600 1 0x00000000 25329719 scrm 600 1 0x00000000 25362488 scrm 600 1 0x00000000 38895685 scrm 600 1 0x00000000 38928454 scrm 600 1 0x00000000 38961223 scrm 600 1 0x00000000 36438091 scrm 600 1 0x00000000 36470860 scrm 600 1 0x00000000 36503629 scrm 600 1 0x00000000 29720660 scrm 600 1 0x00000000 29753429 scrm 600 1 0x00000000 29786198 scrm 600 1 0x00000000 29884503 scrm 600 1 0x00000000 29917272 scrm 600 1 0x00000000 29950041 scrm 600 1 0x00000000 30539866 scrm 600 1 0x00000000 30572635 scrm 600 1 0x00000000 30605404 scrm 600 1 0x00000000 39059564 scrm 600 1 0x00000000 39092333 scrm 600 1 0x00000000 39125102 scrm 600 1 0x00000000 40697970 scrm 600 1 0x00000000 40730739 scrm 600 1 0x00000000 40763508 scrm 600 1
![]() |
不加-s参数,将打印所有IPC信息。
scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/logs]ipcs ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status ------ Semaphore Arrays -------- key semid owner perms nsems ------ Message Queues -------- key msqid owner perms used-bytes messages |
scrm@intlqa142012x:[/home/scrm]for semid in `ipcs -s | grep scrm | cut -f2 -d" "`; > do ipcrm -s $semid; done
scrm@intlqa142012x:[/home/scrm]ipcs -s | grep scrm
![]() |
查看the number of semaphores that are available on the system
admin@intlqa142012x:[/home/admin]ipcs -l
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
------ Messages: Limits --------
max queues system wide = 16
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384
如果有权限的话,可以修改这些限制: kernel.msgmni = 1024 kernel.sem = 250 256000 32 1024 Then load these settings with the command:sysctl -p admin@intlqa142012x:[/home/admin]sysctl -p error: permission denied on key 'net.ipv4.ip_forward' error: permission denied on key 'net.ipv4.conf.default.rp_filter' error: permission denied on key 'net.ipv4.conf.default.accept_source_route' error: permission denied on key 'kernel.sysrq' error: permission denied on key 'kernel.core_uses_pid' Your Apache process should now be able to create the needed semaphores and run properly. 说明:kernel.msgmni admin@intlqa142012x:[/home/admin]cat /proc/sys/kernel/msgmni 16 在root下可用sysctl kernel.msgmni检查该参数: admin@intlqa142012x:[/home/admin]sysctl kernel.msgmni kernel.msgmni = 16 也可以在命令行下,使用sysctl -w kernel.msgmni=XXX重新设定。 admin@intlqa142012x:[/home/admin]sysctl -w kernel.msgmni=32 error: permission denied on key 'kernel.msgmni' |