Apache Failed, Semaphore Errors

Found a really weird proble related to apache, where the apache won’t start. I checked the config file for apache and found everything alright.

Looking into the logfiles I finally found the following:

[Mon Jul 16 20:29:32 2012] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed

Some servers may have the message in the following format:

[emerg] (28)No space left on device: Couldn’t create accept lock

The first thing I thought here was the disk space, so I checked the disk space on the server and to my surprise the / partition was having 25% free space.

I then moved to check free inodes and that one was found good as well.

This further confused things, till I finally realized semphores. The apache wasn’t terminated properly which might have left semaphores.

To confirm I used ipcs. The following is the command I used:

[[email protected] ~]# ipcs -s | grep apache

Where,

	apache is the username with which apache/http is running on the system. This can either apache or httpd or www or nobody. In my case it was apache.

The output did confirm the presence of semaphores left for apache.

0x00000000 49676288   apache    600        1         
0x00000000 49643521   apache    600        1         
0x00000000 49709058   apache    600        1         
0x00000000 49741827   apache    600        1         
0x00000000 49774596   apache    600        1         
0x00000000 49807365   apache    600        1         
0x00000000 49840134   apache    600        1         
0x00000000 49872903   apache    600        1         
0x00000000 59867144   apache    600        1         
0x00000000 59834377   apache    600        1         
0x00000000 59899914   apache    600        1         
0x00000000 59932683   apache    600        1         
0x00000000 59965452   apache    600        1         
0x00000000 59998221   apache    600        1         
0x00000000 60030990   apache    600        1         
0x00000000 60063759   apache    600        1         
0x00000000 78053392   apache    600        1         
0x00000000 78020625   apache    600        1         
0x00000000 78086162   apache    600        1         
0x00000000 78118931   apache    600        1         
0x00000000 78151700   apache    600        1         
0x00000000 78184469   apache    600        1         
0x00000000 78217238   apache    600        1         
0x00000000 78250007   apache    600        1         
0x00000000 80642072   apache    600        1         
0x00000000 80609305   apache    600        1         
0x00000000 80674842   apache    600        1         
0x00000000 80707611   apache    600        1         
0x00000000 80740380   apache    600        1         
0x00000000 80773149   apache    600        1         
0x00000000 80805918   apache    600        1         
0x00000000 80838687   apache    600        1         
0x00000000 87654432   apache    600        1         
0x00000000 87621665   apache    600        1         
0x00000000 87687202   apache    600        1         
0x00000000 87719971   apache    600        1         
0x00000000 87752740   apache    600        1         
0x00000000 87785509   apache    600        1         
0x00000000 87818278   apache    600        1         
0x00000000 87851047   apache    600        1         
0x00000000 108920872  apache    600        1         
0x00000000 108888105  apache    600        1         
0x00000000 108953642  apache    600        1         
0x00000000 108986411  apache    600        1         
0x00000000 109019180  apache    600        1         
0x00000000 109051949  apache    600        1         
0x00000000 109084718  apache    600        1         
0x00000000 109117487  apache    600        1         
0x00000000 112164912  apache    600        1         
0x00000000 112132145  apache    600        1         
0x00000000 112197682  apache    600        1         
0x00000000 112230451  apache    600        1         
0x00000000 112263220  apache    600        1         
0x00000000 112295989  apache    600        1         
0x00000000 112328758  apache    600        1         
0x00000000 112361527  apache    600        1         
0x00000000 128352312  apache    600        1         
0x00000000 128319545  apache    600        1         
0x00000000 128385082  apache    600        1         
0x00000000 128417851  apache    600        1         
0x00000000 128450620  apache    600        1         
0x00000000 128483389  apache    600        1         
0x00000000 128516158  apache    600        1         
0x00000000 128548927  apache    600        1         
0x00000000 135528512  apache    600        1         
0x00000000 135495745  apache    600        1         
0x00000000 135561282  apache    600        1         
0x00000000 135594051  apache    600        1         
0x00000000 135626820  apache    600        1         
0x00000000 135659589  apache    600        1         
0x00000000 135692358  apache    600        1         
0x00000000 135725127  apache    600        1         
0x00000000 137003080  apache    600        1         
0x00000000 136970313  apache    600        1         
0x00000000 137035850  apache    600        1         
0x00000000 137068619  apache    600        1         
0x00000000 137101388  apache    600        1         
0x00000000 137134157  apache    600        1         
0x00000000 137166926  apache    600        1         
0x00000000 137199695  apache    600        1         
0x00000000 141951056  apache    600        1         
0x00000000 141918289  apache    600        1         
0x00000000 141983826  apache    600        1         
0x00000000 142016595  apache    600        1         
0x00000000 142049364  apache    600        1         
0x00000000 142082133  apache    600        1         
0x00000000 142114902  apache    600        1         
0x00000000 142147671  apache    600        1         
0x00000000 144539736  apache    600        1         
0x00000000 144506969  apache    600        1         
0x00000000 144572506  apache    600        1         
0x00000000 144605275  apache    600        1         
0x00000000 144638044  apache    600        1         
0x00000000 144670813  apache    600        1         
0x00000000 144703582  apache    600        1         
0x00000000 144736351  apache    600        1         
0x00000000 144834656  apache    600        1         
0x00000000 144801889  apache    600        1         
0x00000000 144867426  apache    600        1         
0x00000000 144900195  apache    600        1         
0x00000000 144932964  apache    600        1         
0x00000000 144965733  apache    600        1         
0x00000000 144998502  apache    600        1         
0x00000000 145031271  apache    600        1         
0x00000000 150077544  apache    600        1         
0x00000000 150044777  apache    600        1         
0x00000000 150110314  apache    600        1         
0x00000000 150143083  apache    600        1         
0x00000000 150175852  apache    600        1         
0x00000000 150208621  apache    600        1         
0x00000000 150241390  apache    600        1         
0x00000000 150274159  apache    600        1         
0x00000000 152666224  apache    600        1         
0x00000000 152633457  apache    600        1         
0x00000000 152698994  apache    600        1         
0x00000000 152731763  apache    600        1         
0x00000000 152764532  apache    600        1         
0x00000000 152797301  apache    600        1         
0x00000000 152830070  apache    600        1         
0x00000000 152862839  apache    600        1         
0x00000000 155549816  apache    600        1         
0x00000000 155517049  apache    600        1         
0x00000000 155582586  apache    600        1         
0x00000000 155615355  apache    600        1         
0x00000000 155648124  apache    600        1         
0x00000000 155680893  apache    600        1         
0x00000000 155713662  apache    600        1         
0x00000000 155746431  apache    600        1

Using ipcrm we can remove these semaphores from the server in order to clear this off completely.

You can use any of the following commands/combinations to get rid of these semaphores.

ipcs -s | grep apache | awk ‘ { print $2 } ‘ | xargs ipcrm sem

or

for i in `ipcs -s | grep apache`; do ipcrm sem $i; done

or

ipcs -s | grep apache | perl -e ‘while () { @a=split(/\s+/); print `ipcrm sem $a[1]`}’

Start apache and it should come up cleanly.

No Comments

Post a Comment

Time limit is exhausted. Please reload CAPTCHA.