Tim Makins: February 2008 Archives

My website was recently having a problem accessing pages with active content, driven from the cgi bin. When I type their address in my browser, I get a '500 Internal Server Error' page. Its text read as follows:

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator and inform them of the time the error occurred, and anything you might have done that may have caused the error.

Apache/1.3.39 Server at www.mapability.com Port 80

When I check in the Error Log, it says: '(24)Too many open files: couldn't spawn child process:'. How to determine what is causing this problem, and what is opening all these files? I have a number of active pages that access various databases - it could be any one of them. My server is running Apache. I also use MySQL and php pages. Maybe there is some script that is not working correctly, and is leaving open files - is there a way to find this out, and what would be the method of fixing it?

I never did determine a way to check which files were open, or if there was some way to close them again. Many search results suggested that I increase the limit of open files so that the error would go away, but that seemed rather a silly way to address the problem - I would rather find out what was causing it in the first place, and fix that. More searching, and I eventually found an interesting page on the Forum belonging to Gallery 2, the photo-gallery software that I use. It mentioned a problem that another user was having with the 'Too many open files: couldn't spawn child process' error, and suggested a simple fix. The pertinent answer read:

Our guess is that you used file / flock based locking ("site admin", bottom of the page has locking options).
If you used database based locking, an open file limit of ~100 should have been sufficient.

Of course, once I knew what the answer was, it was easy to find more occurrences, using a Search such as "too many open files" "gallery 2". I soon discovered that this was a 'Known Issue' for Gallery 2.

 

Tim Will Fix It
I made the change myself. Initially, the error still occurred, though not quite as much as before.

To change this yourself, login to your Gallery, and click on Site Admin >> General Settings. Down the bottom of the page is a section:

Locking System
Gallery uses a system of locks to prevent simultaneous changes from interfering with each other. There are two types of locking, each with its advantages and disadvantages. File based locking is fast and efficient, but won't work on NFS filesystems and will be unreliable on Windows. Database locking is slower but is more reliable. If you are unsure which to choose, we recommend using file locking. If you're getting many lock timeouts, you can try switching to database locking instead. It's ok to switch back and forth.

The default setting for 'Lock system ' is File, so change it to Database and click 'Save'. The problem started improving straight away, but took a few days before the system finally settled down again. No doubt a gradual improvement will be the same in your case, too.

Ever had your website taken over? Its no fun. The other day I logged on to find that my site was partially infested with files that I hadn't put there. In certain directories there was an .htaccess file, and another with random numbers then .php - for example something like 180185.php. Had I put them there? Were they files belonging to my Blog software, Movable Type? I opened up the .htaccess in Notepad, and discovered that it took control of all 404 errors (File does not exist) and directed all requests to the numbered .php file, which then opened up a page listing all kinds of mobile phone hacks etc. A LOT of bandwidth was being used, besides filling the Google cache with all kinds of illegal rubbish purporting to come from me.

How had this got there? After a lot of searching, and a lot of red-herrings, I finally tracked down the cause to my Blogging software, Movable Type. It turned out that when installed, its settings are such that all new entries to the Blog, plus all their indexes and archives, are stored in folders with permissions set to 777. All new files have their permissions set to 666. Is that a problem? Yes, because it allows ANYONE to write to those folders, as happened to me. This page: 'Securing your weblog - CGIWrap and SuEXEC' explains the problem in greater detail as well as describing the fix.

Basically, there are two things you need to do. First, you open your 'mt-check.cgi' page in a browser - you will find it in your 'mt' directory. If (and ONLY if) it tells you "(Probably) Running under cgiwrap or suexec" then you can add an extra layer of Security to your site, by adding some lines of text to your 'mt-config.cgi' file. All you do is to open the file with Notepad or similar (NOT a word processor) and add the following lines at the bottom:

#
#=ADDED TO SET CORRECT PERMISSIONS FOR FILES AND FOLDERS=
#
DBUmask 0022
DirUmask 0022
HTMLUmask 0022
UploadUmask 0022

Save the file, and try a new entry to your Blog. You will find that the new entry has its permissions set to 644, and the folder it is in has its permissions set to 755. SUCCESS !! Now all you have to do is to go through your site and set the permissions for all those older Movable Type entry-files and folders to the above value as well!

CAVEAT: The above worked fine for me. I make no guarantees it will work for you, so would suggest you alter things and test things one step at a time, and make sure you know how to recover if you have made a mistake.