Odesskiy Listok site went down last night for no apparent reason. No software or operating system upgrades were performed that could have potentially caused this failure. Hardware was also fully functional. After a rather long investigation, we discovered that the cause of this problem was Clean URLs, which suddenly stopped working. Instead of redirecting, as they should have and did before, they were causing "The page cannot be displayed" (Cannot find server or DNS Error) error messages to be displayed in Internet Explorer and blank pages in Firefox browser.

The problem could be averted by commenting out "RewriteEngine on" in the .htaccess file and turning Clean URLs off (in Drupal 4.7.x, for example, this is done by going directly to ?q=admin/settings). These actions brought the site back to life. However for full functionality, we still needed to get Clean URLs working again.

Upon further investigation, we discovered that the reason Clean URLs were no longer working was that the Apache module mod_rewrite was failing to do its work. Performing latest security updates to Apache and the kernel did not fix modrewrite behavior. The problem was eventually solved by removing the rewrite log file, which had reached 2 Gigabytes in size.

LESSON LEARNED: If you are logging anything on a machine running SuSe Linux, make sure that you either rotate your log files or delete them periodically. If any of your log files ever reach 2 Gigabytes, you are guaranteed to have problems.