OT: Apache fun - can't explain crashes...
A coworker of mine had a Tomcat-driven site switched over to JVM 1.4.2
from JVM 1.3.1. He got an email from the client about two hours later
saying they were getting a bunch of errors:
Internal Servlet Error:
javax.servlet.ServletException: common.cException: common.cException:
java.io.FileNotFoundException: /space/icrp/WEB-INF/icrp.properties (Too
many open files)
at
org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:461)
at
_0002ferror_0002ejsperror_jsp_5._jspService(_0002ferror_0002ejsperror_jsp_5.java:187)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:119)
...
Root cause:
common.cException: common.cException: common.cException:
java.io.FileNotFoundException: /space/icrp/WEB-INF/icrp.properties (Too
many open files)
at common.cUtils.closeConnections(cUtils.java:871)
at
_0002ferror_0002ejsperror_jsp_5._jspService(_0002ferror_0002ejsperror_jsp_5.java:177)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:119)
...
After he got the email he restarted the site only to have it behave in a
similar fashion over half an hour later. We scoured online with the above
Java error msg to no avail.
My initial thought was to peruse the Apache logs to see if there were
entries that correlated to when the crashes occured and what were the
resulting message; this is what I found:
143.65.99.20 - - [07/Mar/2005:20:38:51 -0500] "GET /wizsearch.jsp?add=...
HTTP/1.1" 500 612 "http://www..." "Mozilla/4.0 (compatible; MSIE 6.0;
Windows NT 5.1)"
217.44.173.38 - - [07/Mar/2005:20:38:51 -0500] "GET
/abstract.jsp?SID=67466&ProjectID=164428 HTTP/1.1" 500 612
"http://www.../projectlist.jsp?SID=67466&ShowCart=false" "Mozilla/5.0
(Macintosh; U; PPC Mac OS X; en-gb) AppleWebKit/125.5.5 (KHTML, like
Gecko) Safari/125.12"
212.174.145.126 - - [07/Mar/2005:20:38:51 -0500] "GET / HTTP/1.1" 500 607
"-" "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.5)
Gecko/20041107 Firefox/1.0"
169.142.1.1 - - [07/Mar/2005:20:38:51 -0500] "GET /servlet/search/?CSO=5.5
HTTP/1.1" 302 448 "http://www.google.com/search?q=..." "Mozilla/4.0
(compatible; MSIE 6.0; Windows 98)"
62.128.179.3 - - [08/Mar/2005:08:23:57 -0500] "GET /error.jsp?SID=68030
HTTP/1.1" 500 612 "http://www.../projectlist.jsp?SID=68030"
"Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.0)"
62.128.179.2 - - [08/Mar/2005:08:23:57 -0500] "GET /error.jsp?SID=68031
HTTP/1.0" 500 600 "http://www.../wizsearch.jsp?add=..." "Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.0)"
[Mon Mar 7 20:38:51 2005] [info] [client 169.142.1.1] (32)Broken pipe:
client stopped connection before rflush completed
[Mon Mar 7 20:38:51 2005] [info] [client 217.44.173.38] (32)Broken pipe:
client stopped connection before rflush completed
[Mon Mar 7 20:38:51 2005] [info] [client 143.65.99.20] (32)Broken pipe:
client stopped connection before rflush completed
[Mon Mar 7 20:38:51 2005] [info] [client 212.174.145.126] (32)Broken
pipe: client stopped connection before rflush completed
[Tue Mar 8 08:23:57 2005] [info] [client 62.128.179.3] (32)Broken pipe:
client stopped connection before rflush completed
[Tue Mar 8 08:23:57 2005] [info] [client 62.128.179.2] (32)Broken pipe:
client stopped connection before rflush completed
I found out the "rflush" message normally meant that a user has pressed
the browser's "stop" button, but to have the same messages appear
simultaneously 25-30 times from the same 2-4 IPs seems kind of fishy. I
did a nslookup and I don't think those IPs belong to indexing services.
Has anyone seen anything like this, and if so, what does this mean? And
is this really a Tomcat problem, Apache problem, or a problem caused by
something else?
BTW this site is currently using Apache 1.3.26; would we be able to get
rid of this issue if we started using v2.0.xx?
Thanks....
--
Joe Tseng
"I can be Googled. Therefore I am."