A buildup of sockets in CLOSE_WAIT state may indicate an application programming error.
A CLOSE_WAIT socket is a socket in which the connection has received a FIN and has yet to be closed by the application. In this state, it is the responsibility of the application to close the socket, although the operating system will report the socket in this state under any of these circumstances:
1- The socket is empty, and the application needs to relinquish the file descriptor of the socket back to the operating system, which will effectively remove the socket (and will subsequently disappear from netstat output).
2- There is still data in the socket which the application needs to read() from the file descriptor of the socket in order to be closed fully.
3- The connection is still in-use for unidirectional transmission by the application.
In all of the above scenarios, there is no tuning from an operating system perspective that can close this socket without killing the running process ID of the application, and subsequently closing the application. Otherwise, the application will need to close the socket via a call such as close().
A socket in CLOSE_WAIT may possibly consume TCP memory if it is not closed properly, of if the received data in the buffer is not read into the application, as this space will consume the total amount of TCP memory available on the system (visible in /proc/net/sockstat).
For Java, see also http://stackoverflow.com/questions/5636316/troubleshooting-connections-stuck-in-close-wait-status and http://serverfault.com/questions/160558/how-to-not-get-so-many-apache-close-wait-connections.
For PHP, Checked netstat command output and found more CLOSE_WAIT sessions. #netstat -an |grep CLOSE
Checked number of http process and ownership of httpd daemon using the below commands.
- #ps -aef |grep httpd #we found httpd daemon is with apache ownership.
- #ps -aef |grep http |wc -l #we found X number HTTP processes.
Root Cause: A CLOSE_WAIT socket is a socket state allowing the application to perform tasks on a socket that has been advised by the client that no further data will be sent by the client, before it is relinquished to the operating system. The socket is still usable for the application to send data to the client. Therefore, there is no direct way to tune this.
Possible causes include:
1- deliberate use in this mode by the application.
2- application fault resulting in failure to close the socket.
3- applications queries might have blocked in backend database, eg:- PHP has an open connection to the database; it has made a query and has yet to receive a response. Apache 'hanging' at that point and unable to close the connection after receiving FIN from client. To start troubleshooting on the database side, you should enable the slow query log if you haven't already done so. Have it log requests over 1 second or so and see what shows up when the problem occurs.
Note: The slow query log will not log the query until the query has completed. Do not assume that the first query showing up when the problem starts is the problem query. It may or may not be.
Let me conclude, CLOSE_WAIT issue you are having now is related to your application programming side, and I am sorry to say customer side application level troubleshooting is outside the scope AWS support . During our troubleshooting time we were able to identify the cause of your website stop responding due to high number of accumulated CLOSE_WAIT sockets, to fix the CLOSE_WAIT problem you may need to look in to your application side debugging.
