I finally got around to taking a look at the digital camera setup today. Rob had complained the client had stopped working on Rosalba.
After looking at the code start up and not complain, yet not produce any window output, it looks like it was a network problem. I tried rebooting Rosalba, but that didn't fix anything.
Using netstat -an, I looked for the port 5010 on both rosalba and ottavia, since that is the port that was being used by the camera. Ottavia was saying there were 6 established connections after Rosalba had rebooted (rosalba is 22.214.171.124). I can only presume 6 instances of the camera code had somehow shutdown in such a way they had not closed the connection.
[root@ottavia controls]#netstat -an | grep 5010
tcp 0 0 0.0.0.0:5010 0.0.0.0:* LISTEN
tcp 0 0 126.96.36.199:5010 188.8.131.52:57366 ESTABLISHED
tcp 0 0 184.108.40.206:5010 220.127.116.11:58417 ESTABLISHED
tcp 1 0 18.104.22.168:46459 22.214.171.124:5010 CLOSE_WAIT
tcp 0 0 126.96.36.199:5010 188.8.131.52:57211 ESTABLISHED
tcp 0 0 184.108.40.206:5010 220.127.116.11:57300 ESTABLISHED
tcp 0 0 18.104.22.168:5010 22.214.171.124:57299 ESTABLISHED
tcp 0 0 126.96.36.199:5010 188.8.131.52:57315 ESTABLISHED
I switched the code to use port 5022 which worked fine. However, I'm not sure what would have caused the original connection closure failures, as I test several close methods (including the kill command on the server end used by the medm screen), and none seemed to generate this broken connection state. I rebooted Ottavia, and this seemed to fix the connections, and allowed port 5010 to work. I also tried creating 10 connections, which all seem to run fine simultaneously. So its not someone overloading that port with too many connections which caused the problem. Its like the the port stopped working somehow, which froze the connection status, but how or why I don't know at this point.