AWS Thinkbox Discussion Forums

pulse too many open files error

Hi there,

Our pulse hung up last night, and based on the logs it might have been due to bad file handles. It tries to close some, but can’t, and then it keeps accumulating new ones till it runs out.

2015-11-30 03:51:11:  Repository Repair Thread - Performing repository repair
2015-11-30 03:51:11:  Error running repository repair process: Error creating standard output pipe (System.IO.IOException)
2015-11-30 03:51:19:  WARNING: an error occurred while trying to close the input stream: Standard input has not been redirected (System.InvalidOperationException)
2015-11-30 03:51:20:  Pending Job Scan Thread - Performing pending job scan
2015-11-30 03:51:20:  Error running pending job scan process: Error creating standard output pipe (System.IO.IOException)
2015-11-30 03:51:23:  WARNING: an error occurred while trying to close the input stream: Standard input has not been redirected (System.InvalidOperationException)

then later:

2015-11-30 03:53:19:  Pending Job Scan Thread - Performing pending job scan
2015-11-30 03:53:19:  Server Thread - OnConnect: SocketException occurred: Too many open files
2015-11-30 03:53:19:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2015-11-30 03:53:19:  Exception Details
2015-11-30 03:53:19:  SocketException -- Too many open files
2015-11-30 03:53:19:  SocketException.ErrorCode: 10024 (Too many open files)
2015-11-30 03:53:19:  SocketException.SocketErrorCode: TooManyOpenSockets (10024)
2015-11-30 03:53:19:  Win32Exception.NativeErrorCode: 10024
2015-11-30 03:53:19:  Exception.HResult: -2147467259
2015-11-30 03:53:19:  Exception.Source: System
2015-11-30 03:53:19:  Exception.TargetSite: System.Net.Sockets.Socket Accept()
2015-11-30 03:53:19:  Exception.Data: ( )
2015-11-30 03:53:19:    Exception.StackTrace:
2015-11-30 03:53:19:    at System.Net.Sockets.Socket.Accept () [0x00000] in <filename unknown>:0
2015-11-30 03:53:19:    at System.Net.Sockets.Socket+Worker.Accept () [0x00000] in <filename unknown>:0
2015-11-30 03:53:19:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2015-11-30 03:53:19:  Server Thread - OnConnect: Restarting Pulse Server Thread because it exited prematurely!! (System.Exception)
2015-11-30 03:53:19:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2015-11-30 03:53:19:  Exception Details
2015-11-30 03:53:19:  Exception -- Restarting Pulse Server Thread because it exited prematurely!!
2015-11-30 03:53:19:  Exception.HResult: -2146233088
2015-11-30 03:53:19:  Exception.Data: ( )
2015-11-30 03:53:19:    Exception.StackTrace:
2015-11-30 03:53:19:      (null)
2015-11-30 03:53:19:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2015-11-30 03:53:19:  Server Thread - Waiting 5 seconds to restart the thread.

After a while, the entire pulse log is just this.

2015-11-30 04:33:26:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2015-11-30 04:33:26:  Exception Details
2015-11-30 04:33:26:  Exception -- Restarting Pulse Server Thread because it exited prematurely!!
2015-11-30 04:33:26:  Exception.HResult: -2146233088
2015-11-30 04:33:26:  Exception.Data: ( )
2015-11-30 04:33:26:    Exception.StackTrace:
2015-11-30 04:33:26:      (null)
2015-11-30 04:33:26:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2015-11-30 04:33:26:  Server Thread - Waiting 5 seconds to restart the thread.
2015-11-30 04:33:36:  Clean Up Thread - Performing house cleaning
2015-11-30 04:33:36:  Server Thread - OnConnect: SocketException occurred: Too many open files
2015-11-30 04:33:36:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2015-11-30 04:33:36:  Exception Details
2015-11-30 04:33:36:  SocketException -- Too many open files
2015-11-30 04:33:36:  SocketException.ErrorCode: 10024 (Too many open files)
2015-11-30 04:33:36:  SocketException.SocketErrorCode: TooManyOpenSockets (10024)
2015-11-30 04:33:36:  Win32Exception.NativeErrorCode: 10024
2015-11-30 04:33:36:  Exception.HResult: -2147467259
2015-11-30 04:33:36:  Exception.Source: System
2015-11-30 04:33:36:  Exception.TargetSite: System.Net.Sockets.Socket Accept()
2015-11-30 04:33:36:  Exception.Data: ( )
2015-11-30 04:33:36:    Exception.StackTrace:
2015-11-30 04:33:36:    at System.Net.Sockets.Socket.Accept () [0x00000] in <filename unknown>:0
2015-11-30 04:33:36:    at System.Net.Sockets.Socket+Worker.Accept () [0x00000] in <filename unknown>:0
2015-11-30 04:33:36:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2015-11-30 04:33:36:  Server Thread - OnConnect: Restarting Pulse Server Thread because it exited prematurely!! (System.Exception)
2015-11-30 04:33:36:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2015-11-30 04:33:36:  Exception Details
2015-11-30 04:33:36:  Exception -- Restarting Pulse Server Thread because it exited prematurely!!
2015-11-30 04:33:36:  Exception.HResult: -2146233088
2015-11-30 04:33:36:  Exception.Data: ( )
2015-11-30 04:33:36:    Exception.StackTrace:
2015-11-30 04:33:36:      (null)
2015-11-30 04:33:36:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2015-11-30 04:33:36:  Server Thread - Waiting 5 seconds to restart the thread.
2015-11-30 04:33:37:  Clean Up Thread - Collecting statistics
2015-11-30 04:33:38:  Pending Job Scan Thread - Performing pending job scan
2015-11-30 04:34:26:  Repository Repair Thread - Performing repository repair
2015-11-30 04:34:26:  Error running repository repair process: Error creating standard error pipe (System.IO.IOException)
2015-11-30 04:34:27:  WARNING: an error occurred while trying to close the input stream: Standard input has not been redirected (System.InvalidOperationException)
2015-11-30 04:34:37:  Clean Up Thread - Performing house cleaning
2015-11-30 04:34:37:  Error running housecleaning process: Error creating standard error pipe (System.IO.IOException)
2015-11-30 04:34:37:  Clean Up Thread - Collecting statistics
2015-11-30 04:34:39:  Pending Job Scan Thread - Performing pending job scan
2015-11-30 04:34:39:  Server Thread - OnConnect: SocketException occurred: Too many open files
2015-11-30 04:34:39:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2015-11-30 04:34:39:  Exception Details
2015-11-30 04:34:39:  SocketException -- Too many open files
2015-11-30 04:34:39:  SocketException.ErrorCode: 10024 (Too many open files)
2015-11-30 04:34:39:  SocketException.SocketErrorCode: TooManyOpenSockets (10024)
2015-11-30 04:34:39:  Win32Exception.NativeErrorCode: 10024
2015-11-30 04:34:39:  Exception.HResult: -2147467259
2015-11-30 04:34:39:  Exception.Source: System
2015-11-30 04:34:39:  Exception.TargetSite: System.Net.Sockets.Socket Accept()
2015-11-30 04:34:39:  Exception.Data: ( )
2015-11-30 04:34:39:    Exception.StackTrace:
2015-11-30 04:34:39:    at System.Net.Sockets.Socket.Accept () [0x00000] in <filename unknown>:0
2015-11-30 04:34:39:    at System.Net.Sockets.Socket+Worker.Accept () [0x00000] in <filename unknown>:0
2015-11-30 04:34:39:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2015-11-30 04:34:39:  Server Thread - OnConnect: Restarting Pulse Server Thread because it exited prematurely!! (System.Exception)

We are still on an older build, will update to the latest this week, but i could not find any related bugfix in the listed changes.

Thanks for reporting this, we’re currently looking into it.

Privacy | Site terms | Cookie preferences