RCS Error 10060 , Uknown Error 16

Deadline
1

Hello there, facing a problem when trying to send large files (500mb-1.5Gb) over RCS .

Even though the workers are connected via RCS to the remote repository, in around 7-8 minutes after submission the following error pops and results to a job reset :frowning:

Any help would be appreciated.
Thank you ><

2022-06-28 23:38:41: 0: ERROR: UpdateClient.MaybeSendRequestNow caught an exception: POST http://-----@---------------:8080/rcs/v1/update returned “One or more errors occurred. (An error occurred while sending the request.)” (Deadline.Net.Clients.Http.DeadlineHttpRequestException)
2022-06-28 23:39:23: ERROR: GetRepositoryDateTime: Connection Server error: GET http://-----@---------------:8080/db/settings/serverdatetime returned “One or more errors occurred. (A task was canceled.)” (System.Net.WebException)
2022-06-28 23:41:35: ERROR: Info Thread - An error occurred while updating the Worker’s info: Connection Server error: POST http://-----@---------------:8080/db/slaves/info/save returned “One or more errors occurred. (An error occurred while sending the request.)” (System.Net.WebException)
2022-06-28 23:41:35: Exception Details
2022-06-28 23:41:35: SocketException – A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
2022-06-28 23:41:35: SocketException.SocketErrorCode: TimedOut (10060)
2022-06-28 23:41:35: SocketException.ErrorCode: 10060 (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2022-06-28 23:41:35: Win32Exception.NativeErrorCode: 10060
2022-06-28 23:41:35: Exception.Data: ( )
2022-06-28 23:41:35: Exception.HResult: -2147467259
2022-06-28 23:41:35: Exception.StackTrace:
2022-06-28 23:41:35: (null)
2022-06-28 23:41:35: IOException – Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond…
2022-06-28 23:41:35: Exception.TargetSite: Void ThrowException(System.Net.Sockets.SocketError, System.Threading.CancellationToken)
2022-06-28 23:41:35: Exception.Data: ( )
2022-06-28 23:41:35: Exception.Source: System.Net.Sockets
2022-06-28 23:41:35: Exception.HResult: -2146232800
2022-06-28 23:41:35: Exception.StackTrace:
2022-06-28 23:41:35: at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-06-28 23:41:35: at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
2022-06-28 23:41:35: at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
2022-06-28 23:41:35: HttpRequestException – An error occurred while sending the request.
2022-06-28 23:41:35: Exception.TargetSite: Void MoveNext()
2022-06-28 23:41:35: Exception.Data: ( )
2022-06-28 23:41:35: Exception.Source: System.Net.Http
2022-06-28 23:41:35: Exception.HResult: -2146232800
2022-06-28 23:41:35: Exception.StackTrace:
2022-06-28 23:41:35: at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
2022-06-28 23:41:35: at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
2022-06-28 23:41:35: at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
2022-06-28 23:41:35: at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
2022-06-28 23:41:35: at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
2022-06-28 23:41:35: at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) 2022-06-28 23:41:35: AggregateException -- One or more errors occurred. (An error occurred while sending the request.) 2022-06-28 23:41:35: AggregateException.InnerExceptions: ( System.Net.Http.HttpRequestException: An error occurred while sending the request. 2022-06-28 23:41:35: ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.. 2022-06-28 23:41:35: ---> System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-06-28 23:41:35: --- End of inner exception stack trace --- 2022-06-28 23:41:35: at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) 2022-06-28 23:41:35: at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) 2022-06-28 23:41:35: at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) 2022-06-28 23:41:35: --- End of inner exception stack trace --- 2022-06-28 23:41:35: at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) 2022-06-28 23:41:35: at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) 2022-06-28 23:41:35: at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) 2022-06-28 23:41:35: at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) 2022-06-28 23:41:35: at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) 2022-06-28 23:41:35: at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) )
2022-06-28 23:41:35: Exception.TargetSite: TResult GetResultCore(Boolean)
2022-06-28 23:41:35: Exception.Data: ( )
2022-06-28 23:41:35: Exception.Source: System.Private.CoreLib
2022-06-28 23:41:35: Exception.HResult: -2146233088
2022-06-28 23:41:35: Exception.StackTrace:
2022-06-28 23:41:35: at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) 2022-06-28 23:41:35: at Deadline.Net.Clients.Http.HttpClient.GetResponse(HttpRequestMessage request) 2022-06-28 23:41:35: DeadlineHttpRequestException -- POST http://-----@---------------:8080/db/slaves/info/save returned "One or more errors occurred. (An error occurred while sending the request.)" 2022-06-28 23:41:35: DeadlineHttpRequestException.StatusCode: (0) 2022-06-28 23:41:35: DeadlineHttpRequestException.RequestHttpMethod: POST 2022-06-28 23:41:35: DeadlineHttpRequestException.RequestURI: http://-----@---------------:8080/db/slaves/info/save 2022-06-28 23:41:35: Exception.TargetSite: System.Net.Http.HttpResponseMessage GetResponse(System.Net.Http.HttpRequestMessage) 2022-06-28 23:41:35: Exception.Data: ( ) 2022-06-28 23:41:35: Exception.Source: deadline 2022-06-28 23:41:35: Exception.HResult: -2146233088 2022-06-28 23:41:35: Exception.StackTrace: 2022-06-28 23:41:35: at Deadline.Net.Clients.Http.HttpClient.GetResponse(HttpRequestMessage request) 2022-06-28 23:41:35: at Deadline.Net.Clients.Http.HttpClient.b(HttpRequestMessage blp) 2022-06-28 23:41:35: at Deadline.Net.Clients.Http.HttpClient.SendRequestForStream(String method, String uri, String contentType, Dictionary2 headers, HttpContent httpContent)
2022-06-28 23:41:35: at Deadline.Net.Clients.Http.HttpClient.SendRequest(String method, String uri, String contentType, Dictionary`2 headers, HttpContent httpContent)
2022-06-28 23:41:35: at Deadline.StorageDB.Proxy.Utils.NetworkManager.a[b](RequestData sr)
2022-06-28 23:41:35: at Deadline.StorageDB.Proxy.Utils.NetworkManager.SendRequest[T](RequestData requestData)
2022-06-28 23:41:35: at Deadline.StorageDB.Proxy.ProxySlaveStorage.SaveSlaveInfos(SlaveInfo[] infos)
2022-06-28 23:41:35: WebException – Connection Server error: POST http://-----@---------------:8080/db/slaves/info/save returned “One or more errors occurred. (An error occurred while sending the request.)”

2

And in a further step, we raised Submission time-out limits to 10000 in the SMTD and also disabled Pre-cache Asset files on AWS . In that case the scene managed to load but still the following error pops:

2022-06-29 00:52:28: ERROR: UpdateClient.MaybeSendRequestNow caught an exception: POST http://-----@---------------:8080/rcs/v1/update returned “One or more errors occurred. (An error occurred while sending the request.)” (Deadline.Net.Clients.Http.DeadlineHttpRequestException)

Any help would be very appreciated, this is a new behaviour of deadline for us and it has created great problems for our workflow :frowning:

Few details about our setup, working with 3dsMax 2019 & Vray 5 , and using deadline 10.1.21.4 .

3

And similar error pops on Pulse :

UpdateClient.MaybeSendRequestNow caught an exception: POST http://-----@---------------:8080/rcs/v1/update returned “One or more errors occurred. (An error occurred while sending the request.)” (Deadline.Net.Clients.Http.DeadlineHttpRequestException)
2022-06-29 00:47:40: UpdateClient.MaybeSendRequestNow caught an exception: POST http://-----@---------------:8080/rcs/v1/update returned “One or more errors occurred. (An error occurred while sending the request.)” (Deadline.Net.Clients.Http.DeadlineHttpRequestException)
2022-06-29 00:48:08: Another repository repair process is already in progress

4

This is just a generic error that the server did not respond.
Where are the rcs logs? Could you show them?

5

Thanks for the response!

I would be happy to provide the logs from the RCS but I am not sure where to find them.
Do you know where they are stored perhaps?

Thank you : )

6

Oh I tracked it down, here it is a caption of the RCS log :frowning:

2022-06-29 12:19:32: BEGIN - **
2022-06-29 12:19:32: Operating System: Windows 10 Pro
2022-06-29 12:19:32: CPU Architecture: x64
2022-06-29 12:19:32: CPUs: 44
2022-06-29 12:19:32: Video Card: NVIDIA GeForce GTX 1080 Ti
2022-06-29 12:19:32: Deadline Remote Connection Server 10.1 [v10.1.21.4 Release (690edd3fe)]
2022-06-29 12:19:32: Connected to “\*"
2022-06-29 12:19:32: Turning off Kestrel logging
2022-06-29 12:19:32: DEBUG: The RCS is capable of sending responses compressed with :br
2022-06-29 12:19:33: Using ASP.Net Core server.
2022-06-29 12:19:33: Listening for http requests on 0.0.0.0 port 8080 loopbackOnly False…
2022-06-29 12:19:33: Counters: http-process-calls-current=0 http-process-calls-count=0 http-process-slow-calls-count=0 http-process-last-slow=none http-process-histo: ApxMean=NaN P99=NaN BoundsMs=0 Counts=0 assign-tasks-calls-current=0 assign-tasks-calls-count=0
2022-06-29 12:20:33: Update timeout has been set to 30 seconds
2022-06-29 12:20:33: Stdout Redirection Enabled: True
2022-06-29 12:20:33: Stdout Handling Enabled: False
2022-06-29 12:20:33: Popup Handling Enabled: False
2022-06-29 12:20:33: Using Process Tree: True
2022-06-29 12:20:33: Hiding DOS Window: True
2022-06-29 12:20:33: Creating New Console: False
2022-06-29 12:20:33: Running as user: *********
2022-06-29 12:20:33: Executable: “C:\Program Files\Thinkbox\Deadline10\bin\deadlinecommand.exe”
2022-06-29 12:20:33: Argument: -RunCommandForRepository “Repository” "\*” -DoHouseCleaning True False True
2022-06-29 12:20:33: Full Command: “C:\Program Files\Thinkbox\Deadline10\bin\deadlinecommand.exe” -RunCommandForRepository “Repository” “\*" -DoHouseCleaning True False True
2022-06-29 12:20:33: Startup Directory: “C:\Program Files\Thinkbox\Deadline10\bin”
2022-06-29 12:20:33: Process Priority: BelowNormal
2022-06-29 12:20:33: Process Affinity: default
2022-06-29 12:20:33: Process is now running
2022-06-29 12:20:34: Counters: http-process-calls-current=0 http-process-calls-count=0 http-process-slow-calls-count=0 http-process-last-slow=none http-process-histo: ApxMean=NaN P99=NaN BoundsMs=0 Counts=0 assign-tasks-calls-current=0 assign-tasks-calls-count=0
2022-06-29 12:20:36: Performing house cleaning
2022-06-29 12:20:36: Performing Job Cleanup Scan…
2022-06-29 12:20:36: Job Cleanup Scan - Loading completed jobs
2022-06-29 12:20:36: Job Cleanup Scan - Loaded 7 completed and 1 active/pending jobs in 103.800 ms
2022-06-29 12:20:36: Job Cleanup Scan - Scanning completed jobs
2022-06-29 12:20:36: Job Cleanup Scan - Deleted 0 and archived 0 completed jobs in 2.866 ms
2022-06-29 12:20:36: Job Cleanup Scan - Done.
2022-06-29 12:20:36: Purging Unsubmitted Jobs
2022-06-29 12:20:36: Unsubmitted Job Scan - Loading unsubmitted jobs
2022-06-29 12:20:36: Unsubmitted Job Scan - Loaded 0 unsubmitted jobs in 1.158 ms
2022-06-29 12:20:36: Unsubmitted Job Scan - Done.
2022-06-29 12:20:36: Purging Deleted Jobs
2022-06-29 12:20:36: Deleted Job Scan - Loading deleted jobs
2022-06-29 12:20:36: Deleted Job Scan - Loaded 3 deleted jobs in 5.152 ms
2022-06-29 12:20:36: Deleted Job Scan - Purged 0 deleted jobs in 7.200 µs
2022-06-29 12:20:36: Deleted Job Scan - Done.
2022-06-29 12:20:36: Purging JobSubmissions
2022-06-29 12:20:36: Deleting activated and cancelled JobSubmissions.
2022-06-29 12:20:36: Deleting JobSubmissions and jobs stuck in ready for more than 3 days.
2022-06-29 12:20:36: JobSubmissions Purge - Done.
2022-06-29 12:20:36: Purging Obsolete Workers
2022-06-29 12:20:36: Obsolete Worker Scan - Skipping because it is disabled in the Repository Options
2022-06-29 12:20:36: Purging Obsolete AWS Portal Workers
2022-06-29 12:20:36: Obsolete AWS Portal Worker Scan - Loading Worker states
2022-06-29 12:20:36: Obsolete AWS Portal Worker Scan - Loading 18 Worker states in 26.753 ms
2022-06-29 12:20:36: Obsolete AWS Portal Worker Scan - Purging offline/stalled Workers older than 5 days, 0 hours
2022-06-29 12:20:36: Obsolete AWS Portal Worker Scan - Purged 0 obsolete AWS Portal Workers in 50.900 µs
2022-06-29 12:20:36: Obsolete AWS Portal Worker Scan - Done.
2022-06-29 12:20:36: Purging Old Worker Reports
2022-06-29 12:20:36: Worker Report Scan - Loading Worker report collections
2022-06-29 12:20:36: Worker Report Scan - Found 17 report collections and 9 job state objects in 10.764 ms
2022-06-29 12:20:36: Worker Report Scan - Loading Worker IDs
2022-06-29 12:20:36: Worker Report Scan - Loaded 18 Worker IDs in 2.367 ms
2022-06-29 12:20:36: Worker Report Scan - Purged 0 report collections, 0 job state objects, and 0 licensed Worker stubs in 0
2022-06-29 12:20:36: Worker Report Scan - Done.
2022-06-29 12:20:36: Purging Timed Out Workers in Throttle Queue
2022-06-29 12:20:36: Purging Old Statistics
2022-06-29 12:20:36: Old Statistics - Skipping job statistics because the option to purge them is disabled in the Repository Options
2022-06-29 12:20:36: Old Statistics - Purging Worker statistics that are older than Mar 01/22 12:20:32
2022-06-29 12:20:36: Old Statistics - Purged old Worker statistics in 2.103 ms
2022-06-29 12:20:36: Old Statistics - Purging repository statistics that are older than Mar 01/22 12:20:32
2022-06-29 12:20:36: Old Statistics - Purged old repository statistics in 742.200 µs
2022-06-29 12:20:36: Purging expired Remote Command Tokens
2022-06-29 12:20:36: Purging Deleted Document Stubs From Database
2022-06-29 12:20:36: Deleted Document Stubs - Deleting stubs that are older than 3 days
2022-06-29 12:20:36: Deleted Document Stubs - Deleted 0 stubs in 1.040 ms
2022-06-29 12:20:36: Triggering house cleaning events…
2022-06-29 12:20:36: Process exit code: 0
2022-06-29 12:20:36: Update timeout has been set to 30 seconds
2022-06-29 12:20:36: Stdout Redirection Enabled: True
2022-06-29 12:20:36: Stdout Handling Enabled: False
2022-06-29 12:20:36: Popup Handling Enabled: False
2022-06-29 12:20:36: Using Process Tree: True
2022-06-29 12:20:36: Hiding DOS Window: True
2022-06-29 12:20:36: Creating New Console: False
2022-06-29 12:20:36: Running as user: *********
2022-06-29 12:20:36: Executable: “C:\Program Files\Thinkbox\Deadline10\bin\deadlinecommand.exe”
2022-06-29 12:20:36: Argument: -RunCommandForRepository “Repository” "\*” -DoRepositoryRepair True False True
2022-06-29 12:20:36: Full Command: “C:\Program Files\Thinkbox\Deadline10\bin\deadlinecommand.exe” -RunCommandForRepository “Repository” "\*" -DoRepositoryRepair True False True
2022-06-29 12:20:36: Startup Directory: “C:\Program Files\Thinkbox\Deadline10\bin”
2022-06-29 12:20:36: Process Priority: BelowNormal
2022-06-29 12:20:36: Process Affinity: default
2022-06-29 12:20:36: Process is now running
2022-06-29 12:20:39: Performing repository repair
2022-06-29 12:20:39: Performing Orphaned Task Scan…
2022-06-29 12:20:39: Orphaned Task Scan - Loading rendering jobs
2022-06-29 12:20:39: Orphaned Task Scan - Loaded 1 rendering jobs in 78.619 ms
2022-06-29 12:20:39: Orphaned Task Scan - Scanning for orphaned tasks
2022-06-29 12:20:39: Orphaned Task Scan - Cleaned up 0 orphaned tasks in 54.681 ms
2022-06-29 12:20:39: Orphaned Task Scan - Done.
2022-06-29 12:20:39: Performing Orphaned Limit Stub Scan…
2022-06-29 12:20:39: Orphaned Limit Stub Scan - Loading limits
2022-06-29 12:20:39: Orphaned Limit Stub Scan - Loaded 9 limits in 9.116 ms
2022-06-29 12:20:39: Orphaned Limit Stub Scan - Loading Worker states
2022-06-29 12:20:39: Orphaned Limit Stub Scan - Loaded 18 Worker states in 2.037 ms
2022-06-29 12:20:39: Orphaned Limit Stub Scan - Returned 0 orphaned limit stubs in 8.961 ms
2022-06-29 12:20:39: Orphaned Limit Stub Scan - Done.
2022-06-29 12:20:39: Checking Available Database Connections
2022-06-29 12:20:39: Available Database Connections - Skipping because there are no Low Database Connection notification email addresses set in the Repository Options
2022-06-29 12:20:39: Performing Stalled Worker Scan…
2022-06-29 12:20:39: Stalled Worker Scan - Loading Worker states
2022-06-29 12:20:39: Stalled Worker Scan - Loaded 18 Worker states in 2.139 ms
2022-06-29 12:20:39: Stalled Worker Scan - Scanning Worker states
2022-06-29 12:20:39: Stalled Worker Scan - Cleaned up 0 stalled Worker in 4.856 ms
2022-06-29 12:20:39: Stalled Worker Scan - Done.
2022-06-29 12:20:39: Performing Stalled Pulse Scan…
2022-06-29 12:20:39: Stalled Pulse Scan - Loading pulse states
2022-06-29 12:20:39: Stalled Pulse Scan - Loaded 1 pulse states in 4.913 ms
2022-06-29 12:20:39: Stalled Pulse Scan - Scanning pulse states
2022-06-29 12:20:39: Stalled Pulse Scan - Cleaned up 0 stalled pulses in 1.188 ms
2022-06-29 12:20:39: Stalled Pulse Scan - Done.
2022-06-29 12:20:39: Performing Stalled Balancer Scan…
2022-06-29 12:20:39: Stalled Balancer Scan - Loading balancer states
2022-06-29 12:20:39: Stalled Balancer Scan - Loaded 0 balancer states in 2.449 ms
2022-06-29 12:20:39: Stalled Balancer Scan - Done.
2022-06-29 12:20:39: Performing Stalled License Forwarder Scan…
2022-06-29 12:20:39: Stalled License Forwarder Scan - Loading forwarder states
2022-06-29 12:20:39: Stalled License Forwarder Scan - Loaded 0 forwarder states in 2.293 ms
2022-06-29 12:20:39: Stalled License Forwarder Scan - Done.
2022-06-29 12:20:39: Performing Stalled Connection Server Scan…
2022-06-29 12:20:39: Stalled Connection Server Scan - Loading Connection Server states
2022-06-29 12:20:39: Stalled Connection Server Scan - Loaded 1 Connection Server states in 4.530 ms
2022-06-29 12:20:39: Stalled Connection Server Scan - Scanning Connection Server states
2022-06-29 12:20:39: Stalled Connection Server Scan - Cleaned up 0 stalled Connection Servers in 939.500 µs
2022-06-29 12:20:39: Stalled Connection Server Scan - Done.
2022-06-29 12:20:39: Triggering repository repair events…
2022-06-29 12:20:40: Process exit code: 0
2022-06-29 12:20:40: Update timeout has been set to 30 seconds
2022-06-29 12:20:40: Stdout Redirection Enabled: True
2022-06-29 12:20:40: Stdout Handling Enabled: False
2022-06-29 12:20:40: Popup Handling Enabled: False
2022-06-29 12:20:40: Using Process Tree: True
2022-06-29 12:20:40: Hiding DOS Window: True
2022-06-29 12:20:40: Creating New Console: False
2022-06-29 12:20:40: Running as user:*******
2022-06-29 12:20:40: Executable: “C:\Program Files\Thinkbox\Deadline10\bin\deadlinecommand.exe”

7

No errors here… so it should be a network error then - firewall, wrong port, etc.