Deadline 8 Proxy Server Issues

Deadline
1

Hi,

If anyone has any ideas to lend I’d be extremely grateful! I’ve not a whole lot of knowledge with regards to Deadline and have very little time to try and set up an external location where a VPN is not a possible solution.

We have Deadline 8 Repository set up on a Windows Server 2008 machine, running Thinkbox licensing and the Repository Database which is connecting to a shared RAID drive on a separate fileserver for the actual repository directory. We have about 15 local machines running with another 30 over a VPN connection and now trying to set up another 30 via a proxy. The Deadline repository version is 8.0.0.69 but all clients are now 8.0.6.5 or 8.0.8.1 (after getting the strange null reference proxy errors).

I’ve followed the guide from Thinkbox (deadline.thinkboxsoftware.com/fe … xy-panacea) as well as all the configuration documents but running into the error - “Proxy Server Error: The underlying connection was closed. An unexpected error occurred on a receive.” All Ports appear to be open correctly and working from the outside.

From the Deadline Monitor on the client side errors are showing in the console repeating:

2016-09-16 12:21:43:  ---------- Inner Stack Trace (System.Net.Sockets.SocketException) ----------
2016-09-16 12:21:43:     at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
2016-09-16 12:21:43:     at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
2016-09-16 12:21:45:  Error occurred while updating pulse cache: 
2016-09-16 12:21:45:  Proxy Server error: The underlying connection was closed: An unexpected error occurred on a receive. (System.Net.WebException)
2016-09-16 12:21:45:     at g.a(Exception A_0, NetworkManager A_1, String A_2, Int32 A_3, String A_4)
2016-09-16 12:21:45:     at Deadline.StorageDB.Proxy.ProxyPulseStorage.GetModifiedPulses(PulseInfoSettings[]& modifiedPulses, String[]& deletedPulseIds, Nullable`1 lastInfoAutoUpdate, Nullable`1 lastSettingsAutoUpdate, Nullable`1 lastDeletionAutoUpdate)
2016-09-16 12:21:45:     at Deadline.StorageDB.PulseStorage.a(Object o)
2016-09-16 12:21:45:  ---------- Inner Stack Trace (System.Net.Sockets.SocketException) ----------
2016-09-16 12:21:45:     at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
2016-09-16 12:21:45:     at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
2016-09-16 12:21:45:  Error occurred while updating licenseForwarder cache: 
2016-09-16 12:21:45:  Proxy Server error: The underlying connection was closed: An unexpected error occurred on a receive. (System.Net.WebException)
2016-09-16 12:21:45:     at g.a(Exception A_0, NetworkManager A_1, String A_2, Int32 A_3, String A_4)
2016-09-16 12:21:45:     at Deadline.StorageDB.Proxy.ProxyLicenseForwarderStorage.GetModifiedLicenseForwarders(LicenseForwarderInfoSettings[]& modifiedLicenseForwarders, String[]& deletedLicenseForwarderIds, Nullable`1 lastSettingsAutoUpdate, Nullable`1 lastInfoAutoUpdate, Nullable`1 lastDeletionAutoUpdate)
2016-09-16 12:21:45:     at Deadline.StorageDB.LicenseForwarderStorage.a(Object o)
2016-09-16 12:21:45:  ---------- Inner Stack Trace (System.Net.Sockets.SocketException) ----------
2016-09-16 12:21:45:     at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
2016-09-16 12:21:45:     at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
2016-09-16 12:21:46:  Error occurred while updating slave cache: The underlying connection was closed: An unexpected error occurred on a receive. (System.Net.WebException)

Thanks everyone!

2

We’ll need the Proxy log for this guy. It should show what the error is.

Also, it’s not great practice to have different versions across the farm. Try to upgrade when you can, especially the Repository as that’s safe and can optionally auto-upgrade anything that was installed after 8.0.6 (there was a bug in previous versions of the installer):
docs.thinkboxsoftware.com/produc … downgrades

After getting the log, I’ll check to see what’s up there. We did fix a problem recently in 8.0.8 with regard to the proxy and certain errors with delete jobs, so make sure the Proxy is on SP8.

3

Thanks for such a quick response eamsle, sorry about the delay, things have been fairly hectic. Stupid question though, where is the Proxy log? Server or client side? I just had a quick look around but can’t find anything in an obvious location sorry!

Thanks very much for the confirmation about the upgrades, it sounds pretty positive. I’ll certainly try to be running things this way from here on out after this push is over.

4

Ah, the proxy log is on the client side where the other app logs are. If the Proxy is running as a standard application, go to “Help”, the “Explore Log Folder”. Here’s the docs for finding the right file:

docs.thinkboxsoftware.com/produc … ation-logs

5

Thanks heaps for your help again Edwin. I no longer have access to the off site computers but have temporarily installed it locally on a Mac, where the same error does occur which I also tested the other day. This Mac is running the latest version of Deadline Client so as not to get the previous errors prevent start up. I’ve attached the logs collected from the Windows Server Proxy - running through the command line tool (which has been running for a few days now) and the attempted connection to the proxy server from the Mac. Hopefully these are the right ones, sorry if I’ve confused things!
Deadline-Logs.zip (4.24 KB)

6

Hey DB_

I am trying to re-create your setup so I can track down this problem.

Do you have an nginx server setup? If so can you send me your config file.

also are you able to ping the proxy machine from your client machine?

Thanks
Bhavek

7

I forgot to mention, can you run the poxyserver with the -verbose flag please.

8

Thanks for the assistance, it’s really appreciated! We don’t run an nginx server, the idea was to just get something barebones running.

From memory, after a call to our local Internet provider I found they intentionally block all ICMP requests, so I can’t ping or trace route but all TCP traffic was fine. Connecting over HTTP to the machine does successfully reach the server (as we have quite a few services running), in this case we are using the default port 8080 for the proxy which also successfully resolved.

I ran the proxyserver again with the -verbose and at the end attempted 4x connections from the local machine (on the internal network) using the same proxy repository to generate the same error on receive. Again, cheers for the help!
Deadline proxyserver Verbose.log.zip (1.62 KB)

9

Interesting. From the verbose log, It looks like you are getting traffic coming in to your machine.

What are the specs on your windows server? Also (this is a long shot) what timezone’s are you in? Are both the client and server installed in the same timezone?

10

Thanks agian, I’m sorry for the delay when I tried to get into the forums yesterday they were down. The stats for the Windows machine: Windows server 2008 (not R2) the system is a HP Proliant DL360 - a Xeon E5504 quad core, 12gb ram, with (if it might make a difference) a NIC - ‘HP NC382i DP Multifunction Gigabit Server Adapter’ only running on ipv4. One other thing though (again may be completely irrelevant), with the verbose output the client connections are 192.168 addresses which are not to correct internal/external addresses of course, is the proxy server assigning these incoming connections by default somehow? Is it possible to change these assignments around if it’s hitting a different network issue?

I’ve also checked the time zones, they are both identical - UTC+10 Brisbane, the time sync is actually up to a minute off from my client to the machine however, would this make all the difference? Anything is worth a shot :slight_smile:
Edit: Just to update this I did sync the times and the error is identical :frowning:

Thanks again!

11

Hey, sorry for the late reply on this, (I have been tinkering with this on and off)

the addresses you are getting from the proxy are being sent from the client. Try doing ipconfig (or ifconfig) on your client machine.

The specs on your windows server should be more than enough for the proxy. The only way I have managed to re-create your error is when I run the proxy in a very slow network. I used this tool jagt.github.io/clumsy/index.html to simulate a slow network. However I only got your error when I had a lag and an 80% drop rate.

Ill do some tests with your timezone to be sure.

12

If you want we can setup a call sometime next week for me to remote in to your machine and have a look.

13

Thank’s so much for taking a look into it Bhavek, It looks like you may be onto something here, I’ve found a few anomalies in Windows server 2008 with Internet connectivity across that port right now, no idea as yet what might be causing this. I will have to wait until I can take it offline for some further testing (hopefully tonight/tomorrow), however if you would be available next week for remote access that would be fantastic, feel free to contact me by email and we can set this up if you’re happy to give it a try. In the mean time I’ll keep you posted here on any updates.