I think I’m finally able to shed some light on an issue that has resurfaced regarding submitting jobs from Nuke and 3dsmax via the Deadline submission script. When we first upgraded to DL 4.1 SP1 last year, we ran into a weird issue where mainly the 3dsmax SMTD GUI would hang upon opening then break throwing many max script errors. After a lengthy diagnosis (three weeks) and help from (at the time Prime Focus) tech support, we resolved the issue simply by getting our Windows workstations up to par with DotNet Framwork. (We keep our workstations offline for the most part mainly for security, but this poses problems when it comes to updates). So after finally resolving the above things worked well, then we stopped using Deadline for about 6 months.
6 months later… now we’re trying to integrate Deadline back into our pipeline and once again its been a bit of a hassle. Jobs would submit ok initially, but would render throwing tons (thousands) of errors per each slave. I knew this time it wasn’t a Windows update or Dot Net issue as all of our Workstations had been updated. Diving deeper I came across one of our other servers running Pulse by accident (was testing a while back), but we already had Pulse running on a dedicated system and the repo was configured only for that system. So when I shut that additional (non-configured) Pulse slave down, things ran better. For the record, we killed the other Pulse slave as well for now, just to remove more variables from the equation- (we may re-enable it later if we need it).
The next item we figured out was that our systems were not all in sync with their system times. Again, since we’re primarily offline with our main network, time isn’t synchronizing to an outside time server (NTP). This continued to throw errors as a lot of the slaves were showing that they were stalled even though everything was fine. It also affected when the slaves picked up a new job in the queue in a timely fashion. To resolve this I decided to punch a hole in our gateway server, which allows very limited internet access to our workstations. Again, though we’re mainly offline with our workstations, providing very limited access for our online production tools helps a lot.
This takes me to the next item that I’ve had to recently diagnose, which has been the most obscure. With the above two issues knocked out of the way our initial rendering tests ran fine, however, the submission script nightmare had crept back into the equation. Before I knew it, everyone was having issues submitting through Nuke and 3dsmax again.
Since we have that gateway server providing limited internet access I had to allow DNS resolution so that the production group wouldn’t have to memorize IP’s of the various websites allowed in. To further complicate the situation, our X Server runs the DNS service with all of the local systems mapped accordingly. So the X Server DNS information along with the gateway system are all plugged into each workstations (not using DHCP). Long story short, I removed that gateway DNS IP’s (for now) from each workstation and also changed a setting on their network adapters. I had read a while back that non-DHCP networks should use the “Enable NetBIOS over TCP/IP” setting under the “Advanced TCP/IP Settings” window. Switching this back to Default (Use NetBios from the server) resolved the main issue and also allows us to see all of our systems on our network via the Explorer.
So you know, our set up is pretty straight forward:
Gigabit network with 24 & 48 port smart switches
X Server (10.5) and X Raid
Windows license server (optionally running Pulse)
Linux Deadline Repository (Suse 11)
Render Farm (50+)
Mac Pro workstations running Vista 64 (12+)