AWS Thinkbox Discussion Forums

pending job scan hung up

Deprecated Deadline 7 Bug Reports
#1

Our pending job scan somehow hung up around 2am, and pulse was unable to kill it or start it since:

2015-09-23 02:51:45: Attempting a hard kill of parent process with id 28801 because it failed to exit.
2015-09-23 02:51:46: Attempting a hard kill of parent process with id 28801 because it failed to exit.
2015-09-23 02:51:46: Error running pending job scan process: Process timed out after 1800 seconds without an update. (FranticX.Processes.ManagedProcessAbort)

2015-09-23 03:22:54: Attempting a hard kill of parent process with id 5291 because it failed to exit.
2015-09-23 03:22:54: Error running pending job scan process: Process timed out after 1800 seconds without an update. (FranticX.Processes.ManagedProcessAbort)

etc
etc

We have a cron job that attempts to start pulse via the launcher in case of crashes every 4 minutes. This seems to also cause issues, as the process list looks like this:

root     22609  0.0  0.0 270208 17640 ?        Dl   05:08   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     23921  0.0  0.0 106096  1148 ?        Ss   07:52   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     23922  0.0  0.0 270140 17632 ?        Dl   07:52   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     24048  0.0  0.0 106096  1152 ?        Ss   04:16   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     24049  0.0  0.0 270176 17640 ?        Dl   04:16   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     24113  0.0  0.0 106096  1148 ?        Ss   08:48   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     24114  0.0  0.0 270208 17632 ?        Dl   08:48   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     24307  0.0  0.0 106096  1144 ?        Ss   06:04   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     24308  0.0  0.0 270208 17636 ?        Dl   06:04   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     24697  0.0  0.0 106096  1148 ?        Ss   07:00   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     24698  0.0  0.0 270208 17636 ?        Dl   07:00   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     25454  0.0  0.0 106096  1148 ?        Ss   03:24   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     25455  0.0  0.0 270208 17632 ?        Dl   03:24   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     25681  0.0  0.0 106096  1148 ?        Ss   07:56   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     25682  0.0  0.0 270208 17640 ?        Dl   07:56   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     25705  0.0  0.0 106096  1148 ?        Ss   05:12   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     25706  0.0  0.0 270208 17636 ?        Dl   05:12   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     26088  0.0  0.0 106096  1148 ?        Ss   06:08   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     26089  0.0  0.0 270208 17636 ?        Dl   06:08   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     26200  0.0  0.0 106096  1152 ?        Ss   04:20   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     26201  0.0  0.0 270140 17632 ?        Dl   04:20   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     27095  0.0  0.0 106096  1152 ?        Ss   08:52   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     27096  0.0  0.0 270176 17636 ?        Dl   08:52   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     27203  0.0  0.0 106096  1152 ?        Ss   03:28   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     27204  0.0  0.0 270208 17640 ?        Dl   03:28   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     27562  0.0  0.0 106096  1148 ?        Ss   05:16   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     27563  0.0  0.0 270208 17640 ?        Dl   05:16   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     27769  0.0  0.0 106096  1148 ?        Ss   07:04   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     27770  0.0  0.0 270208 17640 ?        Dl   07:04   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     27837  0.0  0.0 106096  1152 ?        Ss   08:00   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     27838  0.0  0.0 270208 17636 ?        Dl   08:00   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     28903  0.0  0.0 106096  1148 ?        Ss   08:56   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     28904  0.0  0.0 270208 17640 ?        Dl   08:56   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     29124  0.0  0.0 106096  1152 ?        Ss   06:12   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     29125  0.0  0.0 270208 17636 ?        Dl   06:12   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     29161  0.0  0.0 106096  1148 ?        Ss   04:24   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     29162  0.0  0.0 270208 17636 ?        Dl   04:24   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     29506  0.0  0.0 106096  1148 ?        Ss   07:08   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     29507  0.0  0.0 270208 17640 ?        Dl   07:08   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     29669  0.0  0.0 106096  1148 ?        Ss   05:20   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     29670  0.0  0.0 270176 17640 ?        Dl   05:20   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     30101 30.8  3.9 5682084 1298864 ?     Sl   Sep11 5310:05 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinepulse.exe -nogui
root     30371  0.0  0.0 106096  1152 ?        Ss   03:32   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     30372  0.0  0.0 270208 17636 ?        Dl   03:32   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     30976  0.0  0.0 106096  1144 ?        Ss   06:16   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     30977  0.0  0.0 270176 17640 ?        Dl   06:16   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     30992  0.0  0.0 106096  1152 ?        Ss   08:04   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     30993  0.0  0.0 270208 17636 ?        Dl   08:04   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     31004  0.0  0.0 106096  1152 ?        Ss   04:28   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     31005  0.0  0.0 270208 17640 ?        Dl   04:28   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     31053  0.0  0.0 106096  1152 ?        Ss   09:00   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     31054  0.0  0.0 270176 17644 ?        Dl   09:00   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     31259  0.0  0.0 106096  1152 ?        Ss   02:32   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     31260  0.0  0.0 270208 17636 ?        Dl   02:32   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     32050  0.0  0.0 106096  1152 ?        Ss   03:36   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     32051  0.0  0.0 270208 17636 ?        Dl   03:36   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     32122  0.0  0.0 106096  1148 ?        Ss   02:36   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     32123  0.0  0.0 270208 17640 ?        Dl   02:36   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     32631  0.0  0.0 106096  1148 ?        Ss   07:12   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     32632  0.0  0.0 270208 17632 ?        Dl   07:12   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     32680  0.0  0.0 106096  1152 ?        Ss   05:24   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     32681  0.0  0.0 270140 17636 ?        Dl   05:24   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui
root     32724  0.0  0.0 106096  1148 ?        Ss   08:08   0:00 /bin/sh -c /opt/Thinkbox/Deadline7/bin/deadlinelauncher -pulse -nogui >/dev/null 2>&1
root     32725  0.0  0.0 270208 17636 ?        Dl   08:08   0:00 ./mono --runtime=v4.0 /opt/Thinkbox/Deadline7/bin/deadlinelauncher.exe -pulse -nogui

The last entries in the launcher log:

2015-09-23 02:24:01:  Launcher Thread - Received command: LaunchPulsePromptForUpgrade
2015-09-23 02:26:56:  Comparing version files...
2015-09-23 02:28:14:  ::ffff:172.18.1.107 has connected
2015-09-23 02:28:14:  Launcher Thread - Received command: LaunchPulsePromptForUpgrade
2015-09-23 02:29:57:  Local version file: /opt/Thinkbox/Deadline7/bin/Version
2015-09-23 02:29:57:  Network version file: /mnt/isila/deadline/repository7/bin/Linux/Version
2015-09-23 02:29:58:  Comparing version files...
#2

We are still investigating, but only a reboot of the server recovered pulse completely. I could not kill the hung up pulse / launcher processes in any way.

#3

Hey Laszlo,

This sounds familiar to this issue:
forums.thinkboxsoftware.com/vie … 11&t=13722

The workaround that we’re implementing for 7.2 is to only make 3 attempts to kill the process and then move on so that Pulse isn’t blocked forever.

Cheers,
Ryan

#4

Yeah sounds very similar. It would be great if some of these error statuses were propagated to the monitor. When we look at the pulse tab, everything looks dandy. It would be AWESOME if it showed the:

  • last successful pending job scan time
  • last successful repo repair time
  • last successful housecleaning time

If any of those has been longer than N minutes, the pulse machine could turn BOLD and RED, start blinking and send email notifications to people of interest :slight_smile:

#5

In the Monitor, you can choose to display this information in the status bar:

monitor_last_scans_in_status_bar.png
monitor_last_scans_in_status_bar.png813×545 34.8 KB

The settings for enabling these are in the Monitor Options (under Miscellaneous):
docs.thinkboxsoftware.com/produc … tings.html

Cheers,
Ryan

#6

Woa, thanks for the pointer!

Privacy | Site terms | Cookie preferences