AWS Thinkbox Discussion Forums

stalled slaves

Deprecated Deadline 6 Bug Reports
#1

Found a couple of slaves that have been stalled for days this morning.

Log #1 (last lines):
2013-02-16 08:52:53: BEGIN - LAPRO0324\ScanlineVFX
2013-02-16 08:52:53: 0: Slave timed out – canceling current task…

The machine’s slave is hanging (slave window is up, but its contents are just a white screen). In the monitor its shown as if it was still rendering the frame (2days 20hours and counting). The slave process is using 100% of one cpu core, not sure what its doing.

There are 3 other slaves in the list that report having been working for 3.7 days, but the jobs they claim to be working on don’t even exist any longer (they were deleted a while back). Uploaded a screenshot of that. Remoting into these machines shows the slaves to be in a weird state, using 100% of one of the cpu cores, again last piece of log info is:

2013-02-15 11:48:41: BEGIN - LAPRO0315\ScanlineVFX
2013-02-15 11:48:41: 0: Slave timed out – canceling current task…

Jeff Harkness also noticed that the slave list tends to show false information from time to time, like showing slaves working on your jobs, when they are not actually active (or been suspended, completed), probably the same issue?

Essentially, the slaves are hanging, but it goes unnoticed. Does the local deadline launcher do some slave status reporting? Or its all up to the slave process?

cheers,
laszlo

monitor_false_info.jpg
monitor_false_info.jpg1023×363 65.6 KB

#2

Note that the origin of the slave’s hanging is most likely the beta11->beta12 update.

The last active bits of log information from the slaves look like this:

2013-02-14 15:41:40: 0: STDOUT: mel: READY FOR INPUT
2013-02-14 15:41:40: Error occurred while writing report log:
2013-02-14 15:41:40: Exception Details
2013-02-14 15:41:40: DirectoryNotFoundException – Could not find a part of the path ‘\inferno2\deadline\repository6\reports\jobs\27\c\511d76346a982e0a644c027c.bz2’.
2013-02-14 15:41:40: Exception.Data: ( )
2013-02-14 15:41:40: Exception.TargetSite: Void WinIOError(Int32, System.String)
2013-02-14 15:41:40: Exception.Source: mscorlib
2013-02-14 15:41:40: Exception.StackTrace:
2013-02-14 15:41:40: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
2013-02-14 15:41:40: at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite)
2013-02-14 15:41:40: at Deadline.StorageDB.JobStorage.WriteJobReportFile(Report report, String reportLog)
2013-02-14 15:41:45: 0: INFO: Ending Maya Job
2013-02-14 15:41:45: 0: INFO: Waiting for Maya to shut down
2013-02-14 15:41:45: 0: INFO: Maya has shut down
2013-02-14 15:42:17: Scheduler Thread - Synchronizing job files
2013-02-14 15:42:17: Scheduler Thread - Synchronization time for job files: 0.000 s
2013-02-14 15:42:17: Scheduler Thread - Synchronizing plugin files
2013-02-14 15:42:17: Scheduler Thread - Synchronization time for plugin files: 171.875 ms
2013-02-14 15:42:18: Constructor: MayaBatch
2013-02-14 15:42:18: 0: Task timeout is being ignored because this is a Post Job Script Task
2013-02-14 15:42:18: 0: Loaded job: [TEST] Software Render: TST_001_0001_maya_animation_mayaRigCache.ma version: v0060 (511d5be6969b591d3856186a)
2013-02-14 15:42:18: 0: Plugin executing post job script
2013-02-14 15:42:18: 0: Executing Post Job Script: “//inferno2/projects/test/work/jha/testl.py”
2013-02-14 15:42:20: Exception occurred when setting object value: Object reference not set to an instance of an object. (System.NullReferenceException)
2013-02-14 15:42:20: 0: INFO: Executing plugin script C:\Documents and Settings\ScanlineVFX\Local Settings\Application Data\Thinkbox\Deadline6\slave\LAPRO0325\plugins\MayaBatch.py
2013-02-14 15:42:20: 0: INFO: About: Maya Batch Plugin for Deadline
2013-02-14 15:42:20: 0: INFO: The job’s environment will be merged with the current environment before rendering
2013-02-14 15:42:20: 0: INFO: Executing post job script //inferno2/projects/test/work/jha/testl.py
2013-02-14 15:42:20: Error occurred while writing report log:
2013-02-14 15:42:20: Exception Details
2013-02-14 15:42:20: DirectoryNotFoundException – Could not find a part of the path ‘\inferno2\deadline\repository6\reports\jobs\27\d\511d765c6a982e0a644c027d.bz2’.
2013-02-14 15:42:20: Exception.Data: ( )
2013-02-14 15:42:20: Exception.TargetSite: Void WinIOError(Int32, System.String)
2013-02-14 15:42:20: Exception.Source: mscorlib
2013-02-14 15:42:20: Exception.StackTrace:
2013-02-14 15:42:20: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
2013-02-14 15:42:20: at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite)
2013-02-14 15:42:20: at Deadline.StorageDB.JobStorage.WriteJobReportFile(Report report, String reportLog)
2013-02-14 15:44:55: Scheduler Thread - Synchronizing job files
2013-02-14 15:44:55: Scheduler Thread - Synchronization time for job files: 0.000 s
2013-02-14 15:44:55: Scheduler Thread - Synchronizing plugin files
2013-02-14 15:44:55: Scheduler Thread - Synchronization time for plugin files: 93.750 ms
2013-02-14 15:44:56: Constructor: MayaBatch
2013-02-14 15:44:56: 0: Task timeout is 72000 seconds (Regular Task Timeout)
2013-02-14 15:44:56: 0: Loaded job: [TEST] Animation Publish: TST_000_0000_maya_animation_deadlineCachingTest.ma version: v0004 (511d71cfb4e5bb08b04b2a95)
2013-02-14 15:45:12: Scheduler Thread - Cancelling task because task “0_0-0” could not be found

Then a while later you get the:

2013-02-15 11:44:56: BEGIN - LAPRO0325\ScanlineVFX
2013-02-15 11:44:56: 0: Slave timed out – canceling current task…

And then nothing.

#3

The issue appears to be that the slave’s Render thread gets hung up, but the slave’s Info thread (which updates the slave’s state in the database) is still running fine. Stalled slave detection works by determining that the slave’s state hasn’t been updated for a while, but if the slave is still updating its state, everything looks fine.

Thanks for the logs. That should help us narrow down this problem. Just to confirm, is there anything of use between these sections in the log?

Or is there literally nothing between them?

Thanks!

  • Ryan
#4

Literally nothing… in fact they are separate files, as they are from separate days. So the first bit are the last lines from the previous file, and the next two lines are all thats in the new log file.

#5

Thanks for confirming!

#6

We had ~8 slaves overnight that did this again (same behavior)

#7

Can you post their logs too? We’re still working on tracking this down, and the more logs, the better.

Thanks!

#8

Yeah this happens every night, we lose a couple of slaves.

Some new logs:

Last lines of log that were active (these are just outputs from maya):

2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : toplevel widget is QDialog
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.properties : DEBUG : setting property geometry
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.properties : DEBUG : setting property sizePolicy
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.properties : DEBUG : setting property minimumSize
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.properties : DEBUG : setting property maximumSize
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.properties : DEBUG : setting property windowTitle
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.properties : DEBUG : setting property modal
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : push QDialog Dialog
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : push QVBoxLayout verticalLayout_3
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.properties : DEBUG : setting property text
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : push QPushButton m_uUpdateDB
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : pop widget QPushButton m_uUpdateDB
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : new topwidget Dialog
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : push QHBoxLayout horizontalLayout_3
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.properties : DEBUG : setting property sizePolicy
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.properties : DEBUG : setting property text
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : push QLabel label_2
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : pop widget QLabel label_2
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : new topwidget Dialog
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : push QComboBox m_uRigs
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : pop widget QComboBox m_uRigs
2013-02-19 21:03:37: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : new topwidget Dialog

There was also a task_2-LAPRO0344-0000.log with the same contents.

Another slave, white slave app, reporting rendering, but is hanging. Log last lines (same as the task_22-LAPRO0324-0000.log):

2013-02-19 20:41:27: 0: STDOUT: Tue Feb 19 20:41:26 2013 (+0ms) : PUBLISH : ScanlineResourceList @ vehPersianShipA_40 - ID: 12028
2013-02-19 20:41:27: 0: STDOUT: Tue Feb 19 20:41:26 2013 (+0ms) : PUBLISH : RigCacheExporter @ vehPersianShipA_4 - ID: 11913
2013-02-19 20:41:27: 0: STDOUT: Tue Feb 19 20:41:26 2013 (+16ms) : PUBLISH : MayaRigCacheExporter @ vehPersianShipA_4 - ID: 18767
2013-02-19 20:41:27: 0: STDOUT: Tue Feb 19 20:41:26 2013 (+0ms) : PUBLISH : ScanlineResourceList @ vehPersianShipA_4 - ID: 11912
2013-02-19 20:41:27: 0: STDOUT: Tue Feb 19 20:41:26 2013 (+0ms) : PUBLISH : RigCacheExporter @ vehPersianShipA_399 - ID: 56282
2013-02-19 20:41:27: 0: STDOUT: Tue Feb 19 20:41:27 2013 (+15ms) : PUBLISH : MayaRigCacheExporter @ vehPersianShipA_399 - ID: 56283
2013-02-19 20:41:27: 0: STDOUT: Tue Feb 19 20:41:27 2013 (+0ms) : PUBLISH : ScanlineResourceList @ vehPersianShipA_399 - ID: 56281

No other log, its just hanging with a white slave screen, monitor reports slave as active.

On another machine, the slave was completely crashed, and it reported stalled in the monitor. Last logs:
2013-02-19 19:25:28: 0: STDOUT: Constructing shading groups
2013-02-19 19:25:28: 0: STDOUT: Rendering current frame.
2013-02-19 19:25:41: 0: STDOUT: Frame triangle count: 1101492
2013-02-19 19:25:41: 0: STDOUT: ====================================
2013-02-19 19:25:41: 0: STDOUT: Resource Usage At End Of Frame
2013-02-19 19:25:41: 0: STDOUT: ====================================
2013-02-19 19:25:41: 0: STDOUT: 455470 Page faults
2013-02-19 19:25:41: 0: STDOUT: 4687.313 Mb Peak total size(Estimated)
2013-02-19 19:25:41: 0: STDOUT: 4066.037 Mb Peak arena size
2013-02-19 19:25:41: 0: STDOUT: ====================================
2013-02-19 19:25:41: 0: STDOUT: 1716.418 Mb Current
2013-02-19 19:25:41: 0: STDOUT: 24.126 Mb Transforms
2013-02-19 19:25:41: 0: STDOUT: 820.445 Mb Arrays
2013-02-19 19:25:41: 0: STDOUT: 11.750 Mb NURBS AG
2013-02-19 19:25:41: 0: STDOUT: 56.768 Mb Data Blocks
2013-02-19 19:25:41: 0: STDOUT: 32.656 Mb MEL
2013-02-19 19:25:41: 0: STDOUT: 7.248 Mb Keys
2013-02-19 19:25:41: 0: STDOUT: 24.500 Mb Render Cache
2013-02-19 19:25:41: 0: STDOUT: 0.076 Mb Render Geometry Arena
2013-02-19 19:25:41: 0: STDOUT: 0.002 Mb arguments
2013-02-19 19:25:41: 0: STDOUT: 96.110 Mb File Texture Mipmaps
2013-02-19 19:25:41: 0: STDOUT: 0.129 Mb Object Arrays
2013-02-19 19:25:41: 0: STDOUT: ====================================
2013-02-19 19:25:41: 0: STDOUT: Postprocessing rendering result.
2013-02-19 19:25:43: 0: STDOUT: Time For Tessellation (hh:mm:ss): 00:00:01
2013-02-19 19:25:43: 0: STDOUT: Time For Shadow Map (hh:mm:ss): 00:00:00
2013-02-19 19:25:43: 0: STDOUT: Time For Post Process (hh:mm:ss): 00:00:01
2013-02-19 19:25:43: 0: STDOUT: Time For Frame Render (hh:mm:ss): 00:00:15
2013-02-19 19:25:43: 0: STDOUT: Finished Rendering //inferno2/projects/tboa/scenes/FB_082_0300/images/render3d/Anim-anim-oars-testPreview/v0057_epe_oar_test/softwareJPG_sRGB_2khd/FB_082_0300_Anim-anim-oars-testPreview_softwareJPG_v0057.1056.jpg.
2013-02-19 19:25:43: 0: STDOUT: ------------------------------------------------------------

#9

Out of curiosity, i attached a debugger to one of these deadlineslave apps that were all white, with 100% cpu usage.

The main thread is being very active in python:

python26.dll!000000001e039e70()
[Frames below may be incorrect and/or missing, no symbols loaded for python26.dll]
python26.dll!000000001e03a734()
python26.dll!000000001e03b12f()
python26.dll!000000001e0c0ee2()
python26.dll!000000001e0c70f4()
python26.dll!000000001e07a0b5()
python26.dll!000000001e0f6c9b()
python26.dll!000000001e0f754a()
python26.dll!000000001e0fa85b()
python26.dll!000000001e0f5b48()
python26.dll!000000001e0f7534()
python26.dll!000000001e0fa85b()
python26.dll!000000001e0f5b48()
python26.dll!000000001e0f7534()
python26.dll!000000001e0fa85b()
python26.dll!000000001e0fbea7()
python26.dll!000000001e0a9933()
python26.dll!000000001e07a0b5()
python26.dll!000000001e091eb1()
python26.dll!000000001e07a0b5()
python26.dll!000000001e0f5948()
sip.pyd!000000001bf3c4fc()
QtCore.pyd!000000001cf285c5()
QtCore.pyd!000000001cf287ea()
QtCore.pyd!000000001cf2900e()
QtCore4.dll!000000006711e793()
QtCore4.dll!0000000067123b52()
QtCore.pyd!000000001ce3e39a()
QtCore4.dll!000000006711d15c()
QtCore.pyd!000000001ce3e41a()
QtGui4.dll!0000000065022786()
QtGui4.dll!00000000650250f8()
QtGui.pyd!000000001ca958b8()
QtCore4.dll!000000006710a3f2()
QtCore4.dll!000000006713066e()
QtCore4.dll!0000000067130fc6()
user32.dll!0000000077c43abc()
user32.dll!0000000077c43f5c()
QtCore4.dll!0000000067132d8a()
QtGui4.dll!0000000065083705()
QtCore4.dll!00000000671097d1()
QtCore4.dll!000000006710e190()
QtGui.pyd!000000001ca99197()
python26.dll!000000001e0b59b9()
python26.dll!000000001e0f74a4()
python26.dll!000000001e0fa85b()
python26.dll!000000001e0f5b48()
python26.dll!000000001e0f7534()
python26.dll!000000001e0fa85b()
python26.dll!000000001e0f5b48()
python26.dll!000000001e0f7534()
python26.dll!000000001e0fa85b()
python26.dll!000000001e0fbea7()
python26.dll!000000001e0fbf79()
python26.dll!000000001e10a8e0()
clr.dll!000006447f1017c7()
00000644801eab33()

All threads:

0 > 6300 Main Thread Main Thread 000000001e039e70 Normal 0
0 1848 Worker Thread Win64 Thread 0000000077ef0f9a Normal 0
0 1764 Worker Thread Win64 Thread 0000000077ef0f9a Highest 0
0 8904 Worker Thread Win64 Thread 0000000077ef0c9a Normal 0
0 8692 RPC Thread RPC Callback Thread 0000000077ef0d2a Normal 0
0 6944 Worker Thread Win64 Thread 0000000077ef0d2a Normal 0
0 3632 Worker Thread Win64 Thread 0000000077ef0f9a Normal 0
0 5832 Worker Thread Win64 Thread 0000000077ef0f9a Normal 0
0 5516 Worker Thread Win64 Thread 0000000077ef0c9a Normal 0
0 2552 Worker Thread Win64 Thread 0000000077ef0f9a Normal 0
0 1220 Worker Thread Win64 Thread 0000000077ef0a2a Normal 0
0 5376 Worker Thread Win64 Thread 0000000077ef0d2a Above Normal 0
0 6688 Worker Thread Win64 Thread 0000000077ef0d2a Normal 0
0 1236 Worker Thread Win64 Thread 0000000077ef0a2a Normal 0
0 3240 Worker Thread Win64 Thread 0000000077ef0a2a Normal 0
0 7700 Worker Thread Win64 Thread 0000000077ef0c9a Normal 0
0 7836 Worker Thread Win64 Thread 0000000077ef0a2a Normal 0
0 4432 Worker Thread Win64 Thread 0000000077ef0a2a Normal 0
0 3040 Worker Thread Win64 Thread 0000000077ef0a2a Normal 0
0 280 Worker Thread Win64 Thread 0000000077ef0a2a Normal 0
0 8856 Worker Thread Win64 Thread 0000000077ef0a2a Normal 0
0 4688 Worker Thread Win64 Thread 0000000077ef0a2a Normal 0
0 5276 Worker Thread Win64 Thread 0000000077ef0a2a Normal 0
0 8728 Worker Thread Win64 Thread 0000000077ef0a2a Normal 0

#10

procmon shows activity like this:

9:37:52.8479752 AM deadlineslave.exe 8040 TCP Send lapro0324.scanlinevfxla.com:10681 -> deadline.scanlinevfxla.com:27017 SUCCESS Length: 197, startime: 30871256, endtime: 30871256, connid: 0, seqnum: 3717452112
9:37:52.8479813 AM deadlineslave.exe 8040 TCP Receive lapro0324.scanlinevfxla.com:10681 -> deadline.scanlinevfxla.com:27017 SUCCESS Length: 1460, connid: 0, seqnum: 3717452112
9:37:52.8480478 AM deadlineslave.exe 8040 TCP Receive lapro0324.scanlinevfxla.com:10681 -> deadline.scanlinevfxla.com:27017 SUCCESS Length: 5086, connid: 4294965980, seqnum: 3717452112
9:37:52.8513990 AM deadlineslave.exe 8040 CreateFile C:\ SUCCESS Desired Access: Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Open For Free Space Query, Attributes: n/a, ShareMode: None, AllocationSize: n/a, OpenResult: Opened
9:37:52.8514433 AM deadlineslave.exe 8040 QueryFullSizeInformationVolume C:\ SUCCESS TotalAllocationUnits: 8,389,938, CallerAvailableAllocationUnits: 622,871, ActualAvailableAllocationUnits: 622,871, SectorsPerAllocationUnit: 8, BytesPerSector: 512
9:37:52.8514599 AM deadlineslave.exe 8040 CloseFile C:\ SUCCESS
9:37:52.8515284 AM deadlineslave.exe 8040 CreateFile S:\ SUCCESS Desired Access: Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Open For Free Space Query, Attributes: n/a, ShareMode: None, AllocationSize: n/a, OpenResult: Opened
9:37:52.8515508 AM deadlineslave.exe 8040 QueryFullSizeInformationVolume S:\ SUCCESS TotalAllocationUnits: 357,894,053, CallerAvailableAllocationUnits: 85,512,826, ActualAvailableAllocationUnits: 85,512,826, SectorsPerAllocationUnit: 8, BytesPerSector: 512
9:37:52.8515632 AM deadlineslave.exe 8040 CloseFile S:\ SUCCESS
9:37:52.9533511 AM deadlineslave.exe 8040 RegQueryKey HKCU\Software\Classes SUCCESS Query: Name
9:37:52.9533691 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 NAME NOT FOUND
9:37:52.9533844 AM deadlineslave.exe 8040 RegOpenKey HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 SUCCESS
9:37:52.9534050 AM deadlineslave.exe 8040 RegQueryKey HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 SUCCESS Query: Name
9:37:52.9534261 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 NAME NOT FOUND
9:37:52.9534386 AM deadlineslave.exe 8040 RegQueryValue HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32(Default) SUCCESS Type: REG_SZ, Length: 76, Data: C:\WINDOWS\system32\wbem\wbemprox.dll
9:37:52.9534517 AM deadlineslave.exe 8040 RegQueryKey HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 SUCCESS Query: Name
9:37:52.9534670 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 NAME NOT FOUND
9:37:52.9534779 AM deadlineslave.exe 8040 RegQueryValue HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32(Default) SUCCESS Type: REG_SZ, Length: 76, Data: C:\WINDOWS\system32\wbem\wbemprox.dll
9:37:52.9534915 AM deadlineslave.exe 8040 RegCloseKey HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 SUCCESS
9:37:52.9536176 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters REPARSE
9:37:52.9536334 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:52.9536497 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Hostname SUCCESS Type: REG_SZ, Length: 20, Data: lapro0324
9:37:52.9536643 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:52.9536800 AM deadlineslave.exe 8040 RegOpenKey HKLM\Software\Policies\Microsoft\System\DNSclient NAME NOT FOUND
9:37:52.9536930 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters REPARSE
9:37:52.9537044 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:52.9537166 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Domain SUCCESS Type: REG_SZ, Length: 2, Data:
9:37:52.9537283 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:52.9537488 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters REPARSE
9:37:52.9537601 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:52.9537719 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Hostname SUCCESS Type: REG_SZ, Length: 20, Data: lapro0324
9:37:52.9537836 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:52.9537986 AM deadlineslave.exe 8040 RegOpenKey HKLM\Software\Policies\Microsoft\System\DNSclient NAME NOT FOUND
9:37:52.9538092 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters REPARSE
9:37:52.9538201 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:52.9538320 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Domain SUCCESS Type: REG_SZ, Length: 2, Data:
9:37:52.9538433 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:52.9538898 AM deadlineslave.exe 8040 Thread Create SUCCESS Thread ID: 8500
9:37:52.9539572 AM deadlineslave.exe 8040 RegQueryKey HKCU\Software\Classes SUCCESS Query: Name
9:37:52.9539695 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{8BC3F05E-D86B-11D0-A075-00C04FB68820} NAME NOT FOUND
9:37:52.9539810 AM deadlineslave.exe 8040 RegOpenKey HKCR\CLSID{8BC3F05E-D86B-11D0-A075-00C04FB68820} SUCCESS
9:37:52.9539989 AM deadlineslave.exe 8040 RegCloseKey HKCR\CLSID{8BC3F05E-D86B-11D0-A075-00C04FB68820} SUCCESS
9:37:52.9600343 AM deadlineslave.exe 8040 Thread Exit SUCCESS Thread ID: 8500, User Time: 0.0000000, Kernel Time: 0.0000000
9:37:52.9601532 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Control\ComputerName REPARSE
9:37:52.9601742 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Control\ComputerName SUCCESS
9:37:52.9601930 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Control\ComputerName\ActiveComputerName SUCCESS
9:37:52.9602086 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Control\ComputerName\ActiveComputerName\ComputerName SUCCESS Type: REG_SZ, Length: 20, Data: LAPRO0324
9:37:52.9602231 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Control\ComputerName\ActiveComputerName SUCCESS
9:37:52.9602334 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Control\ComputerName SUCCESS
9:37:52.9704753 AM deadlineslave.exe 8040 TCP Send lapro0324.scanlinevfxla.com:10681 -> deadline.scanlinevfxla.com:27017 SUCCESS Length: 109, startime: 30871257, endtime: 30871257, connid: 0, seqnum: 3717452112
9:37:52.9704756 AM deadlineslave.exe 8040 TCP Send lapro0324.scanlinevfxla.com:10681 -> deadline.scanlinevfxla.com:27017 SUCCESS Length: 1238, startime: 30871257, endtime: 30871257, connid: 0, seqnum: 3717452112
9:37:52.9705441 AM deadlineslave.exe 8040 TCP Receive lapro0324.scanlinevfxla.com:10681 -> deadline.scanlinevfxla.com:27017 SUCCESS Length: 112, connid: 0, seqnum: 3717452112
9:37:52.9711419 AM deadlineslave.exe 8040 TCP Send lapro0324.scanlinevfxla.com:10681 -> deadline.scanlinevfxla.com:27017 SUCCESS Length: 166, startime: 30871257, endtime: 30871257, connid: 0, seqnum: 3717452112
9:37:52.9711457 AM deadlineslave.exe 8040 TCP Receive lapro0324.scanlinevfxla.com:10681 -> deadline.scanlinevfxla.com:27017 SUCCESS Length: 61, connid: 0, seqnum: 3717452112
9:37:52.9721516 AM deadlineslave.exe 8040 RegQueryKey HKCU\Software\Classes SUCCESS Query: Name
9:37:52.9721666 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 NAME NOT FOUND
9:37:52.9721808 AM deadlineslave.exe 8040 RegOpenKey HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 SUCCESS
9:37:52.9721997 AM deadlineslave.exe 8040 RegQueryKey HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 SUCCESS Query: Name
9:37:52.9722187 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 NAME NOT FOUND
9:37:52.9722309 AM deadlineslave.exe 8040 RegQueryValue HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32(Default) SUCCESS Type: REG_SZ, Length: 76, Data: C:\WINDOWS\system32\wbem\wmiutils.dll
9:37:52.9722427 AM deadlineslave.exe 8040 RegQueryKey HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 SUCCESS Query: Name
9:37:52.9722575 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 NAME NOT FOUND
9:37:52.9722685 AM deadlineslave.exe 8040 RegQueryValue HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32(Default) SUCCESS Type: REG_SZ, Length: 76, Data: C:\WINDOWS\system32\wbem\wmiutils.dll
9:37:52.9722815 AM deadlineslave.exe 8040 RegCloseKey HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 SUCCESS
9:37:52.9731260 AM deadlineslave.exe 8040 Thread Create SUCCESS Thread ID: 8972
9:37:52.9736891 AM deadlineslave.exe 8040 Thread Exit SUCCESS Thread ID: 8972, User Time: 0.0000000, Kernel Time: 0.0000000
9:37:53.3104453 AM deadlineslave.exe 8040 TCP Send lapro0324.scanlinevfxla.com:10681 -> deadline.scanlinevfxla.com:27017 SUCCESS Length: 91, startime: 30871261, endtime: 30871261, connid: 0, seqnum: 3717452112
9:37:53.3104555 AM deadlineslave.exe 8040 TCP Receive lapro0324.scanlinevfxla.com:10681 -> deadline.scanlinevfxla.com:27017 SUCCESS Length: 304, connid: 0, seqnum: 3717452112
9:37:54.2034130 AM deadlineslave.exe 8040 Thread Create SUCCESS Thread ID: 7076
9:37:54.2034337 AM deadlineslave.exe 8040 RegQueryKey HKCU\Software\Classes SUCCESS Query: Name
9:37:54.2034637 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 NAME NOT FOUND
9:37:54.2034917 AM deadlineslave.exe 8040 RegOpenKey HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 SUCCESS
9:37:54.2035273 AM deadlineslave.exe 8040 RegQueryKey HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 SUCCESS Query: Name
9:37:54.2035547 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 NAME NOT FOUND
9:37:54.2035718 AM deadlineslave.exe 8040 RegQueryValue HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32(Default) SUCCESS Type: REG_SZ, Length: 76, Data: C:\WINDOWS\system32\wbem\wbemprox.dll
9:37:54.2035882 AM deadlineslave.exe 8040 RegQueryKey HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 SUCCESS Query: Name
9:37:54.2036083 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 NAME NOT FOUND
9:37:54.2036201 AM deadlineslave.exe 8040 RegQueryValue HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32(Default) SUCCESS Type: REG_SZ, Length: 76, Data: C:\WINDOWS\system32\wbem\wbemprox.dll
9:37:54.2036357 AM deadlineslave.exe 8040 RegCloseKey HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 SUCCESS
9:37:54.2037747 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters REPARSE
9:37:54.2037943 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2038114 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Hostname SUCCESS Type: REG_SZ, Length: 20, Data: lapro0324
9:37:54.2038272 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2038432 AM deadlineslave.exe 8040 RegOpenKey HKLM\Software\Policies\Microsoft\System\DNSclient NAME NOT FOUND
9:37:54.2038566 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters REPARSE
9:37:54.2038679 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2038802 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Domain SUCCESS Type: REG_SZ, Length: 2, Data:
9:37:54.2038917 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2039121 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters REPARSE
9:37:54.2039236 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2039356 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Hostname SUCCESS Type: REG_SZ, Length: 20, Data: lapro0324
9:37:54.2039474 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2039623 AM deadlineslave.exe 8040 RegOpenKey HKLM\Software\Policies\Microsoft\System\DNSclient NAME NOT FOUND
9:37:54.2039729 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters REPARSE
9:37:54.2039837 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2039957 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Domain SUCCESS Type: REG_SZ, Length: 2, Data:
9:37:54.2040071 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2040514 AM deadlineslave.exe 8040 Thread Create SUCCESS Thread ID: 7708
9:37:54.2041214 AM deadlineslave.exe 8040 RegQueryKey HKCU\Software\Classes SUCCESS Query: Name
9:37:54.2041344 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{8BC3F05E-D86B-11D0-A075-00C04FB68820} NAME NOT FOUND
9:37:54.2041459 AM deadlineslave.exe 8040 RegOpenKey HKCR\CLSID{8BC3F05E-D86B-11D0-A075-00C04FB68820} SUCCESS
9:37:54.2041644 AM deadlineslave.exe 8040 RegCloseKey HKCR\CLSID{8BC3F05E-D86B-11D0-A075-00C04FB68820} SUCCESS
9:37:54.2102874 AM deadlineslave.exe 8040 Thread Exit SUCCESS Thread ID: 7708, User Time: 0.0000000, Kernel Time: 0.0000000
9:37:54.2104112 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Control\ComputerName REPARSE
9:37:54.2104274 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Control\ComputerName SUCCESS
9:37:54.2104449 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Control\ComputerName\ActiveComputerName SUCCESS
9:37:54.2104593 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Control\ComputerName\ActiveComputerName\ComputerName SUCCESS Type: REG_SZ, Length: 20, Data: LAPRO0324
9:37:54.2104716 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Control\ComputerName\ActiveComputerName SUCCESS
9:37:54.2104816 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Control\ComputerName SUCCESS
9:37:54.2334302 AM deadlineslave.exe 8040 RegQueryKey HKCU\Software\Classes SUCCESS Query: Name
9:37:54.2334452 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 NAME NOT FOUND
9:37:54.2334589 AM deadlineslave.exe 8040 RegOpenKey HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 SUCCESS
9:37:54.2334754 AM deadlineslave.exe 8040 RegQueryKey HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 SUCCESS Query: Name
9:37:54.2334943 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 NAME NOT FOUND
9:37:54.2335069 AM deadlineslave.exe 8040 RegQueryValue HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32(Default) SUCCESS Type: REG_SZ, Length: 76, Data: C:\WINDOWS\system32\wbem\wmiutils.dll
9:37:54.2335190 AM deadlineslave.exe 8040 RegQueryKey HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 SUCCESS Query: Name
9:37:54.2335340 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 NAME NOT FOUND
9:37:54.2335449 AM deadlineslave.exe 8040 RegQueryValue HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32(Default) SUCCESS Type: REG_SZ, Length: 76, Data: C:\WINDOWS\system32\wbem\wmiutils.dll
9:37:54.2335573 AM deadlineslave.exe 8040 RegCloseKey HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 SUCCESS
9:37:54.2338082 AM deadlineslave.exe 8040 RegQueryKey HKCU\Software\Classes SUCCESS Query: Name
9:37:54.2338205 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 NAME NOT FOUND
9:37:54.2338323 AM deadlineslave.exe 8040 RegOpenKey HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 SUCCESS
9:37:54.2338475 AM deadlineslave.exe 8040 RegQueryKey HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 SUCCESS Query: Name
9:37:54.2338633 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 NAME NOT FOUND
9:37:54.2338749 AM deadlineslave.exe 8040 RegQueryValue HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32(Default) SUCCESS Type: REG_SZ, Length: 76, Data: C:\WINDOWS\system32\wbem\wbemprox.dll
9:37:54.2338854 AM deadlineslave.exe 8040 RegQueryKey HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 SUCCESS Query: Name
9:37:54.2339001 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 NAME NOT FOUND
9:37:54.2339109 AM deadlineslave.exe 8040 RegQueryValue HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32(Default) SUCCESS Type: REG_SZ, Length: 76, Data: C:\WINDOWS\system32\wbem\wbemprox.dll
9:37:54.2339227 AM deadlineslave.exe 8040 RegCloseKey HKCR\CLSID{4590F811-1D3A-11D0-891F-00AA004B2E24}\InprocServer32 SUCCESS
9:37:54.2339969 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters REPARSE
9:37:54.2340124 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2340277 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Hostname SUCCESS Type: REG_SZ, Length: 20, Data: lapro0324
9:37:54.2340416 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2340572 AM deadlineslave.exe 8040 RegOpenKey HKLM\Software\Policies\Microsoft\System\DNSclient NAME NOT FOUND
9:37:54.2340699 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters REPARSE
9:37:54.2340809 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2340931 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Domain SUCCESS Type: REG_SZ, Length: 2, Data:
9:37:54.2341047 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2341248 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters REPARSE
9:37:54.2341358 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2341476 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Hostname SUCCESS Type: REG_SZ, Length: 20, Data: lapro0324
9:37:54.2341592 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2341742 AM deadlineslave.exe 8040 RegOpenKey HKLM\Software\Policies\Microsoft\System\DNSclient NAME NOT FOUND
9:37:54.2341850 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters REPARSE
9:37:54.2341959 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2342077 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Domain SUCCESS Type: REG_SZ, Length: 2, Data:
9:37:54.2342191 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Services\Tcpip\Parameters SUCCESS
9:37:54.2342571 AM deadlineslave.exe 8040 Thread Create SUCCESS Thread ID: 5500
9:37:54.2343108 AM deadlineslave.exe 8040 RegQueryKey HKCU\Software\Classes SUCCESS Query: Name
9:37:54.2343233 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{8BC3F05E-D86B-11D0-A075-00C04FB68820} NAME NOT FOUND
9:37:54.2343348 AM deadlineslave.exe 8040 RegOpenKey HKCR\CLSID{8BC3F05E-D86B-11D0-A075-00C04FB68820} SUCCESS
9:37:54.2343521 AM deadlineslave.exe 8040 RegCloseKey HKCR\CLSID{8BC3F05E-D86B-11D0-A075-00C04FB68820} SUCCESS
9:37:54.2400991 AM deadlineslave.exe 8040 Thread Exit SUCCESS Thread ID: 5500, User Time: 0.0000000, Kernel Time: 0.0000000
9:37:54.2401946 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Control\ComputerName REPARSE
9:37:54.2402108 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Control\ComputerName SUCCESS
9:37:54.2402279 AM deadlineslave.exe 8040 RegOpenKey HKLM\System\CurrentControlSet\Control\ComputerName\ActiveComputerName SUCCESS
9:37:54.2402418 AM deadlineslave.exe 8040 RegQueryValue HKLM\System\CurrentControlSet\Control\ComputerName\ActiveComputerName\ComputerName SUCCESS Type: REG_SZ, Length: 20, Data: LAPRO0324
9:37:54.2402541 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Control\ComputerName\ActiveComputerName SUCCESS
9:37:54.2402640 AM deadlineslave.exe 8040 RegCloseKey HKLM\System\CurrentControlSet\Control\ComputerName SUCCESS
9:37:54.2517480 AM deadlineslave.exe 8040 RegQueryKey HKCU\Software\Classes SUCCESS Query: Name
9:37:54.2517722 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 NAME NOT FOUND
9:37:54.2517955 AM deadlineslave.exe 8040 RegOpenKey HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 SUCCESS
9:37:54.2518189 AM deadlineslave.exe 8040 RegQueryKey HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 SUCCESS Query: Name
9:37:54.2518400 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 NAME NOT FOUND
9:37:54.2518546 AM deadlineslave.exe 8040 RegQueryValue HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32(Default) SUCCESS Type: REG_SZ, Length: 76, Data: C:\WINDOWS\system32\wbem\wmiutils.dll
9:37:54.2518692 AM deadlineslave.exe 8040 RegQueryKey HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 SUCCESS Query: Name
9:37:54.2518844 AM deadlineslave.exe 8040 RegOpenKey HKCU\Software\Classes\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 NAME NOT FOUND
9:37:54.2518955 AM deadlineslave.exe 8040 RegQueryValue HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32(Default) SUCCESS Type: REG_SZ, Length: 76, Data: C:\WINDOWS\system32\wbem\wmiutils.dll
9:37:54.2519101 AM deadlineslave.exe 8040 RegCloseKey HKCR\CLSID{CF4CC405-E2C5-4DDD-B3CE-5E7582D8C9FA}\InprocServer32 SUCCESS

#11

I’m wondering if the issue is completely UI related. Can you pick 10 machines or so and launch the slave in nogui mode on them? Just open a command prompt, switch to the Deadline bin folder, and then run this:

deadlineslave.exe -nogui -console

This will make the slave run without the user interface, and it will log everything to the console window.

If these slaves continue to run for a while without dying, then that could indicate it’s strictly a UI problem.

Thanks!

#12

THats a bit problematic for us to try, we currently have 130 nodes on deadline, and only about 5-10% of them stall like this overnight. Since we have scheduling on to restart slaves if the slave totally crashes out, they would randomly get restarted by the launcher, with the nogui option disabled.

What do you suggest? Is there a way to ask the launcher to start the slave in nogui mode? Because even if we started them like that, chances are some of them would be back with a gui by the morning.

#13

Another slave that did this (reporting rendering something, but its actually hung up) had a weird behaviour, as in the monitor, the job reported complete! In the slave listing, it reported rendering.

There was a task report for that machine reporting stalled:

=======================================================
Error

STALLED SLAVE REPORT

=======================================================
Log

STALLED SLAVE REPORT

Current House Cleaner Information
Machine Performing Cleanup: lapro0216
Version: v6.0.0.50272 R

Stalled Slave: LAPRO0291
Last Slave Update: 2013-02-20 17:38:10
Current Time: 2013-02-20 18:38:17
Time Difference: 1.002 hrs
Maximum Time Allowed Between Updates: 10.000 m

Current Job Name: [TBOA] Software Render: FB_085_0830_maya_animation_samuraianimboat3.ma version: v0014
Current Job ID: 51253627d7c8120f44c381e1
Current Job User: Bill.Bridges
Current Task Names: 1480-1494
Current Task Ids: 6

Searching for job with id “51253627d7c8120f44c381e1”
Found possible job: [TBOA] Software Render: FB_085_0830_maya_animation_samuraianimboat3.ma version: v0014
Searching for task with id “6”
Found possible task: :[1480-1494]
Task’s current slave: LAPRO0291
Slave machine names match, stopping search
Associated Job Found: [TBOA] Software Render: FB_085_0830_maya_animation_samuraianimboat3.ma version: v0014
Associated Task Found: :[1480-1494]
Task’s current slave: LAPRO0291
Task is still rendering, attempting to fix situation.
Requeuing task
Setting slave’s status to Stalled.
Setting last update time to now.

Slave state updated.

=======================================================
Type

=======================================================
Stack Trace

=======================================================
Details

Date: 2013/02/20 18:38:17
Frames: 1480-1494
Elapsed Time: 00:00:00:00

But the slave itself was still there, again using 100% of the cpu.

Last lines of the slave log from the 20st:

2013-02-20 12:47:38: 0: STDOUT: PyQt4.uic.properties : DEBUG : setting property text
2013-02-20 12:47:38: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : push QCheckBox m_uTestImport
2013-02-20 12:47:38: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : pop widget QCheckBox m_uTestImport
2013-02-20 12:47:38: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : new topwidget Dialog
2013-02-20 12:47:38: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : pop layout QVBoxLayout verticalLayout_3
2013-02-20 12:47:38: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : pop widget QDialog Dialog
2013-02-20 12:47:38: 0: STDOUT: PyQt4.uic.uiparser : DEBUG : new topwidget None
2013-02-20 12:47:38: 0: STDOUT: Wed Feb 20 12:47:38 2013 (+186ms) : debug : Registering Node Class : “MocapAnimationExporter”
2013-02-20 12:47:38: 0: STDOUT: Wed Feb 20 12:47:38 2013 (+0ms) : debug : Registering Node Class : “MocapAnimationImporter”
2013-02-20 12:47:38: 0: STDOUT: Wed Feb 20 12:47:38 2013 (+0ms) : debug : Registering Node Class : “RigCacheExporter”
2013-02-20 17:37:02: Scheduler Thread - Cancelling task because task “6_1480-1494” could not be found

Next log from this morning:

2013-02-21 08:47:12: BEGIN - LAPRO0291\ScanlineVFX
2013-02-21 08:47:12: 0: Slave timed out – canceling current task…

#14

Is it the same 5-10%, or is it completely random? I’m guessing it’s random, but just want to confirm.

I was under the impression that in most cases, the slaves don’t actually crash, they just hang, and keep reporting that their active in the Monitor. If this is the case, you don’t need to worry about the scheduling restarting the slaves on those machines in gui mode. So again, just to confirm, do you mostly see hangs, or crashes, or is it usually a combination between the two?

I guess to simplify the test, you can force the launcher to start the slaves in nogui mode by running the launcher itself in nogui mode:

deadlinelauncher.exe -nogui -console

Do this on 10-20 machines, and then monitor them to see if they every crash or hang.

We’re still trying to reproduce this crashing here. We just got some new hardware in, so we’ll be scaling up our tests even more in hopes we start seeing this issue. Out of curiosity, how long, in general, are the logs generated by your jobs? We suspect that it’s a UI issue (which we hope the nogui tests you run will confirm or deny), and if the slave has to process a very long log, maybe that has something to do with it.

#15

The logs can be long, but nothing excessive, 2-3000 lines, 3-400kb per task maybe

l

#16

Good to know about the log size, thanks!

There were a couple other questions though:

  1. Is it the same 5-10% or machines that crash/hang, or is it completely random? I’m guessing it’s random, but just want to confirm.
  2. So again, just to confirm, do you mostly see hangs, or crashes, or is it usually a combination between the two?

Another question we thought of as well:
3) Does this only happen when the slave is rendering a particular type of job (ie: maya, nuke, etc), or does the job type not seem to matter?

We want to make sure that when we’re trying to reproduce, that we’re pushing jobs through that are similar to yours.

Thanks again!

  • Ryan
Privacy | Site terms | Cookie preferences