Hi there,
It seems that the slave logs can easily stop updating (while the job is still going). Most of the machines rendering a particular job have their local log window frozen (its interactive, i can scroll it, but there is no new content appearing), even though the job is still going. We have our internal logs being written to a file, so i see that its doing something.
The last couple of lines of one particular task are:
1-30 19:58:42: 0: STDOUT: Wed Jan 30 19:58:42 2013 (+0ms) : debug : Initiating pc2 file: //inferno2/projects/tboa/scenes/FB_360_0001/cache/common/rigcache/vehThemistoklesWarshipA_7/v0003_mpo_FR1_Layout_360_Test/pc2/mesh_m_high_flag.pc2
2013-01-30 19:58:42: 0: STDOUT: Wed Jan 30 19:58:42 2013 (+47ms) : debug : Initiating pc2 file: //inferno2/projects/tboa/scenes/FB_360_0001/cache/common/rigcache/vehThemistoklesWarshipA_7/v0003_mpo_FR1_Layout_360_Test/pc2/mesh_m_low_flag.pc2
2013-01-30 19:58:42: 0: STDOUT: Wed Jan 30 19:58:42 2013 (+15ms) : debug : Initiating pc2 file: //inferno2/projects/tboa/scenes/FB_360_0001/cache/common/rigcache/vehThemistoklesWarshipA_7/v0003_mpo_FR1_Layout_360_Test/pc2/mesh_m_sim_flag.pc2
Then nothing…
While in our own logs:
03h:41m:49s (step 910/7626) Overall Progress: 17 percent
Wed Jan 30 20:28:48 2013 (+1875ms) : PUBLISH STEP 4a/7 : Frame: 1861, Step time: 1.88 secs. Avg step time: 1.98 secs. Total time spent: 00h:30m:05s Estimated time left: 03h:41m:46s (step 911/7626) Overall Progress: 17 percent
Wed Jan 30 20:28:50 2013 (+1796ms) : PUBLISH STEP 4a/7 : Frame: 1862, Step time: 1.80 secs. Avg step time: 1.98 secs. Total time spent: 00h:30m:07s Estimated time left: 03h:41m:43s (step 912/7626) Overall Progress: 17 percent
Wed Jan 30 20:28:52 2013 (+1796ms) : PUBLISH STEP 4a/7 : Frame: 1863, Step time: 1.80 secs. Avg step time: 1.98 secs. Total time spent: 00h:30m:08s Estimated time left: 03h:41m:39s (step 913/7626) Overall Progress: 17 percent