AWS Thinkbox Discussion Forums

Corona Render crashes on animation render

Deprecated Deadline 7 Bug Reports
#20

Everything thing there looks good - you have the default 20 seconds between slave updates, which is well below the 10 minutes before a slave is considered stalled.

I’m really starting to get stumped here.

I took another look through this thread, and realized I had missed the job’s history you had posted:

Is it at all possible that in the most recent case of this issue, the tasks were simply requeued from the Monitor? The slave would behave just like this in this case as well, since its task has changed.

If that’s not the case, it might be best to contact our support team at support[at]thinkboxsoftware[dot]com, and they can set up a remote session to try and figure out what’s going on.

Cheers,
Ryan

#21

it happens with clean submitted jobs, they timeout after a while.
will send a complete job package to you that does it

#22

Simple scene crashes max 2014
deadline rc3

this is to one machine only, running distributed render job
have attached the job archive and all the logs
we are trying again in 2015
pulse logs.zip (250 KB)
thomas__3dsmax__test_scene __547bbba8544495295883c80d.zip (34.8 KB)
Deadline Logs today.zip (806 KB)

#23

oh and its corona 0.72 rc1

#24

does it in max2015 as well

same error with scene updated in the log
no errors flagged in monitor, scene just starts again
m2015.zip (137 KB)

#25

does it as standalone, no distributed render !?
so 1 workstation no other nodes involved, causes it to flag changed

on the machines not rendering via deadline
we get in scheduler status

  • Waiting
  • DequeuingTask

changing back and forward every 5 seconds
its not related is it?

#26

We tracked down a bug in the slave that could very well be the source of this problem. This bug would prevent the slave from ever reporting its state again after the first time it does it when it starts up. This could result in the behavior you’re seeing, since the slave never would update its state while its running, and would therefore appear as stalled. Note that the behavior you’re seeing has nothing to do with the type of job that the slave is rendering.

We’re fixing this in RC4, which we will push out this week.

To confirm if you’re seeing this bug, you can watch the Slaves panel in the Monitor to see if there are some slaves that don’t seem to ever update their state (you can watch the “Last Status Update” column).

Cheers,
Ryan

#27

Just to clarify, you can watch the “Last Status Update” column to see if the time there every changes. It should get changed every time the slave updates its state, so if it never changes while the slave is running, that means you’re seeing the bug I mentioned.

#28

ok, will check it out tomorrow
thanks

#29

ok
tracking last update status and it seems to be updating ok
we are resubmitting jobs that failed yesterday

one change we have made is we have installed max2015 to all rendernodes (to see if it happens in 2015)
just wondering if the absence of 2015 on some of the nodes could cause this kind of error

it seems really odd that it just starts working as thats the only thing that has changed
we still have 1 node that is old version (havent been able to update that one yet)
as i thought old deadline versions even if offline might have caused issues.

ive uploaded logs of two jobs that failed yesterday to see if you can compare details
Reports_Job.zip (530 KB)

#30

looks like having max2015 and max2014 on all machines has somehow made the whole thing stable
we are running a larger test tonight

it used to to only take 3-16 minutes before to error
how odd

#31

Hmm, as I mentioned yesterday, this shouldn’t be related to the types of jobs that the slaves are rendering (or the rendering software installed on them). The bug I had mentioned was a somewhat random one. Sometimes a slave will report its state just fine after startup, other times it will only do it once and then never again until the slave is restarted again. In the latter case, the slave will eventually get marked as stalled, and its task will eventually get requeued. It’s probably just a coincidence that everything appears to be running smoothly right now.

I see from the logs that you are running RC3, so you’re definitely vulnerable to this bug. RC4 should be available tomorrow, which should fix this issue once and for all.

Cheers,
Ryan

#32

hi ryan

weve run a bunch of stuff overnight and it all finished, no errors
the only thing that seems to be different is every slave has same max software
max design 2014 and max design 2015

it doesnt make any sense to me why it would go away doing that. unless its an some wierdness in the job software checking that modifies the job somehow
will try rc4 asap

#33

Just an FYI that Deadline 7 RC4 has been released!
viewtopic.php?f=204&t=12744

Cheers,
Ryan

#34

Ryan
we are still getting this with a clean install of Deadline 7.01
new mongo, new repository, new client, new submission
stable build of corona, not daily

2014-12-23 10:17:33: 0: INFO: Rendering pass 21/100 (this: 0:01:30, total: 0:02:12)
2014-12-23 10:17:36: 0: INFO: Rendering pass 22/100 (this: 0:01:33, total: 0:02:15)
2014-12-23 10:17:36: 0: INFO: Rendering pass 22/100 (this: 0:01:33, total: 0:02:15)
2014-12-23 10:17:37: 0: INFO: Rendering pass 22/100 (this: 0:01:34, total: 0:02:16)
2014-12-23 10:17:37: 0: INFO: Rendering pass 23/100 (this: 0:01:34, total: 0:02:16)
2014-12-23 10:17:38: 0: INFO: Rendering pass 23/100 (this: 0:01:35, total: 0:02:17)
2014-12-23 10:17:38: 0: INFO: Rendering pass 23/100 (this: 0:01:35, total: 0:02:17)
2014-12-23 10:17:47: 0: INFO: Rendering pass 25/100 (this: 0:01:44, total: 0:02:26)
2014-12-23 10:17:47: 0: INFO: Rendering pass 25/100 (this: 0:01:44, total: 0:02:26)
2014-12-23 10:17:48: 0: INFO: Rendering pass 25/100 (this: 0:01:45, total: 0:02:27)
2014-12-23 10:17:48: 0: INFO: Rendering pass 25/100 (this: 0:01:45, total: 0:02:28)
2014-12-23 10:17:49: Connecting to slave log: QBOX00
2014-12-23 10:17:49: 0: INFO: Rendering pass 25/100 (this: 0:01:46, total: 0:02:28)
2014-12-23 10:18:07: Scheduler Thread - Task “1_1-1” could not be found because task has been modified:
2014-12-23 10:18:07: current status = Rendering, new status = Queued
2014-12-23 10:18:07: current slave = QBOX00, new slave =
2014-12-23 10:18:07: current frames = 1-1, new frames = 1-1
2014-12-23 10:18:07: Scheduler Thread - Cancelling task…
2014-12-23 10:18:09: 0: In the process of canceling current task: ignoring exception thrown by PluginLoader
2014-12-23 10:18:15: Scheduler Thread - In the process of canceling current tasks: ignoring exception thrown by render thread 0
2014-12-23 10:18:17: Scheduler Thread - Synchronizing job auxiliary files from \wbox\DeadlineRepository7\jobs\5498b0de5f874f1498396d89

pulse log below

Connecting to Pulse (WBOX:61083)…
2014-12-23 10:23:25: Deleted Job Scan - Loaded 2 deleted jobs in 0.000 s
2014-12-23 10:23:25: Deleted Job Scan - Purged 0 deleted jobs in 0.000 s
2014-12-23 10:23:25: Deleted Job Scan - Done.
2014-12-23 10:23:25: Purging Old Job Auxiliary Files
2014-12-23 10:23:25: Auxiliary File Scan - Scanning for auxiliary directories
2014-12-23 10:23:25: Auxiliary File Scan - Found 4 auxiliary directories in 0.000 s
2014-12-23 10:23:25: Auxiliary File Scan - Loading job IDs
2014-12-23 10:23:25: Auxiliary File Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:23:25: Auxiliary File Scan - Purged 0 auxiliary folders in 0.000 s
2014-12-23 10:23:25: Auxiliary File Scan - Done.
2014-12-23 10:23:25: Purging Old Job Reports
2014-12-23 10:23:25: Job Report Scan - Loading job report collections
2014-12-23 10:23:25: Job Report Scan - Found 4 report collections in 0.000 s
2014-12-23 10:23:25: Job Report Scan - Loading job IDs
2014-12-23 10:23:25: Job Report Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:23:25: Job Report Scan - Purged 0 report collections in 0.000 s
2014-12-23 10:23:25: Job Report Scan - Purging old job report files
2014-12-23 10:23:25: Job Report Scan - Purged 0 report files in 0.000 s
2014-12-23 10:23:25: Job Report Scan - Done.
2014-12-23 10:23:25: Purging Obsolete Slaves
2014-12-23 10:23:25: Obsolete Slave Scan - Skipping because it is disabled in the Repository Options
2014-12-23 10:23:25: Purging Old Slave Reports
2014-12-23 10:23:25: Slave Report Scan - Loading slave report collections
2014-12-23 10:23:25: Slave Report Scan - Found 3 report collections in 0.000 s
2014-12-23 10:23:25: Slave Report Scan - Loading slave IDs
2014-12-23 10:23:25: Slave Report Scan - Loaded 12 slave IDs in 0.000 s
2014-12-23 10:23:25: Slave Report Scan - Purged 0 report collections in 0.000 s
2014-12-23 10:23:25: Slave Report Scan - Done.
2014-12-23 10:23:25: Purging Old Limits
2014-12-23 10:23:25: Old Limit Scan - Loading machine limits
2014-12-23 10:23:25: Old Limit Scan - Found 2 machine limits in 0.000 s
2014-12-23 10:23:25: Old Limit Scan - Loading job IDs
2014-12-23 10:23:25: Old Limit Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:23:25: Old Limit Scan - Purged 0 machine limits in 0.000 s
2014-12-23 10:23:25: Old Limit Scan - Done.
2014-12-23 10:23:25: Purging Temporary Repository Files
2014-12-23 10:23:25: Temporary File Scan - Scanning for ‘connectReadWriteTest’ files
2014-12-23 10:23:25: Temporary File Scan - Deleted 0 temporary files in 0.000 s
2014-12-23 10:23:25: Temporary File Scan - Done.
2014-12-23 10:23:25: Purging Old Statistics
2014-12-23 10:23:25: Old Statistics - Skipping job statistics because the option to purge them is disabled in the Repository Options
2014-12-23 10:23:25: Old Statistics - Purging slave statistics that are older than Aug 25/14 10:23:25
2014-12-23 10:23:25: Old Statistics - Purged old slave statistics in 0.000 s
2014-12-23 10:23:25: Old Statistics - Purging repository statistics that are older than Aug 25/14 10:23:25
2014-12-23 10:23:25: Old Statistics - Purged old repository statistics in 0.000 s
2014-12-23 10:23:25: Purging Deleted Document Stubs From Database
2014-12-23 10:23:25: Deleted Document Stubs - Deleting stubs that are older than 3 days
2014-12-23 10:23:25: Deleted Document Stubs - Deleted 0 stubs in 0.000 s
2014-12-23 10:23:25: Triggering House Cleaning Events
2014-12-23 10:23:25: Performing repository repair
2014-12-23 10:23:25: Performing Orphaned Task Scan…
2014-12-23 10:23:25: Orphaned Task Scan - Loading rendering jobs
2014-12-23 10:23:25: Orphaned Task Scan - Loaded 1 rendering jobs in 0.000 s
2014-12-23 10:23:25: Orphaned Task Scan - Scanning for orphaned tasks
2014-12-23 10:23:25: Orphaned Task Scan - Cleaned up 0 orphaned tasks in 0.000 s
2014-12-23 10:23:25: Orphaned Task Scan - Done.
2014-12-23 10:23:25: Performing Orphaned Limit Stub Scan…
2014-12-23 10:23:25: Orphaned Limit Stub Scan - Loading limits
2014-12-23 10:23:25: Orphaned Limit Stub Scan - Loaded 2 limits in 0.000 s
2014-12-23 10:23:25: Orphaned Limit Stub Scan - Loading slave states
2014-12-23 10:23:25: Orphaned Limit Stub Scan - Loaded 12 slave states in 0.000 s
2014-12-23 10:23:25: Orphaned Limit Stub Scan - Returned 0 orphaned limit stubs in 0.000 s
2014-12-23 10:23:25: Orphaned Limit Stub Scan - Done.
2014-12-23 10:23:25: Checking Available Database Connections
2014-12-23 10:23:25: Available Database Connections - Skipping because there are no Low Database Connection notification email addresses set in the Repository Options
2014-12-23 10:23:25: Performing Stalled Slave Scan…
2014-12-23 10:23:25: Stalled Slave Scan - Loading slave states
2014-12-23 10:23:25: Stalled Slave Scan - Loaded 12 slave states in 0.000 s
2014-12-23 10:23:25: Stalled Slave Scan - Scanning slave states
2014-12-23 10:23:25: Stalled Slave Scan - Cleaned up 0 stalled slaves in 0.000 s
2014-12-23 10:23:25: Stalled Slave Scan - Done.
2014-12-23 10:23:25: Performing Stalled Pulse Scan…
2014-12-23 10:23:25: Stalled Pulse Scan - Loading pulse states
2014-12-23 10:23:25: Stalled Pulse Scan - Loaded 1 pulse states in 0.000 s
2014-12-23 10:23:25: Stalled Pulse Scan - Scanning pulse states
2014-12-23 10:23:25: Stalled Pulse Scan - Cleaned up 0 stalled pulses in 0.000 s
2014-12-23 10:23:25: Stalled Pulse Scan - Done.
2014-12-23 10:23:25: Performing Stalled Balancer Scan…
2014-12-23 10:23:25: Stalled Balancer Scan - Loading balancer states
2014-12-23 10:23:25: Stalled Balancer Scan - Loaded 0 balancer states in 0.000 s
2014-12-23 10:23:25: Stalled Balancer Scan - Done.
2014-12-23 10:23:25: Triggering Repository Repair Events
2014-12-23 10:24:24: Performing Pending Job Scan…
2014-12-23 10:24:24: Pending Job Scan - Loading pending and active jobs
2014-12-23 10:24:24: Pending Job Scan - Loaded 1 pending and active jobs in 0.000 s
2014-12-23 10:24:24: Pending Job Scan - Scanning pending and active jobs
2014-12-23 10:24:24: Pending Job Scan - Released 0 pending jobs and 0 pending tasks in 0.000 s
2014-12-23 10:24:24: Pending Job Scan - Done.
2014-12-23 10:24:24: Processing Pending Job Events
2014-12-23 10:24:24: Pending Job Events - Checking for pending job events
2014-12-23 10:24:24: Pending Job Events - Processing 0 job events
2014-12-23 10:24:24: Pending Job Events - No more job events to process
2014-12-23 10:24:24: Pending Job Events - Done.
2014-12-23 10:24:24: Performing house cleaning
2014-12-23 10:24:24: Performing Job Cleanup Scan…
2014-12-23 10:24:24: Job Cleanup Scan - Loading completed jobs
2014-12-23 10:24:24: Job Cleanup Scan - Loaded 0 completed jobs in 0.000 s
2014-12-23 10:24:24: Job Cleanup Scan - Done.
2014-12-23 10:24:24: Purging Unsubmitted Jobs
2014-12-23 10:24:24: Unsubmitted Job Scan - Loading unsubmitted jobs
2014-12-23 10:24:24: Unsubmitted Job Scan - Loaded 0 unsubmitted jobs in 0.000 s
2014-12-23 10:24:24: Unsubmitted Job Scan - Done.
2014-12-23 10:24:24: Purging Deleted Jobs
2014-12-23 10:24:24: Deleted Job Scan - Loading deleted jobs
2014-12-23 10:24:24: Deleted Job Scan - Loaded 2 deleted jobs in 0.000 s
2014-12-23 10:24:24: Deleted Job Scan - Purged 0 deleted jobs in 0.000 s
2014-12-23 10:24:24: Deleted Job Scan - Done.
2014-12-23 10:24:24: Purging Old Job Auxiliary Files
2014-12-23 10:24:24: Auxiliary File Scan - Scanning for auxiliary directories
2014-12-23 10:24:24: Auxiliary File Scan - Found 4 auxiliary directories in 0.000 s
2014-12-23 10:24:24: Auxiliary File Scan - Loading job IDs
2014-12-23 10:24:24: Auxiliary File Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:24:24: Auxiliary File Scan - Purged 0 auxiliary folders in 0.000 s
2014-12-23 10:24:24: Auxiliary File Scan - Done.
2014-12-23 10:24:24: Purging Old Job Reports
2014-12-23 10:24:24: Job Report Scan - Loading job report collections
2014-12-23 10:24:24: Job Report Scan - Found 4 report collections in 0.000 s
2014-12-23 10:24:24: Job Report Scan - Loading job IDs
2014-12-23 10:24:24: Job Report Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:24:24: Job Report Scan - Purged 0 report collections in 0.000 s
2014-12-23 10:24:24: Job Report Scan - Purging old job report files
2014-12-23 10:24:24: Job Report Scan - Purged 0 report files in 0.000 s
2014-12-23 10:24:24: Job Report Scan - Done.
2014-12-23 10:24:24: Purging Obsolete Slaves
2014-12-23 10:24:24: Obsolete Slave Scan - Skipping because it is disabled in the Repository Options
2014-12-23 10:24:24: Purging Old Slave Reports
2014-12-23 10:24:24: Slave Report Scan - Loading slave report collections
2014-12-23 10:24:24: Slave Report Scan - Found 3 report collections in 0.000 s
2014-12-23 10:24:24: Slave Report Scan - Loading slave IDs
2014-12-23 10:24:24: Slave Report Scan - Loaded 12 slave IDs in 0.000 s
2014-12-23 10:24:24: Slave Report Scan - Purged 0 report collections in 0.000 s
2014-12-23 10:24:24: Slave Report Scan - Done.
2014-12-23 10:24:24: Purging Old Limits
2014-12-23 10:24:24: Old Limit Scan - Loading machine limits
2014-12-23 10:24:24: Old Limit Scan - Found 2 machine limits in 0.000 s
2014-12-23 10:24:24: Old Limit Scan - Loading job IDs
2014-12-23 10:24:24: Old Limit Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:24:24: Old Limit Scan - Purged 0 machine limits in 0.000 s
2014-12-23 10:24:24: Old Limit Scan - Done.
2014-12-23 10:24:24: Purging Temporary Repository Files
2014-12-23 10:24:24: Temporary File Scan - Scanning for ‘connectReadWriteTest’ files
2014-12-23 10:24:24: Temporary File Scan - Deleted 0 temporary files in 0.000 s
2014-12-23 10:24:24: Temporary File Scan - Done.
2014-12-23 10:24:24: Purging Old Statistics
2014-12-23 10:24:24: Old Statistics - Skipping job statistics because the option to purge them is disabled in the Repository Options
2014-12-23 10:24:24: Old Statistics - Purging slave statistics that are older than Aug 25/14 10:24:24
2014-12-23 10:24:24: Old Statistics - Purged old slave statistics in 0.000 s
2014-12-23 10:24:24: Old Statistics - Purging repository statistics that are older than Aug 25/14 10:24:24
2014-12-23 10:24:24: Old Statistics - Purged old repository statistics in 0.000 s
2014-12-23 10:24:24: Purging Deleted Document Stubs From Database
2014-12-23 10:24:24: Deleted Document Stubs - Deleting stubs that are older than 3 days
2014-12-23 10:24:24: Deleted Document Stubs - Deleted 0 stubs in 0.000 s
2014-12-23 10:24:24: Triggering House Cleaning Events
2014-12-23 10:24:25: Performing repository repair
2014-12-23 10:24:25: Performing Orphaned Task Scan…
2014-12-23 10:24:25: Orphaned Task Scan - Loading rendering jobs
2014-12-23 10:24:25: Orphaned Task Scan - Loaded 1 rendering jobs in 0.000 s
2014-12-23 10:24:25: Orphaned Task Scan - Scanning for orphaned tasks
2014-12-23 10:24:25: Orphaned Task Scan - Cleaned up 1 orphaned tasks in 15.627 ms
2014-12-23 10:24:25: Orphaned Task Scan - Done.
2014-12-23 10:24:25: Performing Orphaned Limit Stub Scan…
2014-12-23 10:24:25: Orphaned Limit Stub Scan - Loading limits
2014-12-23 10:24:25: Orphaned Limit Stub Scan - Loaded 2 limits in 0.000 s
2014-12-23 10:24:25: Orphaned Limit Stub Scan - Loading slave states
2014-12-23 10:24:25: Orphaned Limit Stub Scan - Loaded 12 slave states in 0.000 s
2014-12-23 10:24:25: Orphaned Limit Stub Scan - Returned 0 orphaned limit stubs in 0.000 s
2014-12-23 10:24:25: Orphaned Limit Stub Scan - Done.
2014-12-23 10:24:25: Checking Available Database Connections
2014-12-23 10:24:25: Available Database Connections - Skipping because there are no Low Database Connection notification email addresses set in the Repository Options
2014-12-23 10:24:25: Performing Stalled Slave Scan…
2014-12-23 10:24:25: Stalled Slave Scan - Loading slave states
2014-12-23 10:24:25: Stalled Slave Scan - Loaded 12 slave states in 0.000 s
2014-12-23 10:24:25: Stalled Slave Scan - Scanning slave states
2014-12-23 10:24:25: Stalled Slave Scan - Cleaned up 0 stalled slaves in 0.000 s
2014-12-23 10:24:25: Stalled Slave Scan - Done.
2014-12-23 10:24:25: Performing Stalled Pulse Scan…
2014-12-23 10:24:25: Stalled Pulse Scan - Loading pulse states
2014-12-23 10:24:25: Stalled Pulse Scan - Loaded 1 pulse states in 0.000 s
2014-12-23 10:24:25: Stalled Pulse Scan - Scanning pulse states
2014-12-23 10:24:25: Stalled Pulse Scan - Cleaned up 0 stalled pulses in 0.000 s
2014-12-23 10:24:25: Stalled Pulse Scan - Done.
2014-12-23 10:24:25: Performing Stalled Balancer Scan…
2014-12-23 10:24:25: Stalled Balancer Scan - Loading balancer states
2014-12-23 10:24:25: Stalled Balancer Scan - Loaded 0 balancer states in 0.000 s
2014-12-23 10:24:25: Stalled Balancer Scan - Done.
2014-12-23 10:24:25: Triggering Repository Repair Events
2014-12-23 10:25:24: Performing Pending Job Scan…
2014-12-23 10:25:24: Pending Job Scan - Loading pending and active jobs
2014-12-23 10:25:24: Pending Job Scan - Loaded 1 pending and active jobs in 0.000 s
2014-12-23 10:25:24: Pending Job Scan - Scanning pending and active jobs
2014-12-23 10:25:24: Pending Job Scan - Released 0 pending jobs and 0 pending tasks in 0.000 s
2014-12-23 10:25:24: Pending Job Scan - Done.
2014-12-23 10:25:24: Processing Pending Job Events
2014-12-23 10:25:24: Pending Job Events - Checking for pending job events
2014-12-23 10:25:24: Pending Job Events - Processing 0 job events
2014-12-23 10:25:24: Pending Job Events - No more job events to process
2014-12-23 10:25:24: Pending Job Events - Done.
2014-12-23 10:25:24: Performing house cleaning
2014-12-23 10:25:24: Performing Job Cleanup Scan…
2014-12-23 10:25:24: Job Cleanup Scan - Loading completed jobs
2014-12-23 10:25:24: Job Cleanup Scan - Loaded 0 completed jobs in 0.000 s
2014-12-23 10:25:24: Job Cleanup Scan - Done.
2014-12-23 10:25:24: Purging Unsubmitted Jobs
2014-12-23 10:25:24: Unsubmitted Job Scan - Loading unsubmitted jobs
2014-12-23 10:25:24: Unsubmitted Job Scan - Loaded 0 unsubmitted jobs in 0.000 s
2014-12-23 10:25:24: Unsubmitted Job Scan - Done.
2014-12-23 10:25:24: Purging Deleted Jobs
2014-12-23 10:25:24: Deleted Job Scan - Loading deleted jobs
2014-12-23 10:25:24: Deleted Job Scan - Loaded 2 deleted jobs in 0.000 s
2014-12-23 10:25:24: Deleted Job Scan - Purged 0 deleted jobs in 0.000 s
2014-12-23 10:25:24: Deleted Job Scan - Done.
2014-12-23 10:25:24: Purging Old Job Auxiliary Files
2014-12-23 10:25:24: Auxiliary File Scan - Scanning for auxiliary directories
2014-12-23 10:25:24: Auxiliary File Scan - Found 4 auxiliary directories in 0.000 s
2014-12-23 10:25:24: Auxiliary File Scan - Loading job IDs
2014-12-23 10:25:24: Auxiliary File Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:25:24: Auxiliary File Scan - Purged 0 auxiliary folders in 0.000 s
2014-12-23 10:25:24: Auxiliary File Scan - Done.
2014-12-23 10:25:24: Purging Old Job Reports
2014-12-23 10:25:24: Job Report Scan - Loading job report collections
2014-12-23 10:25:24: Job Report Scan - Found 4 report collections in 0.000 s
2014-12-23 10:25:24: Job Report Scan - Loading job IDs
2014-12-23 10:25:24: Job Report Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:25:24: Job Report Scan - Purged 0 report collections in 0.000 s
2014-12-23 10:25:24: Job Report Scan - Purging old job report files
2014-12-23 10:25:24: Job Report Scan - Purged 0 report files in 0.000 s
2014-12-23 10:25:24: Job Report Scan - Done.
2014-12-23 10:25:24: Purging Obsolete Slaves
2014-12-23 10:25:24: Obsolete Slave Scan - Skipping because it is disabled in the Repository Options
2014-12-23 10:25:24: Purging Old Slave Reports
2014-12-23 10:25:24: Slave Report Scan - Loading slave report collections
2014-12-23 10:25:24: Slave Report Scan - Found 3 report collections in 0.000 s
2014-12-23 10:25:24: Slave Report Scan - Loading slave IDs
2014-12-23 10:25:24: Slave Report Scan - Loaded 12 slave IDs in 0.000 s
2014-12-23 10:25:24: Slave Report Scan - Purged 0 report collections in 0.000 s
2014-12-23 10:25:24: Slave Report Scan - Done.
2014-12-23 10:25:24: Purging Old Limits
2014-12-23 10:25:24: Old Limit Scan - Loading machine limits
2014-12-23 10:25:24: Old Limit Scan - Found 2 machine limits in 0.000 s
2014-12-23 10:25:24: Old Limit Scan - Loading job IDs
2014-12-23 10:25:24: Old Limit Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:25:24: Old Limit Scan - Purged 0 machine limits in 0.000 s
2014-12-23 10:25:24: Old Limit Scan - Done.
2014-12-23 10:25:24: Purging Temporary Repository Files
2014-12-23 10:25:24: Temporary File Scan - Scanning for ‘connectReadWriteTest’ files
2014-12-23 10:25:24: Temporary File Scan - Deleted 0 temporary files in 0.000 s
2014-12-23 10:25:24: Temporary File Scan - Done.
2014-12-23 10:25:24: Purging Old Statistics
2014-12-23 10:25:24: Old Statistics - Skipping job statistics because the option to purge them is disabled in the Repository Options
2014-12-23 10:25:24: Old Statistics - Purging slave statistics that are older than Aug 25/14 10:25:24
2014-12-23 10:25:24: Old Statistics - Purged old slave statistics in 0.000 s
2014-12-23 10:25:24: Old Statistics - Purging repository statistics that are older than Aug 25/14 10:25:24
2014-12-23 10:25:24: Old Statistics - Purged old repository statistics in 0.000 s
2014-12-23 10:25:24: Purging Deleted Document Stubs From Database
2014-12-23 10:25:24: Deleted Document Stubs - Deleting stubs that are older than 3 days
2014-12-23 10:25:24: Deleted Document Stubs - Deleted 0 stubs in 0.000 s
2014-12-23 10:25:24: Triggering House Cleaning Events
2014-12-23 10:25:24: Performing repository repair
2014-12-23 10:25:24: Performing Orphaned Task Scan…
2014-12-23 10:25:24: Orphaned Task Scan - Loading rendering jobs
2014-12-23 10:25:24: Orphaned Task Scan - Loaded 1 rendering jobs in 0.000 s
2014-12-23 10:25:24: Orphaned Task Scan - Scanning for orphaned tasks
2014-12-23 10:25:24: Orphaned Task Scan - Cleaned up 1 orphaned tasks in 15.627 ms
2014-12-23 10:25:24: Orphaned Task Scan - Done.
2014-12-23 10:25:24: Performing Orphaned Limit Stub Scan…
2014-12-23 10:25:24: Orphaned Limit Stub Scan - Loading limits
2014-12-23 10:25:24: Orphaned Limit Stub Scan - Loaded 2 limits in 0.000 s
2014-12-23 10:25:24: Orphaned Limit Stub Scan - Loading slave states
2014-12-23 10:25:24: Orphaned Limit Stub Scan - Loaded 12 slave states in 0.000 s
2014-12-23 10:25:24: Orphaned Limit Stub Scan - Returned 0 orphaned limit stubs in 0.000 s
2014-12-23 10:25:24: Orphaned Limit Stub Scan - Done.
2014-12-23 10:25:24: Checking Available Database Connections
2014-12-23 10:25:24: Available Database Connections - Skipping because there are no Low Database Connection notification email addresses set in the Repository Options
2014-12-23 10:25:24: Performing Stalled Slave Scan…
2014-12-23 10:25:24: Stalled Slave Scan - Loading slave states
2014-12-23 10:25:24: Stalled Slave Scan - Loaded 12 slave states in 0.000 s
2014-12-23 10:25:24: Stalled Slave Scan - Scanning slave states
2014-12-23 10:25:24: Stalled Slave Scan - Cleaned up 0 stalled slaves in 0.000 s
2014-12-23 10:25:24: Stalled Slave Scan - Done.
2014-12-23 10:25:24: Performing Stalled Pulse Scan…
2014-12-23 10:25:24: Stalled Pulse Scan - Loading pulse states
2014-12-23 10:25:24: Stalled Pulse Scan - Loaded 1 pulse states in 0.000 s
2014-12-23 10:25:24: Stalled Pulse Scan - Scanning pulse states
2014-12-23 10:25:24: Stalled Pulse Scan - Cleaned up 0 stalled pulses in 0.000 s
2014-12-23 10:25:24: Stalled Pulse Scan - Done.
2014-12-23 10:25:24: Performing Stalled Balancer Scan…
2014-12-23 10:25:24: Stalled Balancer Scan - Loading balancer states
2014-12-23 10:25:24: Stalled Balancer Scan - Loaded 0 balancer states in 0.000 s
2014-12-23 10:25:24: Stalled Balancer Scan - Done.
2014-12-23 10:25:24: Triggering Repository Repair Events
2014-12-23 10:26:17: Connecting to Pulse log
2014-12-23 10:26:23: Performing Pending Job Scan…
2014-12-23 10:26:23: Pending Job Scan - Loading pending and active jobs
2014-12-23 10:26:23: Pending Job Scan - Loaded 1 pending and active jobs in 0.000 s
2014-12-23 10:26:23: Pending Job Scan - Scanning pending and active jobs
2014-12-23 10:26:23: Pending Job Scan - Released 0 pending jobs and 0 pending tasks in 0.000 s
2014-12-23 10:26:23: Pending Job Scan - Done.
2014-12-23 10:26:23: Processing Pending Job Events
2014-12-23 10:26:23: Pending Job Events - Checking for pending job events
2014-12-23 10:26:23: Pending Job Events - Processing 0 job events
2014-12-23 10:26:23: Pending Job Events - No more job events to process
2014-12-23 10:26:23: Pending Job Events - Done.
2014-12-23 10:26:23: Performing house cleaning
2014-12-23 10:26:23: Performing Job Cleanup Scan…
2014-12-23 10:26:23: Job Cleanup Scan - Loading completed jobs
2014-12-23 10:26:23: Job Cleanup Scan - Loaded 0 completed jobs in 0.000 s
2014-12-23 10:26:23: Job Cleanup Scan - Done.
2014-12-23 10:26:23: Purging Unsubmitted Jobs
2014-12-23 10:26:23: Unsubmitted Job Scan - Loading unsubmitted jobs
2014-12-23 10:26:23: Unsubmitted Job Scan - Loaded 0 unsubmitted jobs in 0.000 s
2014-12-23 10:26:23: Unsubmitted Job Scan - Done.
2014-12-23 10:26:23: Purging Deleted Jobs
2014-12-23 10:26:23: Deleted Job Scan - Loading deleted jobs
2014-12-23 10:26:23: Deleted Job Scan - Loaded 2 deleted jobs in 0.000 s
2014-12-23 10:26:23: Deleted Job Scan - Purged 0 deleted jobs in 0.000 s
2014-12-23 10:26:23: Deleted Job Scan - Done.
2014-12-23 10:26:23: Purging Old Job Auxiliary Files
2014-12-23 10:26:23: Auxiliary File Scan - Scanning for auxiliary directories
2014-12-23 10:26:23: Auxiliary File Scan - Found 4 auxiliary directories in 0.000 s
2014-12-23 10:26:23: Auxiliary File Scan - Loading job IDs
2014-12-23 10:26:23: Auxiliary File Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:26:23: Auxiliary File Scan - Purged 0 auxiliary folders in 0.000 s
2014-12-23 10:26:23: Auxiliary File Scan - Done.
2014-12-23 10:26:23: Purging Old Job Reports
2014-12-23 10:26:23: Job Report Scan - Loading job report collections
2014-12-23 10:26:23: Job Report Scan - Found 4 report collections in 0.000 s
2014-12-23 10:26:23: Job Report Scan - Loading job IDs
2014-12-23 10:26:23: Job Report Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:26:23: Job Report Scan - Purged 0 report collections in 0.000 s
2014-12-23 10:26:23: Job Report Scan - Purging old job report files
2014-12-23 10:26:23: Job Report Scan - Purged 0 report files in 0.000 s
2014-12-23 10:26:23: Job Report Scan - Done.
2014-12-23 10:26:23: Purging Obsolete Slaves
2014-12-23 10:26:23: Obsolete Slave Scan - Skipping because it is disabled in the Repository Options
2014-12-23 10:26:23: Purging Old Slave Reports
2014-12-23 10:26:23: Slave Report Scan - Loading slave report collections
2014-12-23 10:26:23: Slave Report Scan - Found 3 report collections in 0.000 s
2014-12-23 10:26:23: Slave Report Scan - Loading slave IDs
2014-12-23 10:26:23: Slave Report Scan - Loaded 12 slave IDs in 0.000 s
2014-12-23 10:26:23: Slave Report Scan - Purged 0 report collections in 0.000 s
2014-12-23 10:26:23: Slave Report Scan - Done.
2014-12-23 10:26:23: Purging Old Limits
2014-12-23 10:26:23: Old Limit Scan - Loading machine limits
2014-12-23 10:26:23: Old Limit Scan - Found 2 machine limits in 0.000 s
2014-12-23 10:26:23: Old Limit Scan - Loading job IDs
2014-12-23 10:26:23: Old Limit Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:26:23: Old Limit Scan - Purged 0 machine limits in 0.000 s
2014-12-23 10:26:23: Old Limit Scan - Done.
2014-12-23 10:26:23: Purging Temporary Repository Files
2014-12-23 10:26:23: Temporary File Scan - Scanning for ‘connectReadWriteTest’ files
2014-12-23 10:26:23: Temporary File Scan - Deleted 0 temporary files in 0.000 s
2014-12-23 10:26:23: Temporary File Scan - Done.
2014-12-23 10:26:23: Purging Old Statistics
2014-12-23 10:26:23: Old Statistics - Skipping job statistics because the option to purge them is disabled in the Repository Options
2014-12-23 10:26:23: Old Statistics - Purging slave statistics that are older than Aug 25/14 10:26:23
2014-12-23 10:26:23: Old Statistics - Purged old slave statistics in 0.000 s
2014-12-23 10:26:23: Old Statistics - Purging repository statistics that are older than Aug 25/14 10:26:23
2014-12-23 10:26:23: Old Statistics - Purged old repository statistics in 0.000 s
2014-12-23 10:26:23: Purging Deleted Document Stubs From Database
2014-12-23 10:26:23: Deleted Document Stubs - Deleting stubs that are older than 3 days
2014-12-23 10:26:23: Deleted Document Stubs - Deleted 0 stubs in 0.000 s
2014-12-23 10:26:23: Triggering House Cleaning Events
2014-12-23 10:26:24: Performing repository repair
2014-12-23 10:26:24: Performing Orphaned Task Scan…
2014-12-23 10:26:24: Orphaned Task Scan - Loading rendering jobs
2014-12-23 10:26:24: Orphaned Task Scan - Loaded 1 rendering jobs in 0.000 s
2014-12-23 10:26:24: Orphaned Task Scan - Scanning for orphaned tasks
2014-12-23 10:26:24: Orphaned Task Scan - Cleaned up 0 orphaned tasks in 0.000 s
2014-12-23 10:26:24: Orphaned Task Scan - Done.
2014-12-23 10:26:24: Performing Orphaned Limit Stub Scan…
2014-12-23 10:26:24: Orphaned Limit Stub Scan - Loading limits
2014-12-23 10:26:24: Orphaned Limit Stub Scan - Loaded 2 limits in 0.000 s
2014-12-23 10:26:24: Orphaned Limit Stub Scan - Loading slave states
2014-12-23 10:26:24: Orphaned Limit Stub Scan - Loaded 12 slave states in 0.000 s
2014-12-23 10:26:24: Orphaned Limit Stub Scan - Returned 0 orphaned limit stubs in 0.000 s
2014-12-23 10:26:24: Orphaned Limit Stub Scan - Done.
2014-12-23 10:26:24: Checking Available Database Connections
2014-12-23 10:26:24: Available Database Connections - Skipping because there are no Low Database Connection notification email addresses set in the Repository Options
2014-12-23 10:26:24: Performing Stalled Slave Scan…
2014-12-23 10:26:24: Stalled Slave Scan - Loading slave states
2014-12-23 10:26:24: Stalled Slave Scan - Loaded 12 slave states in 0.000 s
2014-12-23 10:26:24: Stalled Slave Scan - Scanning slave states
2014-12-23 10:26:24: Stalled Slave Scan - Cleaned up 0 stalled slaves in 0.000 s
2014-12-23 10:26:24: Stalled Slave Scan - Done.
2014-12-23 10:26:24: Performing Stalled Pulse Scan…
2014-12-23 10:26:24: Stalled Pulse Scan - Loading pulse states
2014-12-23 10:26:24: Stalled Pulse Scan - Loaded 1 pulse states in 0.000 s
2014-12-23 10:26:24: Stalled Pulse Scan - Scanning pulse states
2014-12-23 10:26:24: Stalled Pulse Scan - Cleaned up 0 stalled pulses in 0.000 s
2014-12-23 10:26:24: Stalled Pulse Scan - Done.
2014-12-23 10:26:24: Performing Stalled Balancer Scan…
2014-12-23 10:26:24: Stalled Balancer Scan - Loading balancer states
2014-12-23 10:26:24: Stalled Balancer Scan - Loaded 0 balancer states in 0.000 s
2014-12-23 10:26:24: Stalled Balancer Scan - Done.
2014-12-23 10:26:24: Triggering Repository Repair Events
2014-12-23 10:27:23: Performing Pending Job Scan…
2014-12-23 10:27:23: Pending Job Scan - Loading pending and active jobs
2014-12-23 10:27:23: Pending Job Scan - Loaded 1 pending and active jobs in 0.000 s
2014-12-23 10:27:23: Pending Job Scan - Scanning pending and active jobs
2014-12-23 10:27:23: Pending Job Scan - Released 0 pending jobs and 0 pending tasks in 0.000 s
2014-12-23 10:27:23: Pending Job Scan - Done.
2014-12-23 10:27:23: Processing Pending Job Events
2014-12-23 10:27:23: Pending Job Events - Checking for pending job events
2014-12-23 10:27:23: Pending Job Events - Processing 0 job events
2014-12-23 10:27:23: Pending Job Events - No more job events to process
2014-12-23 10:27:23: Pending Job Events - Done.
2014-12-23 10:27:23: Performing house cleaning
2014-12-23 10:27:23: Performing Job Cleanup Scan…
2014-12-23 10:27:23: Job Cleanup Scan - Loading completed jobs
2014-12-23 10:27:23: Job Cleanup Scan - Loaded 0 completed jobs in 0.000 s
2014-12-23 10:27:23: Job Cleanup Scan - Done.
2014-12-23 10:27:23: Purging Unsubmitted Jobs
2014-12-23 10:27:23: Unsubmitted Job Scan - Loading unsubmitted jobs
2014-12-23 10:27:23: Unsubmitted Job Scan - Loaded 0 unsubmitted jobs in 0.000 s
2014-12-23 10:27:23: Unsubmitted Job Scan - Done.
2014-12-23 10:27:23: Purging Deleted Jobs
2014-12-23 10:27:23: Deleted Job Scan - Loading deleted jobs
2014-12-23 10:27:23: Deleted Job Scan - Loaded 2 deleted jobs in 0.000 s
2014-12-23 10:27:23: Deleted Job Scan - Purged 0 deleted jobs in 0.000 s
2014-12-23 10:27:23: Deleted Job Scan - Done.
2014-12-23 10:27:23: Purging Old Job Auxiliary Files
2014-12-23 10:27:23: Auxiliary File Scan - Scanning for auxiliary directories
2014-12-23 10:27:23: Auxiliary File Scan - Found 4 auxiliary directories in 0.000 s
2014-12-23 10:27:23: Auxiliary File Scan - Loading job IDs
2014-12-23 10:27:23: Auxiliary File Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:27:23: Auxiliary File Scan - Purged 0 auxiliary folders in 0.000 s
2014-12-23 10:27:23: Auxiliary File Scan - Done.
2014-12-23 10:27:23: Purging Old Job Reports
2014-12-23 10:27:23: Job Report Scan - Loading job report collections
2014-12-23 10:27:23: Job Report Scan - Found 4 report collections in 0.000 s
2014-12-23 10:27:23: Job Report Scan - Loading job IDs
2014-12-23 10:27:23: Job Report Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:27:23: Job Report Scan - Purged 0 report collections in 0.000 s
2014-12-23 10:27:23: Job Report Scan - Purging old job report files
2014-12-23 10:27:23: Job Report Scan - Purged 0 report files in 0.000 s
2014-12-23 10:27:23: Job Report Scan - Done.
2014-12-23 10:27:23: Purging Obsolete Slaves
2014-12-23 10:27:23: Obsolete Slave Scan - Skipping because it is disabled in the Repository Options
2014-12-23 10:27:23: Purging Old Slave Reports
2014-12-23 10:27:23: Slave Report Scan - Loading slave report collections
2014-12-23 10:27:23: Slave Report Scan - Found 3 report collections in 0.000 s
2014-12-23 10:27:23: Slave Report Scan - Loading slave IDs
2014-12-23 10:27:23: Slave Report Scan - Loaded 12 slave IDs in 0.000 s
2014-12-23 10:27:23: Slave Report Scan - Purged 0 report collections in 0.000 s
2014-12-23 10:27:23: Slave Report Scan - Done.
2014-12-23 10:27:23: Purging Old Limits
2014-12-23 10:27:23: Old Limit Scan - Loading machine limits
2014-12-23 10:27:23: Old Limit Scan - Found 2 machine limits in 0.000 s
2014-12-23 10:27:23: Old Limit Scan - Loading job IDs
2014-12-23 10:27:23: Old Limit Scan - Loaded 4 job IDs in 0.000 s
2014-12-23 10:27:23: Old Limit Scan - Purged 0 machine limits in 0.000 s
2014-12-23 10:27:23: Old Limit Scan - Done.
2014-12-23 10:27:23: Purging Temporary Repository Files
2014-12-23 10:27:23: Temporary File Scan - Scanning for ‘connectReadWriteTest’ files
2014-12-23 10:27:23: Temporary File Scan - Deleted 0 temporary files in 0.000 s
2014-12-23 10:27:23: Temporary File Scan - Done.
2014-12-23 10:27:23: Purging Old Statistics
2014-12-23 10:27:23: Old Statistics - Skipping job statistics because the option to purge them is disabled in the Repository Options
2014-12-23 10:27:23: Old Statistics - Purging slave statistics that are older than Aug 25/14 10:27:23
2014-12-23 10:27:23: Old Statistics - Purged old slave statistics in 0.000 s
2014-12-23 10:27:23: Old Statistics - Purging repository statistics that are older than Aug 25/14 10:27:23
2014-12-23 10:27:23: Old Statistics - Purged old repository statistics in 0.000 s
2014-12-23 10:27:23: Purging Deleted Document Stubs From Database
2014-12-23 10:27:23: Deleted Document Stubs - Deleting stubs that are older than 3 days
2014-12-23 10:27:23: Deleted Document Stubs - Deleted 0 stubs in 0.000 s
2014-12-23 10:27:23: Triggering House Cleaning Events
2014-12-23 10:27:23: Performing repository repair
2014-12-23 10:27:23: Performing Orphaned Task Scan…
2014-12-23 10:27:23: Orphaned Task Scan - Loading rendering jobs
2014-12-23 10:27:23: Orphaned Task Scan - Loaded 1 rendering jobs in 0.000 s
2014-12-23 10:27:23: Orphaned Task Scan - Scanning for orphaned tasks
2014-12-23 10:27:23: Orphaned Task Scan - Cleaned up 0 orphaned tasks in 0.000 s
2014-12-23 10:27:23: Orphaned Task Scan - Done.
2014-12-23 10:27:23: Performing Orphaned Limit Stub Scan…
2014-12-23 10:27:23: Orphaned Limit Stub Scan - Loading limits
2014-12-23 10:27:23: Orphaned Limit Stub Scan - Loaded 2 limits in 0.000 s
2014-12-23 10:27:23: Orphaned Limit Stub Scan - Loading slave states
2014-12-23 10:27:23: Orphaned Limit Stub Scan - Loaded 12 slave states in 0.000 s
2014-12-23 10:27:23: Orphaned Limit Stub Scan - Returned 0 orphaned limit stubs in 0.000 s
2014-12-23 10:27:23: Orphaned Limit Stub Scan - Done.
2014-12-23 10:27:23: Checking Available Database Connections
2014-12-23 10:27:23: Available Database Connections - Skipping because there are no Low Database Connection notification email addresses set in the Repository Options
2014-12-23 10:27:23: Performing Stalled Slave Scan…
2014-12-23 10:27:23: Stalled Slave Scan - Loading slave states
2014-12-23 10:27:23: Stalled Slave Scan - Loaded 12 slave states in 0.000 s
2014-12-23 10:27:23: Stalled Slave Scan - Scanning slave states
2014-12-23 10:27:23: Stalled Slave Scan - Cleaned up 0 stalled slaves in 0.000 s
2014-12-23 10:27:23: Stalled Slave Scan - Done.
2014-12-23 10:27:23: Performing Stalled Pulse Scan…
2014-12-23 10:27:23: Stalled Pulse Scan - Loading pulse states
2014-12-23 10:27:23: Stalled Pulse Scan - Loaded 1 pulse states in 0.000 s
2014-12-23 10:27:23: Stalled Pulse Scan - Scanning pulse states
2014-12-23 10:27:23: Stalled Pulse Scan - Cleaned up 0 stalled pulses in 0.000 s
2014-12-23 10:27:23: Stalled Pulse Scan - Done.
2014-12-23 10:27:23: Performing Stalled Balancer Scan…
2014-12-23 10:27:23: Stalled Balancer Scan - Loading balancer states
2014-12-23 10:27:23: Stalled Balancer Scan - Loaded 0 balancer states in 0.000 s
2014-12-23 10:27:23: Stalled Balancer Scan - Done.
2014-12-23 10:27:23: Triggering Repository Repair Events
2014-12-23 10:27:41: Connecting to Pulse log

#35

backburner with stable and daily build of corona doing strip render works fine

#36

Thanks for reporting this! Looks like this is a regression in 7.0.1.1, which we’ll get fixed first thing tomorrow morning. For now, the best thing to do would be to downgrade to 7.0.0.54 so that you can get your renders done.

Sorry for the inconvenience!

Cheers,
Ryan

#37

7.0.1.2 has just been uploaded, and you can download it using the same link that our sales team sent you to download 7.0.1.1.

We’ve tested this new build and confirmed that this bug has been fixed.

Cheers,
Ryan

#38

Hi guys
7.0.1.2 seems to resolve this issue
we did do a clean install of repository and mongo and autoupdated slaves. ( a few would auto update)
all systems on 7.0.1.2R and no dropouts yet

thanks

#39

Glad to hear it! Thanks for letting us know!

Cheers,
Ryan

Privacy | Site terms | Cookie preferences