[Bug] event initialization can hang render

Deadline
1

We have found a few slaves hanging due to an event plugin being added then removed from the repo.
Midway through the render, we got this error:

2018-02-26 14:09:46: Detected new event plugin DraftEventPlugin in the Repository, it will now be initialized 2018-02-26 14:09:46: Events plugin 'DraftEventPlugin' could not be loaded from the repository because: Error executing event plugin script "\\inferno2.scanlinevfxla.com\deadline\repository10\events\DraftEventPlugin\DraftEventPlugin.py": Value cannot be null. 2018-02-26 14:09:46: Parameter name: str (Deadline.Events.DeadlineEventPluginException) 2018-02-26 14:14:46: Detected new event plugin DraftEventPlugin in the Repository, it will now be initialized 2018-02-26 14:14:46: Events plugin 'DraftEventPlugin' could not be loaded from the repository because: Error executing event plugin script "\\inferno2.scanlinevfxla.com\deadline\repository10\events\DraftEventPlugin\DraftEventPlugin.py": Value cannot be null. 2018-02-26 14:14:46: Parameter name: str (Deadline.Events.DeadlineEventPluginException) 2018-02-26 14:19:46: Detected new event plugin DraftEventPlugin in the Repository, it will now be initialized 2018-02-26 14:19:46: Events plugin 'DraftEventPlugin' could not be loaded from the repository because: Error executing event plugin script "\\inferno2.scanlinevfxla.com\deadline\repository10\events\DraftEventPlugin\DraftEventPlugin.py": Value cannot be null. 2018-02-26 14:19:46: Parameter name: str (Deadline.Events.DeadlineEventPluginException) 2018-02-26 14:24:46: Detected new event plugin DraftEventPlugin in the Repository, it will now be initialized 2018-02-26 14:24:46: Events plugin 'DraftEventPlugin' could not be loaded from the repository because: Error executing event plugin script "\\inferno2.scanlinevfxla.com\deadline\repository10\events\DraftEventPlugin\DraftEventPlugin.py": Value cannot be null. 2018-02-26 14:24:46: Parameter name: str (Deadline.Events.DeadlineEventPluginException) 2018-02-26 14:29:47: Detected new event plugin DraftEventPlugin in the Repository, it will now be initialized

The tasks have been hanging since. It seems that it hangs the render process and goes into an infinite loop.

2

Running some tests over here. I enabled the “FontSync” plugin, re-started the Slave so it would theoretically load it, then renamed the plugin in the Repo. I have a simple ping job that’ll run for 60 seconds per task. We’ll see what I noticed throughout the day.

Anything I might be missing here?

3

Well it happened on a few machines out of several thousand here, so not sure if its easy to repro, but maybe worth looking at the robustness of the code popping those messages… Even after the event files were readded, the tasks just hang.

4

Good point. I’ll talk to Jon and the guys about this and see what the best attack plan is here.

If I’m honest, I wonder if the scheduler thread crashed which might be unrelated to the plugins unloading… If that’s the case, the Slave would stop picking up work. It might also be that the scheduler thread has some inter-play with the event unloading.

Were those lines the only parts in the log?

5

Yeah the previous line was still from the task, then that stuff loops for the next 17 hours (when we noticed…)

6

Well, the thought about the loop on the plugin is that we don’t expect the death of the scheduler thread to affect it, so that repeating is a bit odd. The fact that the Slave didn’t pick up work is more more important here.

If you guys are willing to try and break things again, a memory dump for this problem would be nice to have as well.