Potential memory leak in Deadline monitor 6.1

Deadline
1

Hello,

Recently we upgraded our render farm (approx 60 nodes) to Deadline 6.1. I typically have my machine turned on 24/7 so that I can remote in if necessary and noticed that recently, after about a day or two of leaving Deadline Monitor 6.1 open, the software will slowly increase the memory usage from ~150mb to 2gb, sometimes 4gb if left open long enough. This causes the software to slow down incredibly and also slows down my machine. Is this a known issue?

Thanks

Deadline Version: 6.1.0.54665 R
FranticX Version: 2.0.0.54634 R

2

Hello,

This is something we are currently testing, and hope to have some new information in time, though it could take time to find a solution.

3

Hey Brett,

I’ve been looking into this memory leak issue, and I just wanted to follow up on a couple things, and get a bit more info in order to pin this down a bit better.

First of all, have you guys been using Deadline’s Cloud integration (ie, for the Cloud Panel in the Monitor)? If you’re not sure if it’s set up, you can open up ‘Configure Cloud Providers’ under the ‘Tools’ menu (by default, you need to be Super User to do this), and check if any of the listed providers are enabled.

Next, you mentioned you have 60 Slaves, but what’s the typical Job count for you guys (all statues, not just queued)? Similar question, but how many Jobs do Slaves typically churn through in a day when you’d see this come up? Do you put your computer to sleep or anything like that when you leave for the day?

Which OS are you running? If you’re running Windows, would you be willing to send us a memory dump of the Monitor?

Sorry for the twenty questions, I’m just trying to get a better picture of what the typical load on the Monitor should be for you guys, and if there’s any kind of special scenario stuff I should take into account!

Cheers,
Jon

4

Hey Jon,

  • We have not been using the Cloud integration.

  • Here’s a couple graphs to show this week’s render farm activity, hopefully this answers your second question regarding job count and active render count: Jobs: http://i.imgur.com/bA8JtLr.png ; Nodes: http://i.imgur.com/XqgCZzY.png . I use TeamViewer to remote in so I keep my computer logged in and my monitors turned off. I don’t believe my machine has a sleep function enabled.

  • I am using Windows.

5

Thanks Brett, that’s super helpful, gives me a good idea of what your typical Monitor would look like and what kind of RAM it actually should be using (obviously nowhere near what you’re reporting!).

I think at this point, since I’m having trouble replicating the leak, it’d be most helpful if you’d be able to get us a memory dump of the Monitor when it’s at a high amount of memory usage. To do this, once the Monitor’s using up a bunch of RAM just open up the Windows Task Manager, right-click on the Monitor’s task, and select ‘Create Dump File’. If you could zip up the file it creates, and send it to us that’d be super helpful! It’d also be great if you could send us the following DLL from that computer alongside it: “C:\Windows\Microsoft.NET\Framework64\v4.0.30319\SOS.dll”. Not super necessary, but it helps to be able to look at the dump correctly.

Obviously the dump will probably be a bit too big to upload to the forum or send through email, so let me know if you don’t have a good way to get that to us, and I’ll try to sort something out to let you upload it to our Box server.

Cheers,
Jon

6

Before I submit a memory dump, I just noticed this in the Deadline Console:

2014-03-10 17:35:13: WARNING: Function "CleanupDeadlineEventListener" is not defined in the plugin script file \\deadlinela\Deadline_Repository_6x\events\Draft\Draft.py. This will likely lead to a memory leak. 2014-03-10 17:35:13: [stack trace (maximumDepth=4)] FranticX.Diagnostics.Trace2.WriteStack line 0 2014-03-10 17:35:13: Deadline.Events.DeadlineEventPlugin.Dispose line 0 2014-03-10 17:35:13: Deadline.Events.DeadlineEventManager.LoadEventListeners line 0 2014-03-10 17:35:13: Deadline.Events.DeadlineEventManager.OnJobsDeleted line 0 2014-03-10 17:35:13: WARNING: Function "CleanupDeadlineEventListener" is not defined in the plugin script file \\deadlinela\Deadline_Repository_6x\events\Shotgun\Shotgun.py. This will likely lead to a memory leak. 2014-03-10 17:35:13: [stack trace (maximumDepth=4)] FranticX.Diagnostics.Trace2.WriteStack line 0 2014-03-10 17:35:13: Deadline.Events.DeadlineEventPlugin.Dispose line 0 2014-03-10 17:35:13: Deadline.Events.DeadlineEventManager.LoadEventListeners line 0 2014-03-10 17:35:13: Deadline.Events.DeadlineEventManager.OnJobsDeleted line 0

7

Has there been any updates on this. I think we’re running into the same issue but we’re only on 6.0.0.51561.

EDIT Ugh… Never mind it looks like it was a Max memory leak issue…

8

We have been experiencing this problem too. I’ve provided links to the dump file and sos.dll. Please post when you’ve downloaded them so I can flush them from my dropbox. Thanks for looking into this.

dl.dropboxusercontent.com/u/673 … onitor.zip
dl.dropboxusercontent.com/u/6739167/SOS.zip

EDIT: I just discovered one of our workstations deadlinemonitor.exe process was running at 14GB usage. I consider this an urgent issue and hope to see a priority patch soon. Until then, I’ll try to roll out a group policy to kill the process each night.

9

Thanks for posting these! I just finished grabbing them so you should be able to clear those off your dropbox now.

If you’re part of the Deadline 6.2 beta, the build we posted earlier this week has a fix for a leak that we found in the Monitor (I’m not sure if it’s the same one you’re experiencing yet). If you’re not on the beta, I would definitely recommend talking to sales to get on it. Even if it’s a separate leak from what we’ve already fixed, we’ve also added a debug command to the Monitor that should help us figure out where your particular leak is coming from.

Anyways, in the meantime I’ll definitely have a look at your dump file and see what I can figure out.