Machine startup is not work on 10.0.10.4

Makoto_Chiba · February 3, 2018, 7:06pm

After I update to 10.0.10.4, Machine startup is not working.

Idle shutdown is works.
And if I do remote control through deadline monitor and send Start Machine command, it works.
So, only auto startup setting is not works.
Is this bug? or should I check something???

thanks

-Makoto

eamsler · February 5, 2018, 7:43pm

This does sound familiar. The Pulse log will be controlling this.

The two thoughts I have:

Was this upgrade done using the ‘import settings’ wizard in the tools menu?
Does the Pulse log show anything?

We made changes to the machine wake-up to dramatically increase the rate that the Slaves were asked to start. I believe we changed the wake-up interval here:

Can you see if that delay is set to 500? Also, if it is, try setting it to 501 and see if that enables things.

Makoto_Chiba · February 6, 2018, 6:38pm

hi Edwin,

I tried to check Pulse log. But it has nothing special.
I simply updated to current version from older version(10.0.7.4). And major update(from DL8 to DL10) was clean install. I use different folder for database and repo.

And I found important thing.
If I submit job from “fusion”, auto machine startup is works.
But if I submit job from “3dsmax”, it not work. Even xmesh-job and something from 3dsmax are same.
These are happening even if I use same group for both jobs(I don’t use pool now).
I don’t check with the other app. But this is definitely happens.

thanks

-Makoto

eamsler · February 7, 2018, 2:54pm

I should have mentioned this on my last response, but is Pulse set to use verbose logging? I’m expecting it to be very talkative when it’s choosing which machines to start up.

Other impacts may be limits… Actually, we implemented a new feature in SP10 to help out here. Can you try the “Find Render Candidate” option in the job right-click menu? It’s a much more advanced version of the Job Candidate option in the Slave panel. If they are not allowed to render, Pulse would ignore them and that should show up in that tool.

Makoto_Chiba · February 7, 2018, 7:23pm

Edwin,

It has no machine limit, because if I startup PCs manually, render will be started.
And I checked “Find Render Candidate”, but of course PCs are in the “Able to render” list.
I said again, only Machine auto startup is not works with some jobs(currently 3dsmax job is always).

thanks

-Makoto

eamsler · February 8, 2018, 5:58pm

Thanks for that additional work Makoto.

We’ve raised this problem internally. We’re not yet sure where to begin, so sending the verbose Pulse log would be the next step. For privacy reasons, feel free to e-mail that to support@thinkboxsoftware.com.

I believe I have a second client with this problem in North America, so I will chase them for a remote session as well. Would you be open to one as well?

Makoto_Chiba · February 8, 2018, 6:43pm

Edwin,

OK, I sent e-mail with log file to support.
thanks!

-Makoto

evanwong · February 8, 2018, 10:06pm

Hi,

Just wanted to add that I’m having this same issue as well. We recently moved from Deadline 8 to Deadline 10 (currently on 10.0.9.2) and this feature hasn’t worked for us since we upgraded.

The auto-shutdown feature seems to work perfectly but when we set off a 3dsmax job the slaves won’t turn back on. The pulse log says “slave is not valid because of job’s machine limit” but if I manually turn on the slaves then they’ll start rendering fine.

After reading Makoto’s post I tried a VraySpawner job and the slaves powered up fine so it seems that it’s just the 3dsmax jobs that won’t turn on slaves.

Cheers,
Evan

Makoto_Chiba · February 9, 2018, 7:14am

Evan,

Thank you very much!
This is not only my case, so, I think this is bug of repository or SMTD.

I actually having some more problem with Deadline10.
I hope we get stable with DL10 soon.

thanks!

-Makoto

eamsler · February 9, 2018, 4:11pm

Actually, I bet it has something to do with the limit change we made in 10.0 SP9:

Now, Pulse does not share the dequeuing logic with the Slave so it is very likely they have diverged. I will look into this!

The information from the logs and the fact that the Slave begins have been a huge help here. Thanks to both of you!

eamsler · February 9, 2018, 4:43pm

Update over here. The dev team tells me those changes were cosmetic which is good.

Currently I’m going to experiment with purposely putting whitespace into my limits at submission and see if that will break. If others can enable verbose logging and send it my way, that would be helpful as well. To enable it:

Also, screenshots of the configurations of the limits which are failing would be very helpful here as well.

eamsler · February 9, 2018, 5:17pm

Update valid of it’s own post!:

Great work Makoto on noticing that this only affected 3DS Max. It seems that if the LimitGroup property of the job info file ends with a space, it will prevent Pulse from starting the Slave with the exact error. I am able to reproduce this exactly.

I will try to find out what changed in the SMTD so you can begin working now, and we can fix it properly in core after that.

Makoto_Chiba · February 11, 2018, 1:28am

Edwin,

Thank you very much.
I hope this problem will be fixed soon.
And also the other frustrated problems with DL10…

thanks!

eamsler · February 12, 2018, 6:50pm

For this problem specifically, we are trying to make the change in the SMTD first, then adjusting changes in the core code from there. I’m actively working with the development team on this.

For the remaining issues outside of this thread, we are working through them as well. Thanks for raising them!

eamsler · February 12, 2018, 8:44pm

I think we have a fix here, so I’ll need some testing. It’s easier than I’d expected to work around.

It seems that any non-existing limit group (spaces or not) that is added to a job will cause machine wake-up ignore that job.

Now, the SMTD has a new feature that automatically adds some limits. My guess is that the workaround is to uncheck all of these in the “limits” rollout:

Can you guys try that out and report back?

evanwong · February 13, 2018, 1:35am

Hi Edwin,

I tried submitting a new job with all the limit checkboxes turned off but it still didn’t work. I did notice though that when I went into the Job Properties max was still listed in the Required Limits. I tried a few different combos of check boxes, all off, all on and some in between and max was always listed under Required Limits.

When I moved it to the left column under Limits List (which makes it disappear btw) it worked! The machines all started up as usual.

To be honest, I’m not sure what these limits do. Is there a way to submit a job without having max listed under the Required Limits or is that a bug in SMTD?

carp · February 13, 2018, 11:53am

We have the same problem at out side, created a own tread. But am following this one with intressets to.

carp · February 13, 2018, 12:39pm

+1

mepp · February 13, 2018, 5:23pm

Hey guys,

I’ve got a quick fix on the integration-side until the changes make their way into core.

Make backups and copy the two included files into DeadlineRepository10/submission/3dsmax/Main
This will turn all of those auto-limits off by default, and fix a few issues with setting them (ie. the ‘max’ limit not turning off). Due to sticky settings, the options might be enabled the next time you open the dialog, but any new user, or subsequent dialog will have them off. Let me know how it goes.

Cheers
3dsmax_main.zip (188 KB)

Nick_Weeden · February 14, 2018, 10:08am

We had this issue, I just added max and vray limits with unlimited resources. Stumped me for a while as I couldn’t work out why it was suddenly applying a limit!

Nick