AWS Thinkbox Discussion Forums

deadline 3.0 problems

Hi Ryan

We continue testing deadline, now we installed in all our network, but are experience some problems. Some of then I think is lack of knowledge from us and other are from deadline

  • We have several machines that give the error attached, do you know what is this?

  • One of the reasons that we are trying deadline is that we are tired of the stupid things that backburner do on the network. One of those thinks is assign several jobs to the same machine at the same time. On the image that I send you from our monitor we have that problem on st23. That machine is rendering “CapitalLand_RafflesCity_Plano_03b_Promenade_Geral_Matte_01” but it is also assign to another job for more then 50 min!
    By the way, do you have on your site the colors of the jobs on the monitor explained? What means the brown color? And what is the difference between a slave that is offline and a slave that is stalled?

  • We are using vray 1.5 sp2 on max2009 design 64bits and we are experience that in some cases the render done on deadline is very different from the one done on max.

  • We already tested deadline 2.7 in the past, and I remember that you had a problem with the tile render appearing on the viewport of max. That problem still persists, when we turn the tile render on the viewport the deadline script gives an error. Same thing with the region render.

Best regards,
Mário
St23_00198_20080924095643.errorReport.xml (4.34 KB)

Hi Mario,

See the last error message here:
franticfilms.com/software/su … ssages.php

Can you send us the log for this slave? If you’re on XP, the logs can be found in:
C:\Documents and Settings\All Users\Application Data\Frantic Films\Deadline\logs

We need a log that contains the output from when the st23 machine moved from rendering the previous job to rendering CapitalLand_RafflesCity_Plano_03b_Promenade_Geral_Matte_01. If this transition occurred over more than one log, please send us all logs that are involved. Just sort the logs by modified date/time to get them in the proper order. We’re looking to see what the slave did when it dropped its previous job.

This isn’t explicitly documented, but we should update our docs to explain this. Basically, as a job produces more and more errors, it goes from green to brown to red. The darker the job, the more errors there are. A stalled slave is a slave that didn’t shutdown cleanly due to a crash. When a slave shuts down normally, it appears as offline.

Can you post a simple test scene (just some geometry and a camera) that reproduces this problem? Also, can you provide an explanation of what the differences are if they aren’t immediately obvious? If possible, can you narrow down which features/settings result in different renders?

We have yet to figure out this crashing problem, as we’ve had a tough time reproducing it. In 3.0, the gizmos are no longer shown automatically when enabling tile or region rendering because of this problem. One workaround that has been known to help is to increase the heap size for maxscript from 64mb to 128mb in one of the submission script files. Go to \your\repository\submission\3dsmax and open the file “SubmitMaxToDeadline_Functions.ms”. Find this code:

--Increase heal size to 64 MB just in case. if heapSize < 67108864 do heapSize = 67108864

And change it to this:

--Increase heal size to 64 MB just in case. if heapSize < 134217728 do heapSize = 134217728

Cheers,

  • Ryan

Hi Ryan

Thanks for your quick response.
We manage to fix the problem with the error that I send you, it was the max2009 sp1 that was not installed on some machines.

Regarding the problem with the st23 there are several logs and to me they seen like chinese! :wink: But is strange that the size varies from 1kb to 519Mb is this normal?
I send you a rar of several logs on the time that I think the error occurred.

Meanwhile there are another problem that is annoying, we have several jobs that are rendering well and suddenly they stay with 0% of a frame completed forever.
When that happens if I change the time limit to render, suspend the job or delete the job on the monitor the slaves that where rendering that job stay on that job forever… the only way is to end the process station by station (and the remote control on the monitor does not work in this case)
I send you two print screens with 5 minutes apart and the log files of that station on the time that the problem occurred. I hope you could help me!

Best regards,
Mário


Logs St22.rar (608 KB)
logs St23.rar (1.48 MB)

Hmm, there isn’t anything in the st23 logs that would indicate why it dropped one job and moved to the other. Out of curiosity, what is the operating system of the machine you have the repository installed on?

Also, I’m not sure what would cause the render to stall like that (might be vray thing). In any case, Deadline should stop rendering when a task has been requeued. I can’t reproduce this problem, and we’ve been using Deadline 3.0, vray, and 3dsmax in heavy production and haven’t seen anything like this.

Cheers,

  • Ryan

Hi Ryan

Our repository as win2003 server r2 standart edition 32Bits.

The problem with the render stoping forever is happening in many jobs and in all the machines, and if you could not give us a solution we (unfortunately) have to go back to the old backburner.
Are you using xrefs and vrayproxs? We use those a lot and I think is on those jobs that deadline is stoping. We have several renders on mentalray that are rendering with no problems

Best regards
Mário

You could try using the 3ds Command plugin for your vray jobs to see if that helps. This plugin uses 3dsmaxcmd.exe to do the rendering instead of controlling 3dsmax using our Lightning plugin. The 3ds Command plugin is the second plugin from the top:
franticfilms.com/software/su … -ins_Guide

Cheers,

  • Ryan

Hi Ryan

Ok, I will try it! The 3ds command plugin is sending the jobs inside the monitor? Do I have to resend the jobs that are already on the monitor?
Meanwhile I opened pulse on the repository and (I dont know if it is a coinsidence) the crashs stoped, lets see if it stays like that.

I register another error on several machines, could you help me on this
St24_00000_20080924181914.zip (1.34 KB)

The 3ds Command plugin is completely separate from the 3dsmax plugin. It has a different integrated submitter for 3dsmax, and it has a different submitter in the Monitor as well. Jobs already in the queue using the 3dsmax plugin need to be resubmitted. You can tell if a job in the Monitor is using the 3ds Command plugin because it will have the generic Autodesk icon (blue box with the letter ‘A’) instead of the 3dsmax icon.

For this new error, check if there are any other max processes running on the machine in the Task Manager. If there are, kill them before continuing to render on those machines. Maybe try restarting those machines and see if the problem persists.

Cheers,

  • Ryan

Hi Ryan

We resend the animations using the 3ds Command plugin and it is much more stable, though we lose some of the fixtures of deadline like changing vray settings on the monitor, I think for now this is our best solution…

With the 3dsCmd script, comes new problems…
The first one and more problematic is that with this script max does not respect the gamma correction of the file and render the images without gamma (we use 2.2 gamma) I send you a image comparing the two renders.
When we start to use the 3dscmd, sometimes the deadline slave gives a windows error and shuts down but the frame stays forever on the monitor, I send you a error report of that and a print screen. I send you also a error report of another new error that appeared

Thanks,
Mário



Errors 3dsCmd.zip (2.28 KB)

For the error you’re getting with 3ds Command, I’ve attached a patch that should allow Deadline to ignore this popup. Just unzip the attached file to \your\repository\plugins\3dsCmd. After applying the patch, let us know if you continue to have problems rendering.

The gamma problem has come up with our original 3dsmax plugin as well, and we had thought that we were doing something wrong. But because it’s happening with 3dsmax’s command line renderer as well, we’re starting to wonder if it’s a 3dsmax thing and not a Deadline thing.

In about a week or two, we’re going to be releasing a maintenance release of Deadline 3.0 that will fix some crashing issues with the slave (among other issues that have come up). More than likely it will fix the crash you mentioned.

Cheers,

Hi Ryan

Thanks for your response.
I tryed the patch and it started to give a new error. I send you a error log from st15

Meanwhile there was another new error, before the new patch, error log from st11

Thanks again,
Mário
St11_00139_20080925105513.rar (1.44 KB)
St15_00001_20080925144717.rar (1.15 KB)

Ack, sorry about the patch - there was some bad syntax in the plugin file. I’ve fixed it and attached a new patch (same installation instructions as before).

For the other error, Deadline just seems to be the messenger in this case. With the 3ds Command plugin, Deadline just launches the 3dsmax command line renderer and waits for it to complete (it doesn’t have any control over the actual rendering process). This is the information being provided by 3dsmax:

I know, it’s not very helpful. What would be interesting is to see if 3dsmaxcmd.exe reports the same error when rendering outside of Deadline. In the task log, Deadline prints out the render executable and render arguments. You should try opening a command prompt on this slave machine and try rendering using the same executable and options that Deadline is using. Note that you might have to copy the max scene to the slave first and change the path to it in the command line arguments. If you still see the same error, then we know it’s a max issue and not a Deadline issue.

Cheers,

Hi Ryan

It is me again! :wink:

The patch is working!
Meanwhile there is another error that I think if regarding submiting the job in tiles, can you help me?

Thanks,
Mário
St11_00000_20080925160729.zip (1.4 KB)

And it’s me again, with another patch! :smiley:

Same installation instructions as before. Let us know if you still have problems!

Cheers,

Hi Ryan

I can’t tell you if the patch worked, because now, is renderirng but it still gives errors I think a diferent one, see attach.
By the way to install this patchs I can copy the new file over the old one, but do I have to restart the slaves or something?

Thanks,
Mário
new errors.rar (2.83 KB)

Ok, here’s another patch. I’ve tested this out here and it seems to perform the stitching just fine now. Give it a try and let us know if you have better luck. Just copy the patch file over the old one. You shouldn’t have to restart the slave.

Cheers,

Thanks again for the new patch, I didn´t tested yet, but this is starting to be chaotic!
The slaves are still crashing on the network, and now it seems that the problem is worst, and when this happens the windows error keeps the slave opened and the frame assigned forever, if you are not controlling that, eventually all my slaves will be stoped!
Last night I stayed until 4 in the morning controlling renders, because I have deadlines to keep.
Today is the same thing, I have 10 animations that have to be render tomorrow and I can’t trust on deadline because of the constant errors. I have another job, with over 30 animations, that has to be render on monday and I dont want to stay on work on the weekend.
My point is, we like very much the many fixtures of the deadline, but I’m starting to think that the backburner, with all is problem, is more stable then the deadline.
You told me that you are preparing a new version of deadline, can´t you send me a beta for us to test?

Mario

I should be able to prepare a beta today, and I’ll email you the direct link as soon as it’s available. This beta will include the fixes to the 3ds Command plugin that we’ve sorted out in this thread. The slave application itself should also be a lot more stable with this beta.

We apologize for the inconveniences Deadline has caused you this week. A lot changed between Deadline 2.7 and 3.0, and despite a rigorous beta testing program, some issues still found their way into the final release. We appreciate your help and patience getting some of these issues worked out, and hopefully this new beta will allow you to get some sleep this weekend. :slight_smile:

Cheers,

  • Ryan

Hi Ryan

Thanks again for the new beta.
Before installing it, we made a windows update in all our machines that included .net 3.0 sp1, we also made a windows update on the windows 2003 server that as the repository and that also included a .net update (this time .net 2.0 sp1)
After that we installed your new beta and so far, so good. At least we didn´t had, so far, the deadline crash that keeps the slave on the same frame forever!
The only error that we had so far, I send you in attach (I think I already send you one like this)

Nevertheless, you have to see the gamma correction problem, at least for us that is a very important feature. We didn’t had that problem with the other deadline script, only start to have gamma correction problems after we start to use the 3dsCmd

I think I can go home now, it is almost 1:30 in the morning here!!!

Thanks
Mario
same_error_on_diferent_machines.zip (4.35 KB)

Hi Ryan

Here I’m again!
The renders worked fine during the night! :slight_smile: the only error that we are registering are on jobs send in “split render”, I send you some new error reports that I think are diferent from the ones that I already send you.

Thanks,
Mario
ErrorStrips.zip (2.82 KB)

Privacy | Site terms | Cookie preferences