Deadline 4.0 Ubuntu 9.10 *WIP* - Need Dev Support Plz

Making an effort to start looking into some howto’s for 3D operators specifically for Ubuntu gnome desktop as a platform of choice.

This HowTo is for Ubuntu 9.10 / Deadline V4.0

I’m posting this up as I’m running into some weird results and I’m hoping someone might be able to go through this and point out something that might be missing in an effort to rectify some of the issues with Deadline in Ubuntu 9.10. From there I can fix up the walkthrough and others can use this as a point of reference for setting up Deadline 4.0 on Ubuntu 9.10.
Perhaps someone from the Dev team can take a nosey through this to see if there are any resolutions to the Known Issues section, or a way I can get some info to you for investigation :slight_smile: pretty please!
__________________________________________________________________________________________________________________
[size=125]KNOWN ISSUES:[/size]

  • Window Resizing doesn’t work
    -work around is to hack the py found /DeadlineRepository/scripts/Submission/MayaSubmission/MayaSubmission.py

[code]
dialogWidth = 900

labelWidth = 200

dialogHeight=800[/code]

  • mouse gets caught in the slaves section of the monitor in drag select mode.
  • can’t right click and re-Que a task which says it’s `rendering’ even though the slave has moved onto another task.
  • slave window flicks to all slaves currently active as idle. Need to restart monitor to see them as rendering again.
  • slave timeouts fail to re-que the task properly
  • can crash the monitor by selecting the Tools/ViewRemoteCommandWindow and clicking on the Clear Completed button
  • Failing a task = no chance to re-Que it.
  • All kinds of problems re-Queuing / managing tasks in progress…
  • All kinds of refreshing / slave information down the bottom, to the point of not trusting that info anymore at all…
  • If ya have to restart a slave during a render, the current render will not pickup the slave again for rendering no matter what…

__________________________________________________________________________________________________________________
[size=150]Step 1: Download Deadline[/size]
Download the DeadlineClient and DeadlineRepository installers :stuck_out_tongue:

[size=150]Step 2: Installing Mono[/size]
[size=100]2.1: Synaptic package install of 2.4.2-1[/size]
[list]In Ubuntu 9.10 from the synaptic package manager you can install 2.4.2-1 directly search for mono and install mono-utils[/list:u]

[size=100]2.2: Manual Compile of 2.6.1[/size]
[list]* this does fix the menu pull down issues somewhat that exists with the synaptic mono install
*Also prob a good idea to check the synaptic manager for any mono installs and any libgdiplus installs and remove them.
For the die hard compilers 2.6.1 can be obtained from mono-2.6.1.tar.bz2 and libgdiplus-2.6.tar.bz2or libgdiplus-2.6.2.tar.bz2 [/list:u]
NOTE I have noticed these builds seem way less stable than the synaptic 2.4.2-1 install …

Requirements to compile:

$sudo apt-get install build-essential libpng12-dev pkg-config libglib2.0-dev bison libcairo2-dev libungif4-dev libjpeg62-dev libtiff4-dev gettext  libgif-dev

Go to where you downloaded the tars to.
[list]$tar -xf mono-2.6.1.tar.bz2
$tar -xf libgdiplus-2.6.tar.bz2[/list:u]

[size=100]2.2-1: Compile Libgdiplus-2.6[/size]
[list]$cd /libgdiplus-2.6.tar.bz2
$./configure
watch for any errors be sure to fix them installing any -dev
$sudo make
$sudo make install[/list:u]

[size=100]2.2-2: Compile mono-2.6.1[/size]
[list]$cd /mono-2.6.1
$./configure
$sudo make
$sudo make install[/list:u]

[size=150]Step 3: Preparing your Repository and Client machines for installs[/size]
Do step 3.1 on both machines.
[size=100]3.1: Making sure Samba is installed[/size]
[list]$sudo apt-get install smbfs
$sudo apt-get install samba
$sudo apt-get install sambaclient[/list:u]

[size=100]3.2 Set install files to executable[/size]
make sure both files can be executed by ubuntu by right clicking them and setting

[size=100]3.3 Server: Install DeadlineRespository to the Server[/size]
cd to the directory with the DeadlineRespository install file
[list]$sudo ./DeadlineRespository
follow the install prompts, this should install DeadlineRespository into /usr/local/Prime_Focus/DeadlineRespository[/list:u]

[size=100]3.4 Server: Setting SMB Share via smb.conf[/size]
One of the best ways to ensure the Deadline repository and any project folders on the server can be read and written to is just to plain hardset the folder share up in the smb.conf. The nautilus approach can be flakey in so far as working out permissions.
smb.conf is found in /etc/samba
So lets get to editing the smb.conf to share the deadline repository directory
[list]$sudo gedit /etc/samba/smb.conf
it is also a good idea in this file to make sure the following matches your domain name if you’re not working to WORKGROUP.

# Change this to the workgroup/NT-domain name your Samba server will part of workgroup = WORKGROUP
So change WORKGROUP to the domain name.

Pagedown to the bottom of this file and add

[DeadlineRepository]
path = /usr/local/Prime_Focus/DeadlineRepository
comment = DeadlineRepository Share
guest ok = Yes
public = yes
writable = yes
valid users = nobody guest admin
create mask = 0777
directory mask = 0777
force create mode = 0777
force directory mode = 0777
unix extensions = No

As far as I know sharing this via Nautilus as well doesn’t cause any major conflicts, so feel free to do that also if ya must.
Finally;
$sudo chmod -R 777 /usr/local/DeadlineRepository[/list:u]
$sudo chown -R nobody:nogroup /path/to/repository
$sudo chmod -R 777 /path/to/repository

[size=100]3.5 Client: Setting fstab to mount DeadlineRepository on boot[/size]
We have to make a directory to mount into first. I tend to stick with /media as opposed to /dev for everything just so I know where everything is, but if you prefer /dev for shares by all means make the directories there.
[list]$sudo mkdir /media/DeadlineRepository
editing the fstab to mount the deadline repository
$sudo gedit /etc/fstab

//SERVERIP/DeadlineRepository /media/DeadlineRepository  smbfs credentials=/root/.smbcredentials,uid=nobody,gid=nogroup,noperm 0  0  
[/code][/list:u]
noperm from [url=http://support.na.primefocusworld.com/viewtopic.php?f=11&t=3673&p=14579#p14579]s.s.t's thread here[/url]
save it and close it.
credentials=/root/.smbcredentials refers to a file we have to make that will reside in the /root/ folder. This file is called upon by stab everytime it tries to mount the folder and it will use the username and password found in that file to automatically log you into the machine/share.
$sudo gedit /root/.smbcredentials
the format for this file is
[code]username=USERNAME
password=PASSWORD
workgroup=DOMAINNAME

eg:
username=macky
password=woger
workgroup=WORKGROUP

$sudo mount /media/DeadlineRepository
The directory should pop up.

NOTE: It is also a good idea to edit your hosts file and add into this the IP and then computer name for the IP so you can use machine names as well as IP’s
$sudo gedit /etc/hosts

10.1.1.3 macky-desktop is an example of a line added to this file, this just helps ubuntu out to know what is where computer wise.

NOTE: if you have any spaces in a networkdrive share name you can get around it by using

You can check smb shares around you using
$smbtree

NOTE: It is VERY important to note that ALL machines MUST be able to SEE your projects/renderoutputs folders too for a renderfarm, and any scripts folders etc that may be necessary for rendering should be across all boxes as well. Basically the rule of thumb is once you start in on renderfarm stuff, you should make sure everything works across the network proplery and everything can see everything else properly or stuff just dun work.

[size=100]3.6 Client: Install DeadlineClient[/size]
Now that we have the repository folder from the server mounted properly on the client machine, we can install the DeadlineClient.
cd to the directory with the DeadlineClient install file on the client machine.
$sudo ./DeadlineClient
follow the prompts and use the correct ip paths / ports, /media/DeadlineRepository is the full path to the repository folder on the client.

[size=150]Step 4: Creating a startup sh file and setting it to autoload via System/Preferences/Startup Applications[/size]
Pretty straight foward this one since we’re going to use the System/Preferences/Startup Applications
Create a file in your home folder called deadlinestart.sh. Right click this and make sure it is set to be executable.
in this copy paste

#!/bin/bash
# Deadline Launcher
deadlinelauncher

open System/Preferences/Startup Applications
Click ADD
Browse to the script
Give it a name and comment
Save
That should be it.
If you like you can set the launcher to NOT start the slaves when it is fired off, and make a new deadlineslave.sh file that sets the slaves off with no guid

#!/bin/bash
# Deadline Launcher
deadlineslave -nogui

open Syst

[size=150]Setting Up Linux Ubuntu Renderman for Maya 3.0.2 for Deadline[/size]
Assuming you’ve already aliened the rpms and installed etc (if necessary I can write that up too)…

Really important stuff here in so far as Renderman For Maya useage with Deadline.
Renderman for maya doesn’t RIB then render like Proserver does. With Proserver the ribbing process is done entirely before the rendering of the frames commences. How ever with Renderman for Maya whilst the ribbing/compiling happens before the frames are rendered, this is done on a `frames per task’situation. That is to say every block of 10frames rib out from each machine into the same directories by default and if machine01 is ribbing/compiling shaders while machine02 is trying to read them you’ll end up with errors all through the render.
The easiest way to rectify this is to set each renderfarm node to rib/compile etc into it’s OWN folder
To do this you can setup in the root directory of the project folder a RMSworkspace.ws file
Change the following lines to match below.

SetPref WSSubdir.rfmData {renderman/${SCENE}/${HOST}/data} SetPref WSSubdir.rfmShaders {renderman/${SCENE}/${HOST}/shaders} SetPref WSSubdir.rfmTextures {renderman/${SCENE}/${HOST}/textures}
I will double check if the ${SCENE} is still needed, but for now lets discuss how in Ubuntu to get a HOST env var available manually
$sudo gedit .profile
add to this file the following replacing INSERTHOSTNAMEHERE with the hostname of the computer (found easily by typing hostname into a commandline console).
export HOST=“INSERTHOSTNAMEHERE”
save the file
reboot or log out and back in.
Now you have made the hostname for that computer avail to renderman for maya for use with the workspace file and each machine will rib/compile out to it’s own folders.

stuff this up and you’ll error out so be sure to do this step right too;
sudo ln -s /opt/pixar/RenderManForMaya-3.0.2-maya2008/etc/rmanRenderer.xml /usr/autodesk/maya/bin/rendererDesc/

You will also need to edit the .module file properly and make sure maya can see that properly. If I need to add that in here lemme know and I will update.

_____ Okay that’s it for now, will update as posts / info comes through or changes.
Good Luck all!

History:
4.0

  • Desktop locks up when submitting to the farm Fixed in SP1
    -work around is to hack the py found /DeadlineRepository/scripts/Submission/MayaSubmission/MayaSubmission.py and to delete the following near the bottom of the file

ProcessUtils.WaitForExit( process, -1 ) if( process.ExitCode == 0 ): successes = successes + 1 else: failures = failures + 1

Wow this has to be the worst it’s been for me with deadline ever…
Used to work pretty sweet under 8.10 with dl (edit) 3.0 now … looks like I’ll be here all night nursing through the renders as nodes just randomly do shit I’ve never seen before…

4.0.0.40330

ubuntu 9.10

Mono JIT compiler version 2.6.1 (tarball Tue Mar 2 12:13:57 EST 2010)
TLS: __thread
GC: Included Boehm (with typed GC and Parallel Mark)
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none


[code]Error Message
Exception during render: An error occurred in RenderTasks(): Error in PostRenderTasks(): Verification failed because source directory /home/klinik3d1/Deadline/slave/jobsData/mayaOutput_tempK8yTv0 is empty!
at Deadline.Plugins.ScriptPlugin.RenderTasks (Int32 startFrame, Int32 endFrame, System.String& outMessage) [0x00000] in :0

Slave Log
0: STDOUT: File read in 0 seconds.
0: STDOUT: File read in 0 seconds.
0: STDOUT: Result: 1
0: STDOUT: Result: 1
0: STDOUT: Result: 1
0: STDOUT: Result: 1
0: STDOUT: Result: 1
0: STDOUT: Result: 1
0: STDOUT: Result: 1
0: STDOUT: Result: 1
0: STDOUT: Result: 1
0: STDOUT: Result: /media/projects/P694_FANIMANIA/_3D/shots/ANM_rabbit/rabbit_masterAnimation_v00.mb
0: INFO: Deadline is ignoring error: “Error: (03/08 19:52) rfm Error: Couldn’t check out internal rendering license for 8 threads.” because plugin setting Strict Error Checking is enabled and this error is not usually fatal.
0: STDOUT: Error: (03/08 19:52) rfm Error: Couldn’t check out internal rendering license for 8 threads.
0: STDOUT: (03/08 19:52) rfm Info: Batch rendered images will be written to /home/klinik3d1/Deadline/slave/jobsData/mayaOutput_tempK8yTv0/
0: STDOUT: Scene /home/klinik3d1/Deadline/slave/jobsData/RNR_rabbit_top_v00.mb completed.
0: INFO: Process exit code: 0
0: INFO: Moving output files and folders from /home/klinik3d1/Deadline/slave/jobsData/mayaOutput_tempK8yTv0 to /media/projects/P694_FANIMANIA/_FRAMEDUMP/3D_dump/images/NRL_Rabbit_Top
0: An exception occurred: Exception during render: An error occurred in RenderTasks(): Error in PostRenderTasks(): Verification failed because source directory /home/klinik3d1/Deadline/slave/jobsData/mayaOutput_tempK8yTv0 is empty!
at Deadline.Plugins.ScriptPlugin.RenderTasks (Int32 startFrame, Int32 endFrame, System.String& outMessage) [0x00000] in :0 (Deadline.Plugins.RenderPluginException)
0: Unloading plugin: MayaCmd
Scheduler Thread - Render Thread 0 threw an error:
Scheduler Thread - Exception during render: An error occurred in RenderTasks(): Error in PostRenderTasks(): Verification failed because source directory /home/klinik3d1/Deadline/slave/jobsData/mayaOutput_tempK8yTv0 is empty!
at Deadline.Plugins.ScriptPlugin.RenderTasks (Int32 startFrame, Int32 endFrame, System.String& outMessage) [0x00000] in :0

Error Type
RenderPluginException

Error Stack Trace[/code]

[code]
Error Message
Exception during render: An error occurred in RenderTasks(): Error in CheckExitCode(): Renderer returned non-zero error code 139. Check the renderer’s output.
at Deadline.Plugins.ScriptPlugin.RenderTasks (Int32 startFrame, Int32 endFrame, System.String& outMessage) [0x00000] in :0

Slave Log
al.
0: STDOUT: Warning: (03/08 20:17) rfm Warning: R01003 Invalid shading rate: 0
0: STDOUT: (03/08 20:17) rfm Notice: Rendering keyP at 24
0: INFO: Deadline is ignoring error: “Error: (03/08 20:17) rfm Error: T02002 “/media/projects/P694_FANIMANIA/renderman/nrl_rooster_Top_v00/k3dws2/data/0024/top_env_key_spotShape_Shadow.0024.tex” is not a texture file. [ Object: rim_key:skel:rooster_shirt_geoShape Shader: /media/projects/P694_FANIMANIA/renderman/nrl_rooster_Top_v00/k3dws2/shaders/env_key_spotShape_]” because plugin setting Strict Error Checking is enabled and this error is not usually fatal.
0: STDOUT: Error: (03/08 20:17) rfm Error: T02002 “/media/projects/P694_FANIMANIA/renderman/nrl_rooster_Top_v00/k3dws2/data/0024/top_env_key_spotShape_Shadow.0024.tex” is not a texture file. [ Object: rim_key:skel:rooster_shirt_geoShape Shader: /media/projects/P694_FANIMANIA/renderman/nrl_rooster_Top_v00/k3dws2/shaders/env_key_spotShape_]
0: STDOUT: maya encountered a fatal error
0: STDOUT: Signal: 11 (Unknown Signal)
0: STDOUT: Result: /home/klinik/Deadline/slave/jobsData/nrl_rooster_Top_v00.ma
0: STDOUT: // Maya exited with status 139
0: STDOUT: Segmentation fault
0: INFO: Process exit code: 139
0: An exception occurred: Exception during render: An error occurred in RenderTasks(): Error in CheckExitCode(): Renderer returned non-zero error code 139. Check the renderer’s output.
at Deadline.Plugins.ScriptPlugin.RenderTasks (Int32 startFrame, Int32 endFrame, System.String& outMessage) [0x00000] in :0 (Deadline.Plugins.RenderPluginException)
0: Unloading plugin: MayaCmd
Scheduler Thread - Render Thread 0 threw an error:
Scheduler Thread - Exception during render: An error occurred in RenderTasks(): Error in CheckExitCode(): Renderer returned non-zero error code 139. Check the renderer’s output.
at Deadline.Plugins.ScriptPlugin.RenderTasks (Int32 startFrame, Int32 endFrame, System.String& outMessage) [0x00000] in :0

Error Type
RenderPluginException

Error Stack Trace
at Deadline.Plugins.Plugin.RenderTask (Int32 startFrame, Int32 endFrame) [0x00000] in :0
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask () [0x00000] in :0 [/code]

this one being one fo the more annoying as you have to fail the task and wait for the entire job to finish before you can hack up a re que cause you can’t just reque these as they get stuck in a `rendering’ state but with no slave assigned

Error Message
STALLED SLAVE REPORT

Current House Cleaner Information
  Machine Performing Cleanup: Klinik3dserver
  Version: v4.0.0.40330 R

Stalled Slave: K3dws4
Last Slave Update: 2010-03-08 19:20:57
Current Time: 2010-03-08 20:21:10
Time Difference: 1.004 hrs
Maximum Time Allowed Between Updates: 10 m

Current Job Name: Rooster_Top
Current Task Names: 41-50
Current Task Ids: 4

Searching for job with id "000_080_000_77c3378a"
Found possible job: Rooster_Top
Searching for task with id "4"
Found possible task: :[41-50]
Task's current slave: K3dws4
Slave machine names match, stopping search
Associated Job Found: Rooster_Top
Associated Task Found: :[41-50]
Task's current slave: K3dws4
Task is still rendering, attempting to fix situation.
Requeuing task
Setting slave's status to Stalled.
Setting last update time to now.
Slave state updated.
Slave Log
Error Type
Error Stack Trace

EDIT:
Shut the farm down
checked all mappings
checked all logins
checked all hosts files
checked all config paths
checked all installs / updates

THIS ERROR IS KILLING ME

Error Message
STALLED SLAVE REPORT

Current House Cleaner Information
  Machine Performing Cleanup: Klinik3dserver
  Version: v4.0.0.40330 R

Stalled Slave: K3dws4
Last Slave Update: 2010-03-08 21:03:23
Current Time: 2010-03-08 22:03:54
Time Difference: 1.009 hrs
Maximum Time Allowed Between Updates: 10 m

Current Job Name: Rooster_ThreeQrtr
Current Task Names: 61-70
Current Task Ids: 6

Searching for job with id "999_080_999_748a2d02"
Found possible job: Rooster_ThreeQrtr
Searching for task with id "6"
Found possible task: :[61-70]
Task's current slave: K3dws4
Slave machine names match, stopping search
Associated Job Found: Rooster_ThreeQrtr
Associated Task Found: :[61-70]
Task's current slave: K3dws4
Task is still rendering, attempting to fix situation.
Requeuing task
Setting slave's status to Stalled.
Setting last update time to now.

Slave state updated.



Slave Log

EDIIT
Checked Referenced Scenefile, removed RMFM plugin, removed any RM globals that exist and force save that clean with no globals
This appears to be hanging in there now…for the most part… time to go to bed I think

Sorry for not jumping in here sooner. When we first saw this post, it was a work in progress, and the forum doesn’t send notifications when edits are made to existing posts (it only sends notifications for new posts and threads).

So let’s start with the issues you’ve run into:

The Monitor submission scripts don’t support resizing (you can resize the main window, but the controls don’t grow/shrink while doing this). I’m assuming there was a display problem that was resolved by changing the size in the script? Maybe post a before and after screen shot to help us understand the problem better?

Yup, this was a bug in 4.0 that was fixed in 4.0 SP1.

We think this is a Mono issue. We find that clicking the mouse again gets rid of the select rectangle. Is this something that affects the usage of the Monitor for you, or is it just more of a graphical glitch?

What is preventing you from doing this? Does the right-click menu show up? Do you get an error when you try to requeue it? We can’t seem to reproduce this, so more details (including error messages, if any) would be helpful.

Not exactly sure what you mean. Is it that all of your rendering slaves are showing up as idle in the slave list as opposed to rendering? Have you confirmed if this is still an issue in 4.0 SP1?

How so? The timeout never kicks in? A timeout error occurs, but the task isn’t requeued? Again, more details would be helpful.

We can’t reproduce this. Can you successfully reproduce this in 4.0 SP1? What’s the state of the remote command window when you click the button? Is the list empty, full of completed commands, or does it contain a mixture of completed/failed/pending commands?

If a task has failed, right-click on it and select Options → Resume Failed Task. If a whole job has failed, right-click on it and select Resume Failed Job.

You’ll have to explain this better. More details, steps to reproduce, and error messages (if any) would be helpful.

Please confirm if this is still the case in 4.0 SP1. Also, I should note that the status for a slave is only updated every 20 to 40 seconds. This is normal, and it has been this way since Deadline 1.0. Is it possible that the outdated data you see in the slave list is just a result of the 20-40 second interval between updates?

Unfortunately, we can’t reproduce this one either. Does the slave print out any errors when it is shutting down? Maybe check the slave log after you shut down the slave. You can open the log folder from the slave by selecting Help → Explore Log Folder.

========================================================

Now this maybe be a stupid question, but are you using the Refresh button in the Monitor to refresh the data periodically? The reason I ask is that a lot of the problems you’re mentioning could be related to outdated data. Just thought I would throw this out there, just in case…

========================================================

Deadline 2.7 was Windows only. :wink: Perhaps you’re referring to Deadline 3.0?

It looks like the RFM plugin was the root of your problems here, as both errors seem to be related (and Deadline was just the messenger in this case). Please update on these jobs when you get back in to let us know if removing references to this plugin helped.

The stalled slave errors appear to be related to the 1 hour time difference you mention in another post. Hopefully synchronizing the time between all of your machines resolves this issue.

========================================================

Hope some of this information is helpful, and hopefully you can provide us more details on issues you’ve encountered so we can investigate further. Also, in the future, please create new posts for new problems and comments, as the few posts that are here are pretty long, and any edits that are made may go unnoticed.

Cheers,

  • Ryan

Well it can kinda well stuck and clicking around the window to release it might take several clicks before being able to right click / access a slave :slight_smile: More of an annoyance than anything :slight_smile:

I think I need to build some vids for you.
Right clicking and telling it to reque doesn’t work.
hmm

kk the resounding IT `more details’ is clear and I’ll wack together a more robust post with screenies etc for you :slight_smile:

Yeah :slight_smile: Lots. which is why say for example when the slaves suddenly all report as idle at the same time, close the monitor open it and it displays them properly again is kinda weird.
Anyways I’ll get some SS’s

lol blush lets just say overworked just a tad atm

I’d have to say the two main errors last night were:
a) Referenced RM Globals conflicting (Renderman makes these as it loads a scene and they get saved, ya have to force the plugin not to load and to remove all the rman stuff from the DAG and save that for referencing)
b) ws4 was timing out (had changed time out checks etc to avoid that but it didn’t make a lick of diff till I changed the clock) due to the clock.

Still getting a few errors this morning, but need to get some stuff sorted and then work th rough it properly when I’m not so under the gun.

Thanks for the reply, will get stuck in asap