You can see here
https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/aws-portal-setup-overview.html
Portal Link must be able to contact an RCS, and the Asset Server needs access to the File server. So this setting for the connection server is probably ok.
Can’t help you with the rest.
Thank you for your advice.
But, I can’t resolve this problem.
I submit the job, job status is still “queud”, and asset server log is not here in the aws portal server.
I don’t know next step.
In your previous posts, you said that the linux box was the aws portal server.
On that linux box, it would be simpler to install these of these on there:
And then:
URLs:
Starting RCS:
https://docs.thinkboxsoftware.com/products/deadline/10.3/1_User%20Manual/manual/aws-portal-remote-connection-server.html
Check / verify connection:
https://docs.thinkboxsoftware.com/products/deadline/10.3/1_User%20Manual/manual/aws-portal-troubleshooting/verify-connection-to-rcs.html#aws-portal-verify-connection-to-rcs-ref-label
These (old) tutorials are pretty good:
I do see this command in the docs, but I personally have not run this command. It may (or may not) output an error that would be helpful in debugging.
deadlinecommand -AWSPortalPrecacheJob <Job ID(s)>
Thank you for the deatil description.
My environment is special for the security. This is my architecture.
So, RCS and AWS portal is on the different server.
Thanks to your advice, I found out some point.
Infrastructure can’t connect to the RCS.
What is this error? How do I resolve it?
From the AWSPortal server can you reach the RCS?
Try something like: curl -v http://10.111.80.51:8080
e.g.:
[root@test Deadline10]# curl -v http://10.111.80.51:8080
* About to connect() to 10.111.80.51 port 8080 (#0)
* Trying 10.111.80.51...
* Connected to 10.111.80.51 (10.111.80.51) port 8080 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.111.80.51:8080
> Accept: */*
>
* Empty reply from server
* Connection #0 to host 10.111.80.51 left intact
There is also this page (AWS Portal Infrastructure is Incorrectly Configured for Your Remote Connection Server) : AWS Portal Workers Don’t Connect to Repository — Deadline 10.3.1.4 documentation
The docs says (under Configure Infrastructure) Full RCS Hostname “if available” – but it doesn’t say “required”. Maybe someone from Thinkbox knows if you can leave it blank or not.
Not sure which server is complaining about the 502 Bad Gateway – the AWS Portal server or the DL Gateway instance. Was there something at the beginning of the log file?
No, the GetTunnelParams.py file doesn’t need to have an executable bit. It is being interpreted by python, so the python binary is the one that should be executable.
Thank you for your advice.
I resolve this problem.
But I got new problem.
The worker that stood via aws portal has error log.
I have received the answer before. But I may not understand it.
This is the security rule of my infrastructure.
This is the security rule of my worker.
What does cause this problem?
And I want to know how to resolve this problem, if you know the cause.
“Address already in use”
Either hserver or aws portal is already running and you cannot start a new one on the same port.
Hi.
Did you install Houdini on your customized AMI? I’m asking because if you run the Houdini installer (linux), it installs “License Server” (sesinetd) by default. You don’t need sesinetd running on the worker.
The log is complaining about port 1715 which is sesinetd. As a test you should be able to disable sesinetd from running on the worker and see if the render progresses. You should be able to connect to the worker via aws console (or ssh into the worker via the GW instance) to check:
Check for sesinetd process:
ps ax | grep sesi
Check if port 1715 is open:
[ec2-user@test]$ ss -a | grep 1715
tcp LISTEN 0 128 *:1715 *:*
Stop and disable the daemon:
sudo systemctl disable --now sesinetd.service
or if you’re using the older init style:
sudo /etc/init.d/sesinetd stop
sudo chkconfig sesinetd off
Run the ss -a command again to check that port 1715 has closed.
If all that worked you can modify your base AMI.
Hi jarak,
Thank you.
I install houdini on my custom AMI.
I uninstall hserver, then job rendering can work.
But I have new error.
2024-03-15 01:48:01: Scheduler Thread - Scheduler State transition from = 'PreRendering' to = 'PostRendering'
2024-03-15 01:48:01: Scheduler Thread - Scheduler State transition from = 'PostRendering' to = 'EndJob'
2024-03-15 01:48:01: Scheduler Thread - Scheduler State transition from = 'EndJob' to = 'WaitingForJob'
2024-03-15 01:48:01: Scheduler Thread - Seconds before next job scan: 1
2024-03-15 01:48:03: Scheduler Thread - Scheduler State transition from = 'WaitingForJob' to = 'LicenseCheck'
2024-03-15 01:48:03: Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2024-03-15 01:48:03: Scheduler - Returning limit stubs not in use.
2024-03-15 01:48:03: Scheduler Thread - Job's Limit Groups: houdini, redshift
2024-03-15 01:48:03: Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'LicenseCheck'
2024-03-15 01:48:03: Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2024-03-15 01:48:03: Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'StartJob'
2024-03-15 01:48:03: 0: Render Thread - Render State transition from = 'WaitingForTask' to = 'ReceivedTask'
2024-03-15 01:48:03: Scheduler Thread - Scheduler State transition from = 'StartJob' to = 'PreRendering'
2024-03-15 01:48:03: 0: Got task!
2024-03-15 01:48:03: 0: Render Thread - Render State transition from = 'ReceivedTask' to = 'Other'
2024-03-15 01:48:03: 0: Loading Job's Plugin timeout is Disabled
2024-03-15 01:48:03: 0: SandboxedPlugin: Render Job As User disabled, running as current user 'ec2-user'
2024-03-15 01:48:04: '/home/ec2-user/Thinkbox/Deadline10/pythonAPIs/8oI2s+G2DZfcng9zEWRA==' already exists. Skipping extraction of PythonSync.
2024-03-15 01:48:05: 0: Loaded plugin Houdini
2024-03-15 01:48:05: All job files are already synchronized
2024-03-15 01:48:05: Plugin Houdini was already synchronized.
2024-03-15 01:48:05: 0: Executing plugin command of type 'Initialize Plugin'
2024-03-15 01:48:05: 0: INFO: Executing plugin script '/var/lib/Thinkbox/Deadline10/workers/ip-10-128-18-28/plugins/65f3a6d8874df71c50592ce4/Houdini.py'
2024-03-15 01:48:05: 0: INFO: Plugin execution sandbox using Python version 3
2024-03-15 01:48:05: 0: INFO: About: Houdini Plugin for Deadline
2024-03-15 01:48:05: 0: INFO: The job's environment will be merged with the current environment before rendering
2024-03-15 01:48:05: 0: Done executing plugin command of type 'Initialize Plugin'
2024-03-15 01:48:05: 0: Start Job timeout is disabled.
2024-03-15 01:48:05: 0: Task timeout is disabled.
2024-03-15 01:48:05: 0: Loaded job: B_rs_dl_render_test_full_path - /obj/ropnet1/Redshift_ROP (65f3a6d8874df71c50592ce4)
2024-03-15 01:48:06: 0: Executing plugin command of type 'Start Job'
2024-03-15 01:48:06: 0: INFO: Sending StartTaskRequest to S3BackedCacheClient.
2024-03-15 01:48:06: 0: DEBUG: Request:
2024-03-15 01:48:06: 0: DEBUG: JobId: 65f3a6d8874df71c50592ce4
2024-03-15 01:48:06: 0: DEBUG: JobUploadWhitelist:
2024-03-15 01:48:06: 0: DEBUG: JobUploadWhitelistRe: ^.+\.abc$, ^.+\.avi$, ^.+\.bmp$, ^.+\.bw$, ^.+\.cin$, ^.+\.cjp$, ^.+\.cjpg$, ^.+\.cxr$, ^.+\.dds$, ^.+\.dpx$, ^.+\.dwf$, ^.+\.dwfx$, ^.+\.dwg$, ^.+\.dxf$, ^.+\.dxx$, ^.+\.eps$, ^.+\.exr$, ^.+\.fbx$, ^.+\.fxr$, ^.+\.hdr$, ^.+\.icb$, ^.+\.iff$, ^.+\.iges$, ^.+\.igs$, ^.+\.int$, ^.+\.inta$, ^.+\.iris$, ^.+\.jpe$, ^.+\.jpeg$, ^.+\.jpg$, ^.+\.jp2$, ^.+\.mcc$, ^.+\.mcx$, ^.+\.mov$, ^.+\.mxi$, ^.+\.pdf$, ^.+\.pic$, ^.+\.png$, ^.+\.prt$, ^.+\.ps$, ^.+\.psd$, ^.+\.rgb$, ^.+\.rgba$, ^.+\.rla$, ^.+\.rpf$, ^.+\.sat$, ^.+\.sgi$, ^.+\.stl$, ^.+\.sxr$, ^.+\.targa$, ^.+\.tga$, ^.+\.tif$, ^.+\.tiff$, ^.+\.tim$, ^.+\.vda$, ^.+\.vrimg$, ^.+\.vrmesh$, ^.+\.vrsm$, ^.+\.vrst$, ^.+\.vst$, ^.+\.wmf$, ^.+\.ass$, ^.+\.gz$, ^.+\.ifd$, ^.+\.mi$, ^.+\.mi2$, ^.+\.mxi$, ^.+\.rib$, ^.+\.rs$, ^.+\.vrscene$
2024-03-15 01:48:06: 0: DEBUG: S3BackedCache Client Returned Sequence: 1
2024-03-15 01:48:06: 0: INFO: Executing global asset transfer preload script '/var/lib/Thinkbox/Deadline10/workers/ip-10-128-18-28/plugins/65f3a6d8874df71c50592ce4/GlobalAssetTransferPreLoad.py'
2024-03-15 01:48:06: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2024-03-15 01:48:06: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in /opt/Thinkbox/S3BackedCache/bin/task.py...
2024-03-15 01:48:06: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2024-03-15 01:48:06: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2024-03-15 01:48:06: 0: Done executing plugin command of type 'Start Job'
2024-03-15 01:48:06: 0: Plugin rendering frame(s): 1
2024-03-15 01:48:06: 0: Render Thread - Render State transition from = 'Other' to = 'Rendering'
2024-03-15 01:48:06: 0: Executing plugin command of type 'Render Task'
2024-03-15 01:48:06: 0: INFO: Sending StartTaskRequest to S3BackedCacheClient.
2024-03-15 01:48:06: 0: DEBUG: Request:
2024-03-15 01:48:06: 0: DEBUG: JobId: 65f3a6d8874df71c50592ce4
2024-03-15 01:48:06: 0: DEBUG: JobUploadWhitelist:
2024-03-15 01:48:06: 0: DEBUG: JobUploadWhitelistRe: ^.+\.abc$, ^.+\.avi$, ^.+\.bmp$, ^.+\.bw$, ^.+\.cin$, ^.+\.cjp$, ^.+\.cjpg$, ^.+\.cxr$, ^.+\.dds$, ^.+\.dpx$, ^.+\.dwf$, ^.+\.dwfx$, ^.+\.dwg$, ^.+\.dxf$, ^.+\.dxx$, ^.+\.eps$, ^.+\.exr$, ^.+\.fbx$, ^.+\.fxr$, ^.+\.hdr$, ^.+\.icb$, ^.+\.iff$, ^.+\.iges$, ^.+\.igs$, ^.+\.int$, ^.+\.inta$, ^.+\.iris$, ^.+\.jpe$, ^.+\.jpeg$, ^.+\.jpg$, ^.+\.jp2$, ^.+\.mcc$, ^.+\.mcx$, ^.+\.mov$, ^.+\.mxi$, ^.+\.pdf$, ^.+\.pic$, ^.+\.png$, ^.+\.prt$, ^.+\.ps$, ^.+\.psd$, ^.+\.rgb$, ^.+\.rgba$, ^.+\.rla$, ^.+\.rpf$, ^.+\.sat$, ^.+\.sgi$, ^.+\.stl$, ^.+\.sxr$, ^.+\.targa$, ^.+\.tga$, ^.+\.tif$, ^.+\.tiff$, ^.+\.tim$, ^.+\.vda$, ^.+\.vrimg$, ^.+\.vrmesh$, ^.+\.vrsm$, ^.+\.vrst$, ^.+\.vst$, ^.+\.wmf$, ^.+\.ass$, ^.+\.gz$, ^.+\.ifd$, ^.+\.mi$, ^.+\.mi2$, ^.+\.mxi$, ^.+\.rib$, ^.+\.rs$, ^.+\.vrscene$
2024-03-15 01:48:06: 0: DEBUG: S3BackedCache Client Returned Sequence: 1
2024-03-15 01:48:06: 0: INFO: Set HOUDINI_PATHMAP to {"/mnt/cgsv2021/":"/mnt/Data/mntcgsv2021386d7e3204ea8557cff0f48192a0df69/", "//cgsv2021":"/mnt/cgsv2021"}
2024-03-15 01:48:06: 0: INFO: Redshift Path Mapping...
2024-03-15 01:48:06: 0: INFO: source: "/mnt/cgsv2021/" dest: "/mnt/Data/mntcgsv2021386d7e3204ea8557cff0f48192a0df69/"
2024-03-15 01:48:06: 0: INFO: source: "\\cgsv2021" dest: "/mnt/cgsv2021"
2024-03-15 01:48:06: 0: INFO: [REDSHIFT_PATHOVERRIDE_FILE] now set to: "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-18-28/jobsData/65f3a6d8874df71c50592ce4/RSMapping_tempPDE5M0/RSMapping.txt"
2024-03-15 01:48:06: 0: INFO: Starting Houdini Job
2024-03-15 01:48:06: 0: INFO: Stdout Redirection Enabled: True
2024-03-15 01:48:06: 0: INFO: Asynchronous Stdout Enabled: False
2024-03-15 01:48:06: 0: INFO: Stdout Handling Enabled: True
2024-03-15 01:48:06: 0: INFO: Popup Handling Enabled: True
2024-03-15 01:48:06: 0: INFO: QT Popup Handling Enabled: False
2024-03-15 01:48:06: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2024-03-15 01:48:06: 0: INFO: Using Process Tree: True
2024-03-15 01:48:06: 0: INFO: Hiding DOS Window: True
2024-03-15 01:48:06: 0: INFO: Creating New Console: False
2024-03-15 01:48:06: 0: INFO: Running as user: ec2-user
2024-03-15 01:48:06: 0: INFO: Executable: "/opt/hfs19.5.640/bin/hython"
2024-03-15 01:48:06: 0: CheckPathMapping: Swapped "//cgsv2021/Projects02/AWS_Test/RS_deadline_test/render/Redshift_ROP/$Redshift_ROP.exr" with "/mnt/cgsv2021/Projects02/AWS_Test/RS_deadline_test/render/Redshift_ROP/$Redshift_ROP.exr"
2024-03-15 01:48:06: 0: CheckPathMapping: Swapped "//cgsv2021/projects02/AWS_Test/RS_deadline_test/B_rs_dl_render_test_full_path.hip" with "/mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/B_rs_dl_render_test_full_path.hip"
2024-03-15 01:48:06: 0: INFO: Argument: "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-18-28/plugins/65f3a6d8874df71c50592ce4/hrender_dl.py" -f 1 1 1 -o "/mnt/cgsv2021/Projects02/AWS_Test/RS_deadline_test/render/Redshift_ROP/$Redshift_ROP.exr" -d /obj/ropnet1/Redshift_ROP -tempdir "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-18-28/jobsData/65f3a6d8874df71c50592ce4/0_tempVBK4L0" -arnoldAbortOnLicenseFail 1 "/mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/B_rs_dl_render_test_full_path.hip"
2024-03-15 01:48:06: 0: INFO: Full Command: "/opt/hfs19.5.640/bin/hython" "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-18-28/plugins/65f3a6d8874df71c50592ce4/hrender_dl.py" -f 1 1 1 -o "/mnt/cgsv2021/Projects02/AWS_Test/RS_deadline_test/render/Redshift_ROP/$Redshift_ROP.exr" -d /obj/ropnet1/Redshift_ROP -tempdir "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-18-28/jobsData/65f3a6d8874df71c50592ce4/0_tempVBK4L0" -arnoldAbortOnLicenseFail 1 "/mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/B_rs_dl_render_test_full_path.hip"
2024-03-15 01:48:06: 0: INFO: Startup Directory: "/opt/hfs19.5.640/bin"
2024-03-15 01:48:06: 0: INFO: Process Priority: BelowNormal
2024-03-15 01:48:06: 0: INFO: Process Affinity: default
2024-03-15 01:48:06: 0: INFO: Process is now running
2024-03-15 01:48:09: 0: STDOUT: Detected Houdini version: (19, 5, 640)
2024-03-15 01:48:09: 0: STDOUT: ['/var/lib/Thinkbox/Deadline10/workers/ip-10-128-18-28/plugins/65f3a6d8874df71c50592ce4/hrender_dl.py', '-f', '1', '1', '1', '-o', '/mnt/cgsv2021/Projects02/AWS_Test/RS_deadline_test/render/Redshift_ROP/$Redshift_ROP.exr', '-d', '/obj/ropnet1/Redshift_ROP', '-tempdir', '/var/lib/Thinkbox/Deadline10/workers/ip-10-128-18-28/jobsData/65f3a6d8874df71c50592ce4/0_tempVBK4L0', '-arnoldAbortOnLicenseFail', '1', '/mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/B_rs_dl_render_test_full_path.hip']
2024-03-15 01:48:09: 0: STDOUT: Start: 1
2024-03-15 01:48:09: 0: STDOUT: End: 1
2024-03-15 01:48:09: 0: STDOUT: Increment: 1
2024-03-15 01:48:09: 0: STDOUT: Ignore Inputs: False
2024-03-15 01:48:09: 0: STDOUT: Output: /mnt/cgsv2021/Projects02/AWS_Test/RS_deadline_test/render/Redshift_ROP/$Redshift_ROP.exr
2024-03-15 01:48:09: 0: STDOUT: Driver: /obj/ropnet1/Redshift_ROP
2024-03-15 01:48:09: 0: STDOUT: Input File: /mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/B_rs_dl_render_test_full_path.hip
2024-03-15 01:48:09: 0: Sending kill command to process hython-bin with id: 11933
2024-03-15 01:48:09: 0: INFO: Sending EndTaskRequest to S3BackedCacheClient.
2024-03-15 01:48:09: 0: DEBUG: Request:
2024-03-15 01:48:09: 0: DEBUG: JobId: 65f3a6d8874df71c50592ce4
2024-03-15 01:48:09: 0: Done executing plugin command of type 'Render Task'
2024-03-15 01:48:09: ERROR: 0: An exception occurred: FailRenderException : Error: Caught exception: The attempted operation failed.
2024-03-15 01:48:09: at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
2024-03-15 01:48:09: File "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-18-28/plugins/65f3a6d8874df71c50592ce4/Houdini.py", line 412, in HandleStdoutError
2024-03-15 01:48:09: self.FailRender(self.GetRegexMatch(1))
2024-03-15 01:48:09: at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
2024-03-15 01:48:09: at __FranticX_Processes_ManagedProcess_StdoutHandlerDelegateDispatcher.Invoke()
2024-03-15 01:48:09: at FranticX.Processes.ManagedProcess.RegexHandlerCallback.CallFunction()
2024-03-15 01:48:09: at FranticX.Processes.ManagedProcess.e(String cj, Boolean ck)
2024-03-15 01:48:09: at FranticX.Processes.ManagedProcess.Execute(Boolean waitForExit)
2024-03-15 01:48:09: at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
2024-03-15 01:48:09: at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2024-03-15 01:48:09: at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel) (Deadline.Plugins.RenderPluginException)
2024-03-15 01:48:09: 0: Unloading plugin: Houdini
2024-03-15 01:48:09: 0: Executing plugin command of type 'End Job'
2024-03-15 01:48:09: 0: INFO: Sending EndTaskRequest to S3BackedCacheClient.
2024-03-15 01:48:09: 0: DEBUG: Request:
2024-03-15 01:48:09: 0: DEBUG: JobId: 65f3a6d8874df71c50592ce4
2024-03-15 01:48:09: 0: Done executing plugin command of type 'End Job'
2024-03-15 01:48:10: Port Forwarder (houdini:1715): Client connected to port forwarder.
2024-03-15 01:48:11: Sending kill command to process deadlinesandbox.exe with id: 11896
2024-03-15 01:48:12: 0: Render Thread - Render State transition from = 'Rendering' to = 'WaitingForTask'
2024-03-15 01:48:12: Scheduler Thread - Render Thread 0 threw a major error:
2024-03-15 01:48:12: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024-03-15 01:48:12: Exception Details
2024-03-15 01:48:12: RenderPluginException -- FailRenderException : Error: Caught exception: The attempted operation failed.
2024-03-15 01:48:12: at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
2024-03-15 01:48:12: File "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-18-28/plugins/65f3a6d8874df71c50592ce4/Houdini.py", line 412, in HandleStdoutError
2024-03-15 01:48:12: self.FailRender(self.GetRegexMatch(1))
2024-03-15 01:48:12: at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
2024-03-15 01:48:12: at __FranticX_Processes_ManagedProcess_StdoutHandlerDelegateDispatcher.Invoke()
2024-03-15 01:48:12: at FranticX.Processes.ManagedProcess.RegexHandlerCallback.CallFunction()
2024-03-15 01:48:12: at FranticX.Processes.ManagedProcess.e(String cj, Boolean ck)
2024-03-15 01:48:12: at FranticX.Processes.ManagedProcess.Execute(Boolean waitForExit)
2024-03-15 01:48:12: at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
2024-03-15 01:48:12: at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2024-03-15 01:48:12: at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2024-03-15 01:48:12: RenderPluginException.Cause: JobError (2)
2024-03-15 01:48:12: RenderPluginException.Level: Major (1)
2024-03-15 01:48:12: RenderPluginException.HasSlaveLog: True
2024-03-15 01:48:12: RenderPluginException.SlaveLogFileName: /var/log/Thinkbox/Deadline10/deadlineslave_renderthread_0-ip-10-128-18-28-0000.log
2024-03-15 01:48:12: Exception.TargetSite: Deadline.Slaves.Messaging.PluginResponseMemento d(Deadline.Net.DeadlineMessage, System.Threading.CancellationToken)
2024-03-15 01:48:12: Exception.Data: ( )
2024-03-15 01:48:12: Exception.Source: deadline
2024-03-15 01:48:12: Exception.HResult: -2146233088
2024-03-15 01:48:12: Exception.StackTrace:
2024-03-15 01:48:12: at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgm, CancellationToken bgn
2024-03-15 01:48:12: at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken
2024-03-15 01:48:12: at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajt, CancellationToken aju)
I want to have the advice.
Best Regards
Taka.
You’re making progress.
This error is coming from Houdini - “The attempted operation failed.”
You should check your scene, try to render it locally.
Often this error is caused by a locked node that needs to be modified or something similar.
Unfortunately the way the Deadline plugin works, it stops at the first line showing an error, thus not reaching the actual error in the output.
Can you please try disabling “Enable Path Mapping” in the Configure Plugins → Houdini. Set it to “False”. And maybe unset REDSHIFT_PATHOVERRIDE_FILE.
You want Deadline global rules to handle the Mapped Paths (not Houdini and Redshift), so the “Full Command” would have the S3 bucket paths, not your local paths. In post #22 the Global Mapped Paths look correct.
I see in your logs:
2024-03-15 01:48:06: 0: INFO: Set HOUDINI_PATHMAP to {"/mnt/cgsv2021/":"/mnt/Data/mntcgsv2021386d7e3204ea8557cff0f48192a0df69/", "//cgsv2021":"/mnt/cgsv2021"}
and
2024-03-15 01:48:06: 0: INFO: Redshift Path Mapping...
2024-03-15 01:48:06: 0: INFO: source: "/mnt/cgsv2021/" dest: "/mnt/Data/mntcgsv2021386d7e3204ea8557cff0f48192a0df69/"
2024-03-15 01:48:06: 0: INFO: source: "\\cgsv2021" dest: "/mnt/cgsv2021"
The logs show the command that the AWS Deadline worker is using are local paths, not remote bucket paths:
2024-03-15 01:48:06: 0: INFO: Full Command: "/opt/hfs19.5.640/bin/hython" "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-18-28/plugins/65f3a6d8874df71c50592ce4/hrender_dl.py" -f 1 1 1 -o "/mnt/cgsv2021/Projects02/AWS_Test/RS_deadline_test/render/Redshift_ROP/$Redshift_ROP.exr" -d /obj/ropnet1/Redshift_ROP -tempdir "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-18-28/jobsData/65f3a6d8874df71c50592ce4/0_tempVBK4L0" -arnoldAbortOnLicenseFail 1 "/mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/B_rs_dl_render_test_full_path.hip"
Thank you for your advices.
I checked “submit Houdini Scenes” and changed “Enable Path Mapping” to false.
The rendering is working.
But output folder(local) is nothing and the worker log is down below.
2024-03-18 05:35:41: 0: INFO: Starting Houdini Job
2024-03-18 05:35:41: 0: INFO: Stdout Redirection Enabled: True
2024-03-18 05:35:41: 0: INFO: Asynchronous Stdout Enabled: False
2024-03-18 05:35:41: 0: INFO: Stdout Handling Enabled: True
2024-03-18 05:35:41: 0: INFO: Popup Handling Enabled: True
2024-03-18 05:35:41: 0: INFO: QT Popup Handling Enabled: False
2024-03-18 05:35:41: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2024-03-18 05:35:41: 0: INFO: Using Process Tree: True
2024-03-18 05:35:41: 0: INFO: Hiding DOS Window: True
2024-03-18 05:35:41: 0: INFO: Creating New Console: False
2024-03-18 05:35:41: 0: INFO: Running as user: ec2-user
2024-03-18 05:35:41: 0: INFO: Executable: "/opt/hfs19.5.640/bin/hython"
2024-03-18 05:35:41: 0: CheckPathMapping: Swapped "//cgsv2021/projects02/AWS_Test/RS_deadline_test/render/testB/testB.$F4.exr" with "/mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/render/testB/testB.$F4.exr"
2024-03-18 05:35:41: 0: INFO: Argument: "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-29-182/plugins/65f7cc28874df71c50593303/hrender_dl.py" -f 238 238 1 -o "/mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/render/testB/testB.$F4.exr" -d /out/Redshift_ROP1 -tempdir "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-29-182/jobsData/65f7cc28874df71c50593303/0_tempkHIL50" -arnoldAbortOnLicenseFail 1 "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-29-182/jobsData/65f7cc28874df71c50593303/rs_dl_render_test_B.hip"
2024-03-18 05:35:41: 0: INFO: Full Command: "/opt/hfs19.5.640/bin/hython" "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-29-182/plugins/65f7cc28874df71c50593303/hrender_dl.py" -f 238 238 1 -o "/mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/render/testB/testB.$F4.exr" -d /out/Redshift_ROP1 -tempdir "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-29-182/jobsData/65f7cc28874df71c50593303/0_tempkHIL50" -arnoldAbortOnLicenseFail 1 "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-29-182/jobsData/65f7cc28874df71c50593303/rs_dl_render_test_B.hip"
2024-03-18 05:35:41: 0: INFO: Startup Directory: "/opt/hfs19.5.640/bin"
2024-03-18 05:35:41: 0: INFO: Process Priority: BelowNormal
2024-03-18 05:35:41: 0: INFO: Process Affinity: default
2024-03-18 05:35:41: 0: INFO: Process is now running
2024-03-18 05:35:43: 0: STDOUT: Detected Houdini version: (19, 5, 640)
2024-03-18 05:35:43: 0: STDOUT: ['/var/lib/Thinkbox/Deadline10/workers/ip-10-128-29-182/plugins/65f7cc28874df71c50593303/hrender_dl.py', '-f', '238', '238', '1', '-o', '/mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/render/testB/testB.$F4.exr', '-d', '/out/Redshift_ROP1', '-tempdir', '/var/lib/Thinkbox/Deadline10/workers/ip-10-128-29-182/jobsData/65f7cc28874df71c50593303/0_tempkHIL50', '-arnoldAbortOnLicenseFail', '1', '/var/lib/Thinkbox/Deadline10/workers/ip-10-128-29-182/jobsData/65f7cc28874df71c50593303/rs_dl_render_test_B.hip']
2024-03-18 05:35:43: 0: STDOUT: Start: 238
2024-03-18 05:35:43: 0: STDOUT: End: 238
2024-03-18 05:35:43: 0: STDOUT: Increment: 1
2024-03-18 05:35:43: 0: STDOUT: Ignore Inputs: False
2024-03-18 05:35:43: 0: STDOUT: Output: /mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/render/testB/testB.$F4.exr
2024-03-18 05:35:43: 0: STDOUT: Driver: /out/Redshift_ROP1
2024-03-18 05:35:43: 0: STDOUT: Input File: /var/lib/Thinkbox/Deadline10/workers/ip-10-128-29-182/jobsData/65f7cc28874df71c50593303/rs_dl_render_test_B.hip
2024-03-18 05:35:43: 0: STDOUT: Warnings were generated during load.
2024-03-18 05:35:43: 0: STDOUT: Error loading: /var/lib/Thinkbox/Deadline10/workers/ip-10-128-29-182/jobsData/65f7cc28874df71c50593303/rs_dl_render_test_B.hip
2024-03-18 05:35:43: 0: STDOUT: Warning: Bad node type found: Redshift_ROP in /out.
2024-03-18 05:35:43: 0: STDOUT:
2024-03-18 05:35:43: 0: STDOUT: /out/Redshift_ROP1:
2024-03-18 05:35:43: 0: STDOUT: Unknown channel(s) "f[12]" converted to spare parameter(s).
2024-03-18 05:35:43: 0: STDOUT:
2024-03-18 05:35:43: 0: STDOUT: /out/Redshift_ROP1:
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_IPR".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_IPRRV".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_IPRVP".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "renderpreview".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "trange".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "f".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "take".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_separator_rrs2".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_rrs2".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_mainSwitcher".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_group_mainTabSwitcherCameras".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_renderCamera".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_group_mainTabSwitcherCamerasRO".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_overrideCameraRes".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_overrideResScale".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_overrideRes".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "override_camerares".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "res_fraction".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "res_overridex".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "res_overridey".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_group_mainTabSwitcherCamerasOver".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_overscanMode".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_overscanData".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_group_mainTabSwitcherNetworks".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_renderNetworks".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_group_mainTabSwitcherMaterialsOvr".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_matOverride_enable".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_matOverride_mat".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_matOverride_exclude".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_defaultMaterialOverride".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_apply_stylesheets".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_group_mainTabSwitcherSeqRendering".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_updateMeshDeform".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_updateMeshNoReload".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_updateMeshNoReload_list".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_group_mainTabSwitcherOptions".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_initSimulations".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_multihreadLoader".
2024-03-18 05:35:43: 0: STDOUT: Skipping unrecognized parameter "RS_multihreadPacked".
2024-03-18 05:35:44: 0: STDOUT: Skipping unrecognized parameter "RS_fastPackedGeoExpansion".
2024-03-18 05:35:44: 0: STDOUT: -- Too many errors, additional errors were suppressed. --
2024-03-18 05:35:44: 0: STDOUT: The following node types are using incomplete asset definitions:
2024-03-18 05:35:44: 0: STDOUT: Object/rslightdome::2.0
2024-03-18 05:35:44: 0: STDOUT: Object/rslight
2024-03-18 05:35:44: 0: STDOUT: Driver/Redshift_IPR
2024-03-18 05:35:44: 0: STDOUT: Begin Path Mapping
2024-03-18 05:35:45: 0: STDOUT: b''
2024-03-18 05:35:45: 0: STDOUT: End Path Mapping
2024-03-18 05:35:45: 0: STDOUT: ROP type: merge
2024-03-18 05:35:45: 0: STDOUT: Rendering frame 238
2024-03-18 05:35:45: 0: STDOUT: Finished Rendering
2024-03-18 05:35:45: 0: INFO: Process exit code: 0
2024-03-18 05:35:45: 0: INFO: Finished Houdini Job
2024-03-18 05:35:45: 0: INFO: Sending EndTaskRequest to S3BackedCacheClient.
2024-03-18 05:35:45: 0: DEBUG: Request:
2024-03-18 05:35:45: 0: DEBUG: JobId: 65f7cc28874df71c50593303
2024-03-18 05:35:45: 0: Done executing plugin command of type 'Render Task'
2024-03-18 05:35:45: 0: Render time for frame(s): 3.831 s
2024-03-18 05:35:45: 0: Total time for task: 4.353 s
2024-03-18 05:35:46: 0: Saving task log...
2024-03-18 05:35:46: 0: Render Thread - Render State transition from = 'Rendering' to = 'WaitingForTask'
2024-03-18 05:35:46: Scheduler Thread - Render Thread 0 completed its task
2024-03-18 05:35:46: Scheduler Thread - Scheduler State transition from = 'PreRendering' to = 'PostRendering'
2024-03-18 05:35:46: Scheduler Thread - Scheduler State transition from = 'PostRendering' to = 'EndJob'
2024-03-18 05:35:46: Scheduler Thread - Scheduler State transition from = 'EndJob' to = 'WaitingForJob'
2024-03-18 05:35:46: Scheduler Thread - Seconds before next job scan: 1
2024-03-18 05:35:46: Port Forwarder (houdini:1715): Client connected to port forwarder.
2024-03-18 05:35:47: Scheduler Thread - Scheduler State transition from = 'WaitingForJob' to = 'LicenseCheck'
2024-03-18 05:35:47: Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2024-03-18 05:35:47: Scheduler - Returning limit stubs not in use.
2024-03-18 05:35:47: Scheduler Thread - Job's Limit Groups: houdini, redshift
2024-03-18 05:35:47: Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'LicenseCheck'
2024-03-18 05:35:47: Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2024-03-18 05:35:47: Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'StartJob'
2024-03-18 05:35:47: 0: Render Thread - Render State transition from = 'WaitingForTask' to = 'ReceivedTask'
2024-03-18 05:35:47: Scheduler Thread - Scheduler State transition from = 'StartJob' to = 'PreRendering'
2024-03-18 05:35:47: 0: Got task!
2024-03-18 05:35:47: 0: Render Thread - Render State transition from = 'ReceivedTask' to = 'Other'
2024-03-18 05:35:47: 0: Plugin already loaded: Houdini
2024-03-18 05:35:47: 0: Start Job timeout is disabled.
2024-03-18 05:35:47: 0: Task timeout is disabled.
2024-03-18 05:35:47: 0: Job already loaded: rs_dl_render_test_B - /out/Redshift_ROP1
2024-03-18 05:35:47: 0: Plugin rendering frame(s): 239
2024-03-18 05:35:47: 0: Render Thread - Render State transition from = 'Other' to = 'Rendering'
2024-03-18 05:35:48: 0: Executing plugin command of type 'Render Task'
2024-03-18 05:35:48: 0: INFO: Sending StartTaskRequest to S3BackedCacheClient.
2024-03-18 05:35:48: 0: DEBUG: Request:
2024-03-18 05:35:48: 0: DEBUG: JobId: 65f7cc28874df71c50593303
2024-03-18 05:35:48: 0: DEBUG: JobUploadWhitelist: testB.####.exr
2024-03-18 05:35:48: 0: DEBUG: JobUploadWhitelistRe: ^.+\.abc$, ^.+\.avi$, ^.+\.bmp$, ^.+\.bw$, ^.+\.cin$, ^.+\.cjp$, ^.+\.cjpg$, ^.+\.cxr$, ^.+\.dds$, ^.+\.dpx$, ^.+\.dwf$, ^.+\.dwfx$, ^.+\.dwg$, ^.+\.dxf$, ^.+\.dxx$, ^.+\.eps$, ^.+\.exr$, ^.+\.fbx$, ^.+\.fxr$, ^.+\.hdr$, ^.+\.icb$, ^.+\.iff$, ^.+\.iges$, ^.+\.igs$, ^.+\.int$, ^.+\.inta$, ^.+\.iris$, ^.+\.jpe$, ^.+\.jpeg$, ^.+\.jpg$, ^.+\.jp2$, ^.+\.mcc$, ^.+\.mcx$, ^.+\.mov$, ^.+\.mxi$, ^.+\.pdf$, ^.+\.pic$, ^.+\.png$, ^.+\.prt$, ^.+\.ps$, ^.+\.psd$, ^.+\.rgb$, ^.+\.rgba$, ^.+\.rla$, ^.+\.rpf$, ^.+\.sat$, ^.+\.sgi$, ^.+\.stl$, ^.+\.sxr$, ^.+\.targa$, ^.+\.tga$, ^.+\.tif$, ^.+\.tiff$, ^.+\.tim$, ^.+\.vda$, ^.+\.vrimg$, ^.+\.vrmesh$, ^.+\.vrsm$, ^.+\.vrst$, ^.+\.vst$, ^.+\.wmf$, ^.+\.ass$, ^.+\.gz$, ^.+\.ifd$, ^.+\.mi$, ^.+\.mi2$, ^.+\.mxi$, ^.+\.rib$, ^.+\.rs$, ^.+\.vrscene$
2024-03-18 05:35:48: 0: DEBUG: S3BackedCache Client Returned Sequence: 3
I don’t know next step and struggle.
Please advice for this problem.
Best Regards
Taka
This error shows Redshift is not being loaded in Houdini 19.5.640, the nodes are replaced with generic ones.
Do you have redshift configured in the ‘packages’ json or houdini.env?
Thank you for the advice.
Then I installed redshift ver.3.5.19 on the worker and make ~/houdini19.5/houdini.env and rewrite houdini.env.
HOUDINI_PATH="/usr/redshift/redshift4houdini/19.5.640;&"
But rendering status is queued and waiting start.
This is a worker log error.
2024-03-19 05:53:49: Scheduler Thread - Scheduler State transition from = 'PreRendering' to = 'PostRendering'
2024-03-19 05:53:49: Scheduler Thread - Scheduler State transition from = 'PostRendering' to = 'EndJob'
2024-03-19 05:53:49: Scheduler Thread - Scheduler State transition from = 'EndJob' to = 'WaitingForJob'
2024-03-19 05:53:49: Scheduler Thread - Seconds before next job scan: 1
2024-03-19 05:53:50: Listener Thread - ::ffff:10.128.2.4 has connected
2024-03-19 05:53:50: Listener Thread - Received message: StreamLog
2024-03-19 05:53:51: Scheduler Thread - Scheduler State transition from = 'WaitingForJob' to = 'LicenseCheck'
2024-03-19 05:53:51: Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2024-03-19 05:53:51: Scheduler - Returning limit stubs not in use.
2024-03-19 05:53:51: Scheduler Thread - Job's Limit Groups: houdini, redshift
2024-03-19 05:53:51: Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'LicenseCheck'
2024-03-19 05:53:51: Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2024-03-19 05:53:51: Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'StartJob'
2024-03-19 05:53:51: 0: Render Thread - Render State transition from = 'WaitingForTask' to = 'ReceivedTask'
2024-03-19 05:53:51: Scheduler Thread - Scheduler State transition from = 'StartJob' to = 'PreRendering'
2024-03-19 05:53:51: 0: Got task!
2024-03-19 05:53:51: 0: Render Thread - Render State transition from = 'ReceivedTask' to = 'Other'
2024-03-19 05:53:51: 0: Loading Job's Plugin timeout is Disabled
2024-03-19 05:53:51: 0: SandboxedPlugin: Render Job As User disabled, running as current user 'ec2-user'
2024-03-19 05:53:52: '/home/ec2-user/Thinkbox/Deadline10/pythonAPIs/8oI2s+G2DZfcng9zEWRA==' already exists. Skipping extraction of PythonSync.
2024-03-19 05:53:53: 0: Loaded plugin Houdini
2024-03-19 05:53:53: All job files are already synchronized
2024-03-19 05:53:53: Plugin Houdini was already synchronized.
2024-03-19 05:53:53: 0: Executing plugin command of type 'Initialize Plugin'
2024-03-19 05:53:53: 0: INFO: Executing plugin script '/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/plugins/65f9274b874df71c505934ff/Houdini.py'
2024-03-19 05:53:53: 0: INFO: Plugin execution sandbox using Python version 3
2024-03-19 05:53:53: 0: INFO: About: Houdini Plugin for Deadline
2024-03-19 05:53:53: 0: INFO: The job's environment will be merged with the current environment before rendering
2024-03-19 05:53:53: 0: Done executing plugin command of type 'Initialize Plugin'
2024-03-19 05:53:53: 0: Start Job timeout is disabled.
2024-03-19 05:53:53: 0: Task timeout is disabled.
2024-03-19 05:53:53: 0: Loaded job: rs_dl_render_test_B - /out/Redshift_ROP1 (65f9274b874df71c505934ff)
2024-03-19 05:53:53: 0: Executing plugin command of type 'Start Job'
2024-03-19 05:53:53: 0: INFO: Sending StartTaskRequest to S3BackedCacheClient.
2024-03-19 05:53:53: 0: DEBUG: Request:
2024-03-19 05:53:53: 0: DEBUG: JobId: 65f9274b874df71c505934ff
2024-03-19 05:53:53: 0: DEBUG: JobUploadWhitelist: testB.####.exr
2024-03-19 05:53:53: 0: DEBUG: JobUploadWhitelistRe: ^.+\.abc$, ^.+\.avi$, ^.+\.bmp$, ^.+\.bw$, ^.+\.cin$, ^.+\.cjp$, ^.+\.cjpg$, ^.+\.cxr$, ^.+\.dds$, ^.+\.dpx$, ^.+\.dwf$, ^.+\.dwfx$, ^.+\.dwg$, ^.+\.dxf$, ^.+\.dxx$, ^.+\.eps$, ^.+\.exr$, ^.+\.fbx$, ^.+\.fxr$, ^.+\.hdr$, ^.+\.icb$, ^.+\.iff$, ^.+\.iges$, ^.+\.igs$, ^.+\.int$, ^.+\.inta$, ^.+\.iris$, ^.+\.jpe$, ^.+\.jpeg$, ^.+\.jpg$, ^.+\.jp2$, ^.+\.mcc$, ^.+\.mcx$, ^.+\.mov$, ^.+\.mxi$, ^.+\.pdf$, ^.+\.pic$, ^.+\.png$, ^.+\.prt$, ^.+\.ps$, ^.+\.psd$, ^.+\.rgb$, ^.+\.rgba$, ^.+\.rla$, ^.+\.rpf$, ^.+\.sat$, ^.+\.sgi$, ^.+\.stl$, ^.+\.sxr$, ^.+\.targa$, ^.+\.tga$, ^.+\.tif$, ^.+\.tiff$, ^.+\.tim$, ^.+\.vda$, ^.+\.vrimg$, ^.+\.vrmesh$, ^.+\.vrsm$, ^.+\.vrst$, ^.+\.vst$, ^.+\.wmf$, ^.+\.ass$, ^.+\.gz$, ^.+\.ifd$, ^.+\.mi$, ^.+\.mi2$, ^.+\.mxi$, ^.+\.rib$, ^.+\.rs$, ^.+\.vrscene$
2024-03-19 05:53:53: 0: DEBUG: S3BackedCache Client Returned Sequence: 1
2024-03-19 05:53:53: 0: INFO: Executing global asset transfer preload script '/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/plugins/65f9274b874df71c505934ff/GlobalAssetTransferPreLoad.py'
2024-03-19 05:53:53: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2024-03-19 05:53:53: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in /opt/Thinkbox/S3BackedCache/bin/task.py...
2024-03-19 05:53:53: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2024-03-19 05:53:53: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2024-03-19 05:53:53: 0: Done executing plugin command of type 'Start Job'
2024-03-19 05:53:53: 0: Plugin rendering frame(s): 1
2024-03-19 05:53:53: 0: Render Thread - Render State transition from = 'Other' to = 'Rendering'
2024-03-19 05:53:53: 0: Executing plugin command of type 'Render Task'
2024-03-19 05:53:53: 0: INFO: Sending StartTaskRequest to S3BackedCacheClient.
2024-03-19 05:53:53: 0: DEBUG: Request:
2024-03-19 05:53:53: 0: DEBUG: JobId: 65f9274b874df71c505934ff
2024-03-19 05:53:53: 0: DEBUG: JobUploadWhitelist: testB.####.exr
2024-03-19 05:53:53: 0: DEBUG: JobUploadWhitelistRe: ^.+\.abc$, ^.+\.avi$, ^.+\.bmp$, ^.+\.bw$, ^.+\.cin$, ^.+\.cjp$, ^.+\.cjpg$, ^.+\.cxr$, ^.+\.dds$, ^.+\.dpx$, ^.+\.dwf$, ^.+\.dwfx$, ^.+\.dwg$, ^.+\.dxf$, ^.+\.dxx$, ^.+\.eps$, ^.+\.exr$, ^.+\.fbx$, ^.+\.fxr$, ^.+\.hdr$, ^.+\.icb$, ^.+\.iff$, ^.+\.iges$, ^.+\.igs$, ^.+\.int$, ^.+\.inta$, ^.+\.iris$, ^.+\.jpe$, ^.+\.jpeg$, ^.+\.jpg$, ^.+\.jp2$, ^.+\.mcc$, ^.+\.mcx$, ^.+\.mov$, ^.+\.mxi$, ^.+\.pdf$, ^.+\.pic$, ^.+\.png$, ^.+\.prt$, ^.+\.ps$, ^.+\.psd$, ^.+\.rgb$, ^.+\.rgba$, ^.+\.rla$, ^.+\.rpf$, ^.+\.sat$, ^.+\.sgi$, ^.+\.stl$, ^.+\.sxr$, ^.+\.targa$, ^.+\.tga$, ^.+\.tif$, ^.+\.tiff$, ^.+\.tim$, ^.+\.vda$, ^.+\.vrimg$, ^.+\.vrmesh$, ^.+\.vrsm$, ^.+\.vrst$, ^.+\.vst$, ^.+\.wmf$, ^.+\.ass$, ^.+\.gz$, ^.+\.ifd$, ^.+\.mi$, ^.+\.mi2$, ^.+\.mxi$, ^.+\.rib$, ^.+\.rs$, ^.+\.vrscene$
2024-03-19 05:53:53: 0: DEBUG: S3BackedCache Client Returned Sequence: 1
2024-03-19 05:53:53: 0: INFO: Set HOUDINI_PATHMAP to {"/mnt/cgsv2021/":"/mnt/Data/mntcgsv2021386d7e3204ea8557cff0f48192a0df69/", "//cgsv2021":"/mnt/cgsv2021"}
2024-03-19 05:53:53: 0: INFO: Redshift Path Mapping...
2024-03-19 05:53:53: 0: INFO: source: "/mnt/cgsv2021/" dest: "/mnt/Data/mntcgsv2021386d7e3204ea8557cff0f48192a0df69/"
2024-03-19 05:53:53: 0: INFO: source: "\\cgsv2021" dest: "/mnt/cgsv2021"
2024-03-19 05:53:53: 0: INFO: [REDSHIFT_PATHOVERRIDE_FILE] now set to: "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/jobsData/65f9274b874df71c505934ff/RSMapping_temphnvI50/RSMapping.txt"
2024-03-19 05:53:53: 0: INFO: Starting Houdini Job
2024-03-19 05:53:53: 0: INFO: Stdout Redirection Enabled: True
2024-03-19 05:53:53: 0: INFO: Asynchronous Stdout Enabled: False
2024-03-19 05:53:53: 0: INFO: Stdout Handling Enabled: True
2024-03-19 05:53:53: 0: INFO: Popup Handling Enabled: True
2024-03-19 05:53:53: 0: INFO: QT Popup Handling Enabled: False
2024-03-19 05:53:53: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2024-03-19 05:53:53: 0: INFO: Using Process Tree: True
2024-03-19 05:53:53: 0: INFO: Hiding DOS Window: True
2024-03-19 05:53:53: 0: INFO: Creating New Console: False
2024-03-19 05:53:53: 0: INFO: Running as user: ec2-user
2024-03-19 05:53:54: 0: INFO: Executable: "/opt/hfs19.5.640/bin/hython"
2024-03-19 05:53:54: 0: CheckPathMapping: Swapped "//cgsv2021/projects02/AWS_Test/RS_deadline_test/render/testB/testB.$F4.exr" with "/mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/render/testB/testB.$F4.exr"
2024-03-19 05:53:54: 0: INFO: Argument: "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/plugins/65f9274b874df71c505934ff/hrender_dl.py" -f 1 1 1 -o "/mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/render/testB/testB.$F4.exr" -d /out/Redshift_ROP1 -tempdir "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/jobsData/65f9274b874df71c505934ff/0_tempBvHAZ0" -arnoldAbortOnLicenseFail 1 "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/jobsData/65f9274b874df71c505934ff/rs_dl_render_test_B.hip"
2024-03-19 05:53:54: 0: INFO: Full Command: "/opt/hfs19.5.640/bin/hython" "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/plugins/65f9274b874df71c505934ff/hrender_dl.py" -f 1 1 1 -o "/mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/render/testB/testB.$F4.exr" -d /out/Redshift_ROP1 -tempdir "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/jobsData/65f9274b874df71c505934ff/0_tempBvHAZ0" -arnoldAbortOnLicenseFail 1 "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/jobsData/65f9274b874df71c505934ff/rs_dl_render_test_B.hip"
2024-03-19 05:53:54: 0: INFO: Startup Directory: "/opt/hfs19.5.640/bin"
2024-03-19 05:53:54: 0: INFO: Process Priority: BelowNormal
2024-03-19 05:53:54: 0: INFO: Process Affinity: default
2024-03-19 05:53:54: 0: INFO: Process is now running
2024-03-19 05:53:54: 0: STDOUT: Houdini DSO error on '/usr/redshift/redshift4houdini/19.5.640/dso/redshift4houdini.so'
2024-03-19 05:53:54: 0: STDOUT: libXrandr.so.2: cannot open shared object file: No such file or directory
2024-03-19 05:53:54: 0: STDOUT: Houdini DSO error on '/opt/hfs19.5.640/houdini/dso/COP2_EnableGPU.so'
2024-03-19 05:53:54: 0: STDOUT: libXrandr.so.2: cannot open shared object file: No such file or directory
2024-03-19 05:53:54: 0: STDOUT: Houdini DSO error on '/opt/hfs19.5.640/houdini/dso/CHOP_Mouse3D.so'
2024-03-19 05:53:54: 0: STDOUT: libXrandr.so.2: cannot open shared object file: No such file or directory
2024-03-19 05:53:54: 0: STDOUT: Houdini DSO error on '/opt/hfs19.5.640/houdini/dso/COP2_GPUFog.so'
2024-03-19 05:53:54: 0: STDOUT: libXrandr.so.2: cannot open shared object file: No such file or directory
2024-03-19 05:53:54: 0: STDOUT: Houdini DSO error on '/opt/hfs19.5.640/houdini/dso/COP2_GPUEnvironment.so'
2024-03-19 05:53:54: 0: STDOUT: libXrandr.so.2: cannot open shared object file: No such file or directory
2024-03-19 05:53:54: 0: STDOUT: Houdini DSO error on '/opt/hfs19.5.640/houdini/dso/COP2_GPUZComposite.so'
2024-03-19 05:53:54: 0: STDOUT: libXrandr.so.2: cannot open shared object file: No such file or directory
2024-03-19 05:53:54: 0: STDOUT: Houdini DSO error on '/opt/hfs19.5.640/houdini/dso/COP2_GPULighting.so'
2024-03-19 05:53:54: 0: STDOUT: libXrandr.so.2: cannot open shared object file: No such file or directory
2024-03-19 05:53:54: 0: STDOUT: Houdini DSO error on '/opt/hfs19.5.640/houdini/dso/SOP_VDBUI.so'
2024-03-19 05:53:54: 0: STDOUT: libXrandr.so.2: cannot open shared object file: No such file or directory
2024-03-19 05:53:54: 0: STDOUT: Houdini DSO error on '/opt/hfs19.5.640/houdini/dso/OBJ_ReLight.so'
2024-03-19 05:53:54: 0: STDOUT: libXrandr.so.2: cannot open shared object file: No such file or directory
2024-03-19 05:53:54: 0: STDOUT: Houdini DSO error on '/opt/hfs19.5.640/houdini/dso/ROP_OpenGL.so'
2024-03-19 05:53:54: 0: STDOUT: libXrandr.so.2: cannot open shared object file: No such file or directory
2024-03-19 05:53:54: 0: STDOUT: Houdini DSO error on '/opt/hfs19.5.640/houdini/dso/SOP_Substance.so'
2024-03-19 05:53:54: 0: STDOUT: libXrandr.so.2: cannot open shared object file: No such file or directory
2024-03-19 05:53:54: 0: STDOUT: Houdini DSO error on '/opt/hfs19.5.640/houdini/dso/SOP_InstantMeshes.so'
2024-03-19 05:53:54: 0: STDOUT: libXrandr.so.2: cannot open shared object file: No such file or directory
2024-03-19 05:53:55: 0: STDOUT: Detected Houdini version: (19, 5, 640)
2024-03-19 05:53:55: 0: STDOUT: ['/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/plugins/65f9274b874df71c505934ff/hrender_dl.py', '-f', '1', '1', '1', '-o', '/mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/render/testB/testB.$F4.exr', '-d', '/out/Redshift_ROP1', '-tempdir', '/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/jobsData/65f9274b874df71c505934ff/0_tempBvHAZ0', '-arnoldAbortOnLicenseFail', '1', '/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/jobsData/65f9274b874df71c505934ff/rs_dl_render_test_B.hip']
2024-03-19 05:53:55: 0: STDOUT: Start: 1
2024-03-19 05:53:55: 0: STDOUT: End: 1
2024-03-19 05:53:55: 0: STDOUT: Increment: 1
2024-03-19 05:53:55: 0: STDOUT: Ignore Inputs: False
2024-03-19 05:53:55: 0: STDOUT: Output: /mnt/cgsv2021/projects02/AWS_Test/RS_deadline_test/render/testB/testB.$F4.exr
2024-03-19 05:53:55: 0: STDOUT: Driver: /out/Redshift_ROP1
2024-03-19 05:53:55: 0: STDOUT: Input File: /var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/jobsData/65f9274b874df71c505934ff/rs_dl_render_test_B.hip
2024-03-19 05:53:55: 0: Sending kill command to process hython-bin with id: 9464
2024-03-19 05:53:55: 0: INFO: Sending EndTaskRequest to S3BackedCacheClient.
2024-03-19 05:53:55: 0: DEBUG: Request:
2024-03-19 05:53:55: 0: DEBUG: JobId: 65f9274b874df71c505934ff
2024-03-19 05:53:55: 0: Done executing plugin command of type 'Render Task'
2024-03-19 05:53:55: ERROR: 0: An exception occurred: FailRenderException : Error: Caught exception: The attempted operation failed.
2024-03-19 05:53:55: at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
2024-03-19 05:53:55: File "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/plugins/65f9274b874df71c505934ff/Houdini.py", line 412, in HandleStdoutError
2024-03-19 05:53:55: self.FailRender(self.GetRegexMatch(1))
2024-03-19 05:53:55: at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
2024-03-19 05:53:55: at __FranticX_Processes_ManagedProcess_StdoutHandlerDelegateDispatcher.Invoke()
2024-03-19 05:53:55: at FranticX.Processes.ManagedProcess.RegexHandlerCallback.CallFunction()
2024-03-19 05:53:55: at FranticX.Processes.ManagedProcess.e(String cj, Boolean ck)
2024-03-19 05:53:55: at FranticX.Processes.ManagedProcess.Execute(Boolean waitForExit)
2024-03-19 05:53:55: at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
2024-03-19 05:53:55: at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2024-03-19 05:53:55: at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel) (Deadline.Plugins.RenderPluginException)
2024-03-19 05:53:55: 0: Unloading plugin: Houdini
2024-03-19 05:53:55: 0: Executing plugin command of type 'End Job'
2024-03-19 05:53:55: 0: INFO: Sending EndTaskRequest to S3BackedCacheClient.
2024-03-19 05:53:55: 0: DEBUG: Request:
2024-03-19 05:53:55: 0: DEBUG: JobId: 65f9274b874df71c505934ff
2024-03-19 05:53:55: 0: Done executing plugin command of type 'End Job'
2024-03-19 05:53:57: Port Forwarder (houdini:1715): Client connected to port forwarder.
2024-03-19 05:53:57: Sending kill command to process deadlinesandbox.exe with id: 9406
2024-03-19 05:53:59: 0: Render Thread - Render State transition from = 'Rendering' to = 'WaitingForTask'
2024-03-19 05:53:59: Scheduler Thread - Render Thread 0 threw a major error:
2024-03-19 05:53:59: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024-03-19 05:53:59: Exception Details
2024-03-19 05:53:59: RenderPluginException -- FailRenderException : Error: Caught exception: The attempted operation failed.
2024-03-19 05:53:59: at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
2024-03-19 05:53:59: File "/var/lib/Thinkbox/Deadline10/workers/ip-10-128-62-183/plugins/65f9274b874df71c505934ff/Houdini.py", line 412, in HandleStdoutError
2024-03-19 05:53:59: self.FailRender(self.GetRegexMatch(1))
2024-03-19 05:53:59: at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
2024-03-19 05:53:59: at __FranticX_Processes_ManagedProcess_StdoutHandlerDelegateDispatcher.Invoke()
2024-03-19 05:53:59: at FranticX.Processes.ManagedProcess.RegexHandlerCallback.CallFunction()
2024-03-19 05:53:59: at FranticX.Processes.ManagedProcess.e(String cj, Boolean ck)
2024-03-19 05:53:59: at FranticX.Processes.ManagedProcess.Execute(Boolean waitForExit)
2024-03-19 05:53:59: at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
2024-03-19 05:53:59: at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2024-03-19 05:53:59: at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2024-03-19 05:53:59: RenderPluginException.Cause: JobError (2)
2024-03-19 05:53:59: RenderPluginException.Level: Major (1)
2024-03-19 05:53:59: RenderPluginException.HasSlaveLog: True
2024-03-19 05:53:59: RenderPluginException.SlaveLogFileName: /var/log/Thinkbox/Deadline10/deadlineslave_renderthread_0-ip-10-128-62-183-0000.log
2024-03-19 05:53:59: Exception.TargetSite: Deadline.Slaves.Messaging.PluginResponseMemento d(Deadline.Net.DeadlineMessage, System.Threading.CancellationToken)
2024-03-19 05:53:59: Exception.Data: ( )
2024-03-19 05:53:59: Exception.Source: deadline
2024-03-19 05:53:59: Exception.HResult: -2146233088
2024-03-19 05:53:59: Exception.StackTrace:
2024-03-19 05:53:59: at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgm, CancellationToken bgn
2024-03-19 05:53:59: at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken
2024-03-19 05:53:59: at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajt, CancellationToken aju)
I read down below error. So, I do installed yum install libXrandr.i686. But there is not change.
‘/opt/hfs19.5.640/houdini/dso/COP2_GPULighting.so’
2024-03-19 05:53:54: 0: STDOUT: libXrandr.so.2: cannot open shared object file: No such file or directory
My worker has /opt/hfs19.5.640/houdini/dso/COP2_GPULighting.so and so on.
Please advice for me.
The issue is with the plugin configuration,
https://help.maxon.net/r3d/houdini/en-us/Content/html/Houdini+Plugin+Configuration.html
Your Houdini.env should contain the following…
HOUDINI_DSO_ERROR = 2
PATH = "/usr/redshift/bin:$PATH"
HOUDINI_PATH = "/usr/redshift/redshift4houdini/19.5.640;&"
PXR_PLUGINPATH_NAME="/usr/redshift/redshift4solaris/19.5.640;&"
i686 is the 32bit version of this and won’t be used
The issue is with the dependencies of dso libraries.
Try ldd /opt/hfs19.5.640/houdini/dso/COP2_GPULighting.so to see what it depends on, and what it can find.
The libraries might be in an unexpected location, or they may not be installed.
I don’t really know the CentOS packages. On Ubuntu I had to install libxcb-xinerama0 among others.
If you install these additional libs for Maya 2022 (e.g. Maya on CentOS 7.x), then you should be covered for most of the 3d apps and plugins on the AWS AMI (which is still CentOS 7 based).
I think you can just install the ones under “General” and “X Window - Xcb - X11” – those should be the ones needed by Redshift.
Additional required Linux libraries for Maya
You might need to install this for Redshift:
system-lsb-core.x86_64
If you do an ldd on redshift4houdini.so, then grep for “found” you should be able to see which libraries, if any, are missing.
[ec2-user@ip-172-30-3-148 ~]$ cd /usr/redshift/redshift4houdini/19.5.640/dso
[ec2-user@ip-172-30-3-148 dso]$ ls
redshift4houdini.so
[ec2-user@ip-172-30-3-148 dso]$ ldd redshift4houdini.so | grep -i "found"
It should display a list of files – if you are missing a lib, it should display with “not found”
e.g.
libasound.so.2 => not found
Thank you for the reply.
Yes. I understand your wxplanation.
But I don’t know the mapping setting.
My environment(machine: path) is down below.
client(windows): \cgsv2021
aws-portal(linux): /mnt/cgsv2021
worker(linux) on cloud: /mnt/Data/cgsv2021sdasd~~~~~~
configure repositry option -mapping path
Asset Server Settings
Configure Plugins (houdini)
Configure Plugins (redshift)
I add worker environment variable environment variable.
export environment variable=/home/ec2-user/override.txt
"/mnt/cgsv2021" "/mnt/Data/cgsv2021sda~~~~~~~~~"
I don’t know what I shoud do…
Do you have a idea?
It is local to the worker that is rendering. The output path should have been mapped by Deadline to the worker-specific location. Those mappings are created when you make a new infrastructure, and are tied to the infra id. You submit the job with /mnt/cgsv2021 and Deadline transforms this into its own path, which is an s3 backed cache.
These errors seem relevant
2024-03-22 03:16:53: 0: STDOUT: ls: cannot access /dev/disk/by-id/: No such file or directory
2024-03-22 03:16:53: 0: STDOUT: ls: cannot access /dev/disk/by-id/: No such file or directory
2024-03-22 03:16:53: 0: STDOUT: cat: /sys/devices/virtual/dmi/id/board_vendor: No such file or directory
2024-03-22 03:16:53: 0: STDOUT: cat: /sys/devices/virtual/dmi/id/board_name: No such file or directory
2024-03-22 03:16:53: 0: STDOUT: cat: /sys/devices/virtual/dmi/id/board_version: No such file or directory
Thank you for the reply.
Yes. I understand your wxplanation.
But I don’t know the mapping setting.
My environment(machine: path) is down below.
export environment variable=/home/ec2-user/override.txt
"/mnt/cgsv2021" "/mnt/Data/cgsv2021sda~~~~~~~~~"
I don’t know what I shoud do…
Do you have a idea?