I try to use deadline on AWS but I can't

mois.moshev · March 19, 2024, 12:57pm

The issue is with the dependencies of dso libraries.
Try ldd /opt/hfs19.5.640/houdini/dso/COP2_GPULighting.so to see what it depends on, and what it can find.
The libraries might be in an unexpected location, or they may not be installed.
I don’t really know the CentOS packages. On Ubuntu I had to install libxcb-xinerama0 among others.

jarak · March 19, 2024, 5:19pm

If you install these additional libs for Maya 2022 (e.g. Maya on CentOS 7.x), then you should be covered for most of the 3d apps and plugins on the AWS AMI (which is still CentOS 7 based).

I think you can just install the ones under “General” and “X Window - Xcb - X11” – those should be the ones needed by Redshift.
Additional required Linux libraries for Maya

You might need to install this for Redshift:
system-lsb-core.x86_64

If you do an ldd on redshift4houdini.so, then grep for “found” you should be able to see which libraries, if any, are missing.

[ec2-user@ip-172-30-3-148 ~]$ cd /usr/redshift/redshift4houdini/19.5.640/dso

[ec2-user@ip-172-30-3-148 dso]$ ls
redshift4houdini.so

[ec2-user@ip-172-30-3-148 dso]$ ldd redshift4houdini.so | grep -i "found"

It should display a list of files – if you are missing a lib, it should display with “not found”
e.g.
libasound.so.2 => not found

hasekura.takamitsu · March 22, 2024, 3:29am

Thank you for the reply.
Yes. I understand your wxplanation.
But I don’t know the mapping setting.
My environment(machine: path) is down below.

client(windows): \cgsv2021
aws-portal(linux): /mnt/cgsv2021
worker(linux) on cloud: /mnt/Data/cgsv2021sdasd~~~~~~
configure repositry option -mapping path

1906×734 32.8 KB
Asset Server Settings

2898×623 15.3 KB
Configure Plugins (houdini)

houdini1017×744 40.3 KB
Configure Plugins (redshift)

redshift1016×745 25.1 KB
I add worker environment variable environment variable.
export environment variable=/home/ec2-user/override.txt

"/mnt/cgsv2021" "/mnt/Data/cgsv2021sda~~~~~~~~~"

I don’t know what I shoud do…
Do you have a idea?

mois.moshev · March 22, 2024, 8:48am

It is local to the worker that is rendering. The output path should have been mapped by Deadline to the worker-specific location. Those mappings are created when you make a new infrastructure, and are tied to the infra id. You submit the job with /mnt/cgsv2021 and Deadline transforms this into its own path, which is an s3 backed cache.

These errors seem relevant

2024-03-22 03:16:53:  0: STDOUT: ls: cannot access /dev/disk/by-id/: No such file or directory
2024-03-22 03:16:53:  0: STDOUT: ls: cannot access /dev/disk/by-id/: No such file or directory
2024-03-22 03:16:53:  0: STDOUT: cat: /sys/devices/virtual/dmi/id/board_vendor: No such file or directory
2024-03-22 03:16:53:  0: STDOUT: cat: /sys/devices/virtual/dmi/id/board_name: No such file or directory
2024-03-22 03:16:53:  0: STDOUT: cat: /sys/devices/virtual/dmi/id/board_version: No such file or directory

hasekura.takamitsu · March 22, 2024, 9:23am

Thank you for the reply.
Yes. I understand your wxplanation.
But I don’t know the mapping setting.
My environment(machine: path) is down below.

client(windows): \cgsv2021
aws-portal(linux): /mnt/cgsv2021
worker(linux) on cloud: /mnt/Data/cgsv2021sdasd~~~~~~
configure repositry option -mapping path

Asset Server Settings

Configure Plugins (houdini)

Configure Plugins (redshift)

I add worker environment variable environment variable.
export environment variable=/home/ec2-user/override.txt

"/mnt/cgsv2021" "/mnt/Data/cgsv2021sda~~~~~~~~~"

I don’t know what I shoud do…
Do you have a idea?

mois.moshev · March 25, 2024, 12:26pm

The windows path is suspect - look at the yen sign, this means there was an escaped character. You might need double backslashes. I’m not sure this is a problem, however (might just be a display issue).
You need to read the AWS Asset Server logs (/var/log/Thinkbox/AWSAssetServer/) to see whether the mapping works.

jarak · March 25, 2024, 10:35pm

I suspect that the path mapping is incorrect or not working correctly?

Why you are forcing override paths in the environment?

I add worker environment variable environment variable.
export environment variable=/home/ec2-user/override.txt

During these tests, you should let Deadline handle all the path mapping.

Please correct the trailing slash “/” or “\” in the mapped paths, just to rule out any replacement/regex strangeness.

Replace Path:  \\cgsv2021\
Linux Path:  /mnt/cgsv2021/

I had to dig around for an example render output – this is a few years old, but it demonstrates what the mapped paths are supposed to look like in the output.
Deadline Mapped Paths (Global Rules) is swapping out my local path (e.g. \\servershare\ ) with the S3 bucket path (e.g. /mnt/Data/myBucketIDIsALongString/ )

The path for the saved file will be on the EC2 instance with the remote path ((e.g. /mnt/Data/myBucketIDIsALongString/path/to/file )
and then that file will be sync’d back to your local path (e.g. \\servershare) via the Asset Server.

Example output edited to show the paths:


2022-10-25 22:38:57:  0: Job already loaded: sh0010_lighting_default.v035- Redshift- /out/sh0010_strands

2022-10-25 22:38:57:  0: INFO: Running as user: ec2-user
2022-10-25 22:38:57:  0: INFO: Executable: "/opt/plugins/redshift/3508/bin/redshiftCmdLine"

2022-10-25 22:38:57:  0: CheckPathMapping: Swapped "//servershare/00000_project_name_60/sh0010/renders/houdini/lighting/default.v035/strands/rs/sh0010_lighting_default.v035_strands.1180.rs" with "/mnt/Data/myBucketIDIsALongString/00000_project_name_60/sh0010/renders/houdini/lighting/default.v035/strands/rs/sh0010_lighting_default.v035_strands.1180.rs"

2022-10-25 22:38:57:  0: CheckPathMapping: Swapped "//servershare/00000_project_name_60/sh0010/renders/houdini/lighting/default.v035/strands" with "/mnt/Data/myBucketIDIsALongString/00000_project_name_60/sh0010/renders/houdini/lighting/default.v035/strands"

2022-10-25 22:38:57:  0: INFO: Argument: "/mnt/Data/myBucketIDIsALongString/00000_project_name_60/sh0010/renders/houdini/lighting/default.v035/strands/rs/sh0010_lighting_default.v035_strands.1011.rs" -oip "/mnt/Data/myBucketIDIsALongString/00000_project_name_60/sh0010/renders/houdini/lighting/default.v035/strands"

2022-10-25 22:38:57:  0: INFO: Full Command: "/opt/plugins/redshift/3508/bin/redshiftCmdLine" "/mnt/Data/myBucketIDIsALongString/00000_project_name_60/sh0010/renders/houdini/lighting/default.v035/strands/rs/sh0010_lighting_default.v035_strands.1011.rs" -oip "/mnt/Data/myBucketIDIsALongString/00000_project_name_60/sh0010/renders/houdini/lighting/default.v035/strands"


2022-10-25 22:38:58:  0: STDOUT: Loading: /mnt/Data/myBucketIDIsALongString/00000_project_name_60/sh0010/renders/houdini/lighting/default.v035/strands/rs/sh0010_lighting_default.v035_strands.1011.rs
2022-10-25 22:39:05:  0: STDOUT: License acquired
2022-10-25 22:50:53:  0: STDOUT: Rendering time: 11m:46s (1 GPU(s) used)

2022-10-25 22:50:54:  0: STDOUT: Saving: /mnt/Data/myBucketIDIsALongString/00000_project_name_60/sh0010/renders/houdini/lighting/default.v035/strands/sh0010_lighting_default.v035_strands.1011.crypto_material.exr
2022-10-25 22:50:56:  0: STDOUT: Saving: /mnt/Data/myBucketIDIsALongString/00000_project_name_60/sh0010/renders/houdini/lighting/default.v035/strands/sh0010_lighting_default.v035_strands.1011.exr

hasekura.takamitsu · March 26, 2024, 4:46am

Hi everyone,

I completely resolve this problem.
The rendering is working now!

Than you so much! I appreciate that all things.

Best Regards
Taka

mois.moshev · March 26, 2024, 9:18am

Yay! Please share the resolution with us, don’t leave us hanging

hasekura.takamitsu · March 26, 2024, 9:40am

Sorry for not sharing.

I change the mapping setting as jarak said.

before:

Replace Path:  \\cgsv2021
Linux Path:  /mnt/cgsv2021

after

Replace Path:  \\cgsv2021\
Linux Path:  /mnt/cgsv2021/

and I delete /home/ec2-user/override.txt on worker AMI.

Thank you for the advice.

Best regards
Taka