Houdini 18.5.633 Precache

Deadline
1

I’m having some trouble getting Precache to work. I abandoned trying to make it work a year ago but now I’d like to fix it. I’ve rebuilt my entire deadline stack in an effort to make it work. I have one machine running as my RCS server, AWSPortalLink, and AWSPortalAsset server. And an additional server running my mongo database. I can run jobs on AWS fine except for that one piece. I hear there are some speed advantages and also it seems pointless to make the instance wait while I upload assets.

RCS / AWSPortal server is running Alma 8.4
I have full permissions on all the root directories I’m trying to pull from (and it works to upload on-the-fly)

I get a generic error in Houdini:

Error Sending Precache File Request: Status(StatusCode=Unavailable, Detail="")
Failed to push files for job 615caee374c500cb485d756c

and I get this in the AWSPortalAssetServer log:

1633465413.390918 2021-10-05 15:23:33,390 [/opt/Thinkbox/AWSPortalAssetServer/awsportalassetserver.py:PrecacheFil
es:182] [root] [530856] [ThreadPoolExecutor-0_6] [INFO] About to precache files for job 615cb44374c500cb485d7572
1633465413.448299 2021-10-05 15:23:33,448 [/opt/Thinkbox/AWSPortalAssetServer/s3backedcachelib/aws_s3_util.py:upl
oad_file:122] [root] [530856] [ThreadPoolExecutor-0_6] [INFO] Created metadata for filename: {'filename': '/tmp/t
mp10psriph/055c3bbb91cf4268adb61020604c36d0'
}
1633465413.448759 2021-10-05 15:23:33,448 [/opt/Thinkbox/AWSPortalAssetServer/s3backedcachelib/aws_s3_util.py:upload_file:131] [root] [530856] [ThreadPoolExecutor-0_6] [INFO] About to upload /tmp/tmp10psriph/055c3bbb91cf4268adb61020604c36d0 to bucket aws-portal-cache-493e15e0-6c30-42fd-8fba-54c9032bc7a1 at 2a1de645b55540b49a484396aadd15f4
1633465413.973219 2021-10-05 15:23:33,973 [/opt/Thinkbox/AWSPortalAssetServer/s3backedcachelib/aws_s3_util.py:upload_file:156] [root] [530856] [ThreadPoolExecutor-0_6] [INFO] Completed upload to aws-portal-cache-493e15e0-6c30-42fd-8fba-54c9032bc7a1 at 2a1de645b55540b49a484396aadd15f4 in 0.52 s (uploaded 1319 byte file in 0.52 s)
1633465414.135339 2021-10-05 15:23:34,135 [/opt/Thinkbox/AWSPortalAssetServer/awsportalassetserverlib/error_handling.py:wrapper:227] [root] [530856] [ThreadPoolExecutor-0_6] [ERROR] Failed to access GRPC endpoint: <StatusCode.UNKNOWN: (2, 'unknown')>: "Exception in central controller: a bytes-like object is required, not 'str'"
Traceback (most recent call last):
  File "/opt/Thinkbox/AWSPortalAssetServer/awsportalassetserverlib/error_handling.py", line 197, in wrapper
    return func(self, request, context)
  File "/opt/Thinkbox/AWSPortalAssetServer/awsportalassetserver.py", line 226, in PrecacheFiles
    timeout=180)
  File "/opt/Thinkbox/AWSPortalAssetServer/dependencies/grpc/_channel.py", line 492, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/opt/Thinkbox/AWSPortalAssetServer/dependencies/grpc/_channel.py", line 440, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Exception in central controller: a bytes-like object is required, not 'str')>

Is this a problem of something not resolving right in Houdini itself, like a preflight issue or something else like a generic python problem?

Thanks!
Anthony

2

This is likely a python 2 vs 3 issue, because in 3 by default subprocesses work with byte strings.
Cannot tell exactly where it came from, though, some remote communication.

1 Like
3

Yeah, I thought it would be a python version issue. 80% of the time… :slight_smile:
Would that be the case if the init script launches with the python3 folder shipped with deadline - found in /opt/Thinkbox/AWSPortalAssetServer? Or are you saying this may be a bug in the asset server itself caused by the transition to python3?
Thanks!

4

It’s hard to tell since I don’t know the system very well, never used the asset server.
Probably bug in the server.

5

Push. Anyone found a solution to this problem?

I jumped all the hoops so far but exactly this described problem comes up again and again.

The precache always fails to be submitted. Tried with a python 2 and a python 3 version of Houdini.

Stuff gets uploaded to the S3 Bucket but still the Spotfleet Render restarts the same frame all the time without any outcome.

6

Submit a ticket to Thinkbox… I’d do it but I have so many deadlines I can’t keep them straight at the moment. I think it’s a broken app.