AWS Thinkbox Discussion Forums

AWS Asset Server going offline once Infrastructre is created

I have been trying to setup AWS Portal over the last few days and I’ve made some progress but have hit up against a problem I can’t seem to fix.

Both the AWS Portal Link and AWS Asset Server are showing as Online & green, however once I create an Infrastructure the AWS Asset Server goes offline. I assumed that this was a firewall/anti-virus issue at first, however disabling them doesn’t make any difference. I then assumed that there was a communication (or permission) error between the Asset Server and the S3 bucket, however if I manually upload files/folder to the S3 bucket through the console & stop the Infrastructure, selecting ‘empty the bucket’ does delete those files.

I can upload some of the log files but to be honest, I’m not sure which ones would be most useful to troubleshoot this!

Any help or advice on what to look into next would be greatly appreciated.

AJ

I have the exact same issue.

i’ve got this to work before and we’ve successfully rendered with aws portal and aws slaves, but recently the asset server start to fail to transfer assets to the ec2 platform, and now after upgrading deadline to 10.1.5.1, i’m getting the same exact error.

AWS Portal Link and AWS Asset Server both reporting online before creating an infrastructure.
ADter an infrastructure is created, AWS Portal Link is Online, but AWS Asset Server reports “Not Connected”

This is the repeated log from the console:
Error when attempting to communicate with the CentralServer on your Infrastructure: Status(StatusCode=DeadlineExceeded, Detail="Deadline Exceeded")

I can share my logs if you need too

I have a call with support later today to troubleshoot this. I will report back!

Unfortunately, despite a good hour of support today we’re still nowhere closer to fixing the issue. Have another call tomorrow, will keep you posted.

Hey AJ,

That’s great! thank you for helping also solve this thing together! haha.
I’ll keep myself poster here too.

Stay safe and take care!

-nic

If you’re on 10.1.5.1. try replacing "C:\Program Files (x86)\Thinkbox\AWSPortalLink\sshlib\command_factory.py" with command_factory.py (12.6 KB). that should resolve the issue with the Asset Server and Portal link not showing connected.

Be sure to stop the services before you make the swap, and start them back up again. Then try firing up another infrastructure. If that doesn’t work, please reach out to us by one of the options on the support page. We’d love to be quick and active in the forums, we’re just not there yet.

1 Like

Hi Justin,

Actually a member of your support team sent me the very same file and it has fixed the issue!

I have now been able to successfully submit and render on a Spot Fleet, so I’m very happy.

@Masonry_Studios - I would suggest following Justin’s advice on the command_factory.py file. Although I also had another issue where the DashKey on my local machine wasn’t permitting ssh connections so I had to manually delete and reinstall.

2 Likes

@AJ_Jefferies Hey! It worked! Omg what a fix this was. Thank you so much for letting me jump in on this thread. I’m happy to hear that you managed to get yours up and running too!

Now i’m off to trying and figure out how to create my own AMI. Cause the plugins on the default deadline AMIs are older than the ones we use. :confused:

THANKS ALL!

1 Like

Good luck! I’ve had to do the exact same thing, I followed Bobo’s video step by step and it worked perfectly : https://www.youtube.com/watch?v=rRI0mLBYoAA

Hi! :slight_smile:

I’ve been having this same problem, yet replacing the file didn’t work for me. I keep getting both the AWS Portal Link and AWS Asset Server offline when creating an infrastructure. Its the first time I try to run one.

Is there anything else I can try?

I would really appretiate your help. I have several days trying to set it up to run my operations on AWS.

Thank you in advance.

I’m also experiencing this issue with version 10.1.92. I have tried replacing the command_factory.py file and while that did get the AWSPortalLink installer to successfully create the DashKey and Dashkey.pub files, the same error: Error when attempting to communicate with the CentralServer on your Infrastructure: Status(StatusCode=DeadlineExceeded, Detail="Deadline Exceeded")

@Luis_Daniel_Morgado did you ever end up finding a fix for this?

Could you share the whole log you’re seeing those errors in? I haven’t seen that out of 10.1.9.2 AWSPortal, so it’s possible there’s something else going on.

Hi Justine,
I have the same problem that the asset server went offline after setup infrastructure. I’m using Deadline Version (ex: 10.0.3.2): Deadline -10.1.10.6. I have logged the case with AWS but no response so far. It has been 2 days.

any advice?

Hey Nigel,

I’m not sure what’s gone wrong on the Premium Support side of things but let us worry about that.

Looking at the screenshots the issue looks like you don’t have an S3 bucket (there should be info in the “Asset Server S3 Bucket” field).

Stop your infrastructure and go to Tools->Configure Asset Server and hit ‘Create New Bucket’. Then remove the existing root directory entry and hit ok. Then go back into Configure Asset Server and add it again, this will run an end-to-end test of the root directory added. If it fails, go to %PROGRAMDATA%\Thinkbox\AWSPortalAssetServer\logs\ and check that log for errors.

If it does fail, my first guess would be that the D: drive is a network drive and needs to be added under Tools->Configure Repository Options->Mapped Drives.

I’m going to be on vacation for the rest of the week so I won’t be keeping an eye on the forums. So maybe send your logs and findings into a ticket at https://awsthinkbox.zendesk.com/hc/en-us. Or call us at 1-866-419-0283 between 9-5CST.

Hi, I am having the same problems, I have started to install and configure the asset server, in the image I have put the ip of the server that hosts deadline and asset server and portal, we are using Deadline 10.2.0.10

-From asset server we are using a route by smb hosted on a Nas system.
-I don’t know if there is an extra port to open or if something is blocking.

AWSPortalAssetServer Logs:


1676465197.398179 2023-02-15 13:46:37,398 [C:\Program Files (x86)\Thinkbox\AWSPortalAssetServer\awsportalassetserverlib\share_util.py:refresh_shares:94] [root] [3748] [Dummy-1] [INFO] Refreshing shares list.
1676465197.399192 2023-02-15 13:46:37,399 [C:\Program Files (x86)\Thinkbox\AWSPortalAssetServer\awsportalassetserverlib\share_util.py:refresh_shares:101] [root] [3748] [Dummy-1] [INFO] Share: Path: //192.168.1.119/aws/ Id: 192.168.1.119aws20a33cd34f86bd40cf681c6b073556ba
1676465201.715935 2023-02-15 13:46:41,715 [C:\Program Files (x86)\Thinkbox\AWSPortalAssetServer\awsportalassetserver.py:get_and_set_ip_address:88] [root] [3748] [Dummy-1] [INFO] 'C:\Users\Administrador\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2022-11-22T212046.0000000Z' already exists. Skipping extraction of PythonSync.
IPAddress set to 192.168.1.120

AwsPortalLink


1676466627.524184 2023-02-15 14:10:27,524 [root] [4836] [MainThread] [INFO] GetTunnelParams.py started...
1676466627.524184 2023-02-15 14:10:27,524 [root] [4836] [MainThread] [INFO] Using INSTALL_LOCATION - 'C:\\Program Files (x86)\\Thinkbox\\AWSPortalLink'
1676466628.000187 2023-02-15 14:10:28,000 [root] [4836] [MainThread] [INFO] Getting proxyIP and proxyPort
1676466628.089182 2023-02-15 14:10:28,089 [root] [4836] [MainThread] [INFO] Loading the gateway IPs...
1676466628.137215 2023-02-15 14:10:28,137 [root] [4836] [MainThread] [INFO] Loading the configured tunnel from config.json...
1676466628.139213 2023-02-15 14:10:28,139 [root] [4836] [MainThread] [INFO] Loading additional servers settings...
1676466628.143183 2023-02-15 14:10:28,143 [root] [4836] [MainThread] [INFO] Save the AWSPortal Settings
1676466628.228184 2023-02-15 14:10:28,228 [root] [4836] [MainThread] [INFO] Saving output to: C:\Users\ADMINI~1\AppData\Local\Temp\tmp_hd9vf_j.out
1676466628.230226 2023-02-15 14:10:28,230 [root] [4836] [MainThread] [INFO] GetTunnelParams.py complete.

image

Cloudwatch logs instance for awsportal

Wed Feb 15 13:23:03 UTC 2023 -- Gateway configured to connect to RCS via HTTPS with SSL verification disabled.
Wed Feb 15 13:21:02 UTC 2023 -- Could not connect to the RCS. Connection response: '<html><head><title>502 Bad Gateway</title></head><body bgcolor="white"><center><h1>502 Bad Gateway</h1></center><hr><center>nginx/1.12.2</center></body></html>'

Good, I have managed to advance a little trying multiple things and my problem has been solved, the case is that I had to use 2 commands to correct the permissions of the key of the structure, as it says in the documentation of deadline.
https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/aws-portal-troubleshooting/aws-portal-link-error-messages.html

Linux:
sudo chmod 600 /opt/Thinkbox/AWSPortalLink/DashKey

Windows:
icacls “%PROGRAMFILES(X86)%”\Thinkbox/AWSPortalLink/DashKey /inheritance:r
icacls “%PROGRAMFILES(X86)%”\Thinkbox\AWSPortalLink/DashKey /grant:r “%username%”:"(R)"

Translated with DeepL

1 Like
Privacy | Site terms | Cookie preferences