Corrupted render EXR from cloud farm

Hello,

We are using a cloud render farm connect with a VPN to our network.
Sometime the render crash because an error happend ([texturesys] error from loading a texture for example).
The exr in the render output for this task is left “half written” and become corrupted.
We can not open it, delete it, copy it. It even make our explorer crash when trying to open the properties.
We tried through the command line or specialized tool with no success either.
Fortunatly, there is the “local rendering” option that avoid the probleme.
But still, if anyone has a tips to get rid of those corrupted file, i’m happy to hear !
thanks

When you’re trying to delete the file from a command line, how are you doing that, and what error do you get back?

I’d assume if you’ve got permission and no application has a lock on the file nothing would stop you from deleting a file.

We are having the same problem, it randomly fails without any reason, we can delete the file from our server, but it randomly pays again, we have a local render farm.

Our artists send the job and it automatically converts the textures to .TX, we believe that process collapses and corrupts when it tries to write.

2023-07-28 15:01:04:  0: STDOUT: 00:09:19  3140MB         |    40% done - 1029 rays/pixel
2023-07-28 15:01:16:  0: STDOUT: 00:09:32  3160MB         |    45% done - 627 rays/pixel
2023-07-28 15:01:43:  0: STDOUT: 00:09:59  3178MB         |    50% done - 1095 rays/pixel
2023-07-28 15:01:57:  0: STDOUT: 00:10:13  3186MB         |    55% done - 606 rays/pixel
2023-07-28 15:02:16:  0: STDOUT: 00:10:32  3210MB         |    60% done - 829 rays/pixel
2023-07-28 15:02:32:  0: STDOUT: 00:10:48  3225MB ERROR   |   [texturesys] Route_Route_color: Failed OpenEXR read: Error reading pixel data from image file "C:/Route/Route/Route/Route/Route/Route/Route/Route/Route.tx". Unexpected tile x coordinate.
2023-07-28 15:02:32:  0: STDOUT: 00:10:48  3225MB WARNING |   render terminating early:  received abort signal
2023-07-28 15:02:32:  0: STDOUT: 00:10:48  3225MB         |  render done in 4:19.352
2023-07-28 15:02:32:  0: STDOUT: 00:10:48  3225MB         |  [driver_exr] writing file `C:/Route/Route/Route/Route/Route/Route/Route/Route/Route/Route.1008.exr'
2023-07-28 15:02:32:  0: STDOUT: 00:10:48  3182MB         | render done
2023-07-28 15:02:32:  0: WARNING: Strict error checking on, ignoring the following unrecognized error or warning. If it is fatal, please email support@thinkboxsoftware.com with the error message.
2023-07-28 15:02:32:  0: STDOUT: Error: line 1: RuntimeError: [mtoa] Failed batch render
2023-07-28 15:02:32:  0: WARNING: Strict error checking on, ignoring the following unrecognized error or warning. If it is fatal, please email support@thinkboxsoftware.com with the error message.
2023-07-28 15:02:32:  0: STDOUT: Warning: file: C:/Program Files/Autodesk/Maya2022/scripts/others/mayaBatchRenderProcedure.mel line 654: Renderer returned an error while rendering 'defaultRenderLayer', please verify the output image.
2023-07-28 15:02:32:  0: STDOUT: Render completed.
2023-07-28 15:02:32:  0: STDOUT: mel: READY FOR INPUT
2023-07-28 15:02:32:  0: Done executing plugin command of type 'Render Task'

Hello @Sergic

Seem like the path of .TX file is corrupted or is it set like "C:/Route/Route/Route/Route/Route/Route/Route/Route/Route.tx or you may have purposely obfuscated it?

And did you mean that Maya converts the file to .tx or are you running a script to do that?

Hi sorry for the delay, no, I just changed the path for the sake of nda is just an example, what we have detected that maya generates these TX files, and when other nodes try to open the same scene what it does is to corrupt the renders, as mentioned above, we have achieved half a solution by making the .temp.tx files are not synchronized.

Hi, again we continue with errors in the renders daily since July 2023, when they send the maya scene randomly when nodes are assigned to the same task, the .tx textures fail.

Could you try isolating the render from Deadline using this guide from the knowledgebase?

I’m not sure what Deadline would be doing that causes 3dsMax to corrupt the .tx files, but it sounds like it should be reproduceable.