Cinema4d Octane Plugin vdb files unable to delete

Deadline
1

I have a big issue where deadline can not delete the temp files created for each frame for vdb files used in a C4d / Octane render on nodes. Job is submitted through C4d submission as a c4d batch job. Deadline version is the latest:

Deadline Client Version: 10.1.23.6 Release  (773a6289d)
FranticX Client Version: 2.4.0.0 Release  (14916bbc6)

License Mode: 

Repository Version: 10.1.23.6  (773a6289d)
Integration Version: 10.1.23.6  (773a6289d)
3PL Settings Version: 22/07/2022

I have overriden all folder restrictions for read / write / full control and the temp files are not showing read only flags. Run as user is not enabled in the repository.
After the vdb files are copied to the nodes, c:\users\Current_User\App Data\Local\Temp they can not be deleted by the worker and so fill the HDD until the workers stall.
The renders run and complete, but the temp files remain, even after the task completes.
Log section:

2022-10-12 09:11:58:  0: STDOUT: VRAM used/free/max:0Kb/10Gb/10Gb  Out-of-core used:0Kb  RAM used:8.474Gb total:63.915Gb  OpenGL free/total:0/0
2022-10-12 09:11:58:  0: STDOUT: Running Script: C:\ProgramData\Thinkbox\Deadline10\workers\PA-OCTANE04\jobsData\6345a3d4c845d274e0899152\thread0_tempAFcmY0\c4d_Batch_Script.py
2022-10-12 09:11:58:  0: STDOUT: straightAlpha:0
2022-10-12 09:11:58:  0: STDOUT: *** pBufType:1 pColorSpace:1
2022-10-12 09:11:59:  0: STDOUT: WARNING: failed to remove temporary file C:\Users\octane04\AppData\Local\Temp\s998.0 (permission denied)
2022-10-12 09:11:59:  0: STDOUT: WARNING: failed to remove temporary file C:\Users\octane04\AppData\Local\Temp\s998.1 (permission denied)
2022-10-12 09:11:59:  0: STDOUT: WARNING: failed to remove temporary file C:\Users\octane04\AppData\Local\Temp\s998.2 (permission denied)
2022-10-12 09:11:59:  0: STDOUT: WARNING: failed to remove temporary file C:\Users\octane04\AppData\Local\Temp\s998.3 (permission denied)
2022-10-12 09:11:59:  0: STDOUT: WARNING: failed to remove temporary file C:\Users\octane04\AppData\Local\Temp\s998.4 (permission denied)
2022-10-12 09:12:00:  0: STDOUT: WARNING: failed to remove temporary file C:\Users\octane04\AppData\Local\Temp\s998.5 (permission denied)
2022-10-12 09:12:00:  0: STDOUT: WARNING: failed to remove temporary file C:\Users\octane04\AppData\Local\Temp\s998.6 (permission denied)
2022-10-12 09:12:00:  0: STDOUT: WARNING: failed to remove temporary file C:\Users\octane04\AppData\Local\Temp\s998.7 (permission denied)
2022-10-12 09:12:00:  0: STDOUT: WARNING: failed to remove temporary file C:\Users\octane04\AppData\Local\Temp\s998.8 (permission denied)
2022-10-12 09:12:00:  0: STDOUT: Rendering frame 4 at <Wed Oct 12 09:12:00 2022>
2022-10-12 09:12:00:  0: STDOUT:  
2022-10-12 09:12:00:  0: STDOUT: FRAME:4 fps:25

My only suggestion now after testing various options, is the files are in use when the Worker tries to delete them. Moving the delete code to the end of the render completion might solve, but I can’t find this in the c4d_batch.py sent to the worker or in the C4d submission plugin .py file.

This is becoming a nightmare as some frames have 10 vdb files, which create a 21Mb temp file for each, each frame = 210Mb per frame and we run in excess of 5000 frames per job.

Any help would be greatly appreciated.
(We have been an existing customer for years and have just moved to .23 from .18 where this didn’t seem to be an issue.)

2

Hello @Tim_Ellis1

Thanks for reaching out. We appreciate the fact that you are using Deadline for a long time. :slight_smile:

So you think when you upgrade Deadline from 10.1.18.x to 10.1.23.x this started to happen?

I think a simple job can test the permissions for the user running the Worker:

  • From Deadline Monitor go to Submit> Misc. > Command Script> in the machine limit just select one machine to test.
  • In the big black box: Commands to Execute: type in a command which can create a file like:
    echo This is a sample text file > C:\Users\octane04\AppData\Local\Temp\sample. txt
  • Enable the option “execute in shell” and then select cmd from the drop down.
  • Check if the file gets created then submit another job to delete it, use below in the command to execute to delete it:
    del C:\Users\octane04\AppData\Local\Temp\sample. txt, check if it is deleted.
3

Hi @zainali Thank you. Both jobs ran successfully, creation of the file and the deletion. Local folder checked for file existance, then ran the second job and it deleted the file.

I also wanted to mention, this is a new server with a fresh install of Deadline and repo, I didn’t upgrade v.18 in place in the old repo.

4

@zainali Any further suggestions with this issue please?

Many thanks, Tim

5

Hey @Tim_Ellis1

I need to dive deeper in the C4D plugin. It might be a bug which needs reproduction. To unblock you for now I think we can run a simple maintenance job to delete the temp folder at some schedule. Or an OnJobFinished event which will delete the temp folder when the job is finished.

I skimmed through the plugin “[repo]/plugins/Cinema4DBatch” it does not seem like our plugin is deleting those files. It seems like Cinema does it itself.

I think to troubleshoot further try to submit the same job without C4D batch plugin. You will need to submit the same job by using C4D non batch plugin. To do that just uncheck the box which says “Use Batch Plugin”. Then submit, let it fail and then follow here: Troubleshooting — Deadline 10.1.23.6 documentation

Share your findings.