Obscurity of CoronaDR submission via custom Python script

[Justin_B]

@Tiger

So, the question. How does SMTD do this and can I steal the lines of code I need from somewhere that I can add to my custom script for similar scene submission with all available parameters?

Yep, you totally can. You’re going to have to reverse-engineer what does what with the code in your DeadlineRepository10\submission\3dsMax and this page about SMTD’s scripting as starting points.

---

@Tiger

A lot of information has been written about Job Info File Options, but almost nothing about Plugin Info File:

The Plugin info file doesn’t have a set of pre-defined values like the Job info file does. It’s unique to the plugin (hence the name! ), and tragically aren’t documented. My usual tactic is to find the key I’m interested in and search the application code for how it’s used. It’ll need reverse-engineering to figure out what the keys do.

---

@Tiger

Therefore, I really ask you to explain what I am wrong about
Maybe I’m complicating everything and I don’t need to specify absolutely all the settings in the pluginfo before submitting the scene, (as SMTD does, specifying absolutely all the available parameters in the scene) but only the most necessary, which will be enough and universal for all scenes?

You’re likely going to need to include everything SMTD does if you want to make an identical job to it. I recommend re-using as much of the existing SMTD code as possible to avoid reinventing the wheel.

---

@Tiger

Or am I right after all and is there some way to achieve what I want? I also thought about the possibility that what I want to do is impossible, since the whole process of scene submission via SMTD takes place somewhere deep in the 3ds Max and Deadline code itself, which cannot be repeated, since this is closed software data and only Thinkbox has access to these “lines of code” (correct me if I’m wrong).

The only bits that are hidden is the API documented here. That’s all in the C# code and the source isn’t available. What the function does can be ambiguous, feel free to ask if there’s a call that doesn’t make sense. The only blocker is that there’s no existing documentation for what you’re trying to pull off, so you’re going to be the expert.

[Tiger]

@Justin_B I agree with you, I don’t want to reinvent the wheel, so I want to use the simplest possible way of submitting without having to write a custom submitter from scratch.

I’m just trying to figure out where I can get the lines of code I need (or use the whole code) so that the whole submission process is automated.

For example, you recommend this:

@Justin_B

DeadlineRepository10\submission\3dsMax and this page about SMTD’s scripting as starting points.

But there are a lot of maxscript scripts in the folder that are somehow related to each other. I’ve studied some of them, but basically all they do is communicate with the 3ds Max interface. While I need to understand (do I need to?) how their backend works, what it calls and how to get it to initiate the whole process without having to click through the checkboxes and buttons everywhere on their own.

By the way, I studied this file too:

It also describes almost all the functionality of the 3ds Max plugin, but again about communication with the 3ds Max interface only.

Therefore, I still don’t understand how I can implement what I want.

It seems that there is a full-fledged SMTD script that satisfies me in its functionality. However, it requires the presence of a human, since this script is integrated in the 3ds Max interface. Accordingly, to submit a job through it you need to manually click on the SUBMIT button in its interface.

Therefore, the most obvious thing is to try to copy the entire SMTD script code and make it function automatically for my needs. But the problem is that I absolutely don’t understand how and what SMTD maxscript calls to during job submission.

I will explain in the simplest language with a clear example. In my understanding, when I click on the SUBMIT button in the SMTD interface, then some kind of process takes place to record all the data (settings, parameters, variables, etc.) from mine .max scene, at the same time, this data is sent to the Deadline’s API, and then the Deadline’s API recognizes this data and initiates rendering. Like this:

So what happens in the end in the 2nd process?

You have repeatedly recommended me to check the SMTD maxscript files themselves, but all maxscripts are essentially working with pressing buttons or checkboxes in the 3ds Max interface. There is no information about the backend of these scripts and what they call for.

For example, here’s what you can find there:

SMTDFunctions.storeSettingsWithScene - is one of the functions that should explain the process of submission, right?
Okay, I fond the functionality of this function in the SubmitMaxToDeadline_Functions.ms script:

But again, a dead end, since it is unclear where this function leads further and what in fact it does or what it calls to.

In my understanding, there must be something that describes the process that occurs at the moment of submitting via SMTD. Some log files, debug functions, or something where it is described. Otherwise, I cannot track this chain of actions by copying and adapting them to my needs. After all, this maxscript (SMTD) should write all the settings and parameters somewhere in a document or something else, and then feed them automatically to the Deadline API? It’s basically doing that. That’s why I want to understand where and how it does it to repeat the whole process to automate it.

That stuff makes my brain hurt

[Justin_B]

For sure!

@Tiger

It also describes almost all the functionality of the 3ds Max plugin, but again about communication with the 3ds Max interface only.

That’s the application plugin - it’s the starting point the Deadline Worker uses to process 3dsMax jobs. You’re likely finding where the plugin info keys are getting referenced in there.

@Tiger

I will explain in the simplest language with a clear example. In my understanding, when I click on the SUBMIT button in the SMTD interface, then some kind of process takes place to record all the data (settings, parameters, variables, etc.) from mine .max scene, at the same time, this data is sent to the Deadline’s API, and then the Deadline’s API recognizes this data and initiates rendering. Like this:

In your #2 step that’s where we use MaxScript to interrogate the loaded Max file for render settings, output paths, and everything else.

In your highlighted bit of SubmitMaxToDeadline_Functions.ms we’re hooking into setAppData. By the look of it we’re saving all the current settings the scene has into trackViewNodes so we can restore that later.

So instead of calling that code in response to clicking a button in the UI your script would call the function directly.

@Tiger

After all, this maxscript (SMTD) should write all the settings and parameters somewhere in a document or something else, and then feed them automatically to the Deadline API?

The job info and plugin info documents associated with the job are what’s getting fed into that 3dsMax.py you’ve looked into. My usual tactic is to work backwards from a job’s plugin info that’s set the way I want and look through the SMTD code for that keyword to see how it’s pulled.

I’ve never done that with Max however, so my Maxscript knowledge is limited to what I can find online.

Let us know if you have any other questions!

[Tiger]

@Justin_B

I’ve never done that with Max however, so my Maxscript knowledge is limited to what I can find online.

@Justin_B Yes, I understand that and I really appreciate that you’re trying to help me

However, all of the above doesn’t quite answer my questions. I mean, yes, I realized that there is a function like setAppData that accesses 3ds Max memory and assigns a unique ID to my scene. But this still leaves the question open - where does it save this data (the trackViewNodes variable is also unknown to me) and how do I repeat similar actions so that the whole process of clicking the SUBMIT button in SMTD is initiated without the need to press this button in the 3ds Max interface?

Again, let’s say I have a 3D scene that I’ve never opened and I don’t know what settings are listed there. I want to feed/submit it to Deadline so that the scene appears in Deadline Monitor and starts the rendering process. If I were to open this scene manually, I would just launch SMTD in the interface and click on the SUBMIT button. But how can I do this without opening the scene manually?

From all that I understood, when submitting a scene, it is given a unique ID and a folder is created in the DeadlineRepository, where the scene itself and all other assets to it (if any) will be stored:

But I still don’t understand the logic of assigning a unique ID. How and by what principle is it assigned? It’s not just a set of characters, right? After all, if I just repeat the process manually, creating a folder in the jobs folder with a random name in the form of numbers and letters, then Deadline is unlikely to trigger at this and realize that a new task has appeared, right?

The fact is that I am also not an expert in maxscript and many things are unclear and not obvious to me. Of course, it would be better to contact a specialist on my question but then where can I find such a specialist? What if I contact Deadline tech support by creating a ticket, or is this not the best option? I mean I can’t even imagine what kind of specialist I need who can help me solve my question. A Maxscript specialist? A Deadline specialist? A specialist in C#/C++ or Python? Or all of them together?

If I try to solve the issue on my own, then for me it looks like a huge tangle that I’m trying to untangle. It takes and still will take a very long time apparently

[Justin_B]

Did you take a look at the Autodesk docs on setAppData? I’d have thought that would be a decent starting off point - maybe ask about it in the Autodesk forums?

You’ve looked at the SMTD code, so I’ll assume you’ve looked at SubmitMaxToDeadline_Functions.ms - did you have any luck tracing the function calls in SubmitJobFromUI now that you know to check MaxScript API docs? Depending on how complicated your scene files are, it may be simpler to avoid SMTD altogether and steal the bits of Maxscript it uses for your own purposes. With that in mind, how are you submitting jobs currently? Just using SMTD for manually creating jobs? I should have asked why you’re doing this earlier.

If you stick with this project, you’ll end up being the specialist people ask for.

As for your question about the folder in your DeadlineRepository10\jobs directory - that ID is the job ID, and it’s the GUID assigned to the job entry in the MonogoDB. So it’s just whatever MongoDB kicked out when we asked it to create a new object.

[Tiger]

@Justin_B

Did you take a look at the Autodesk docs on setAppData? I’d have thought that would be a decent starting off point - maybe ask about it in the Autodesk forums?

Yeah, I took a look at these docs. But I don’t quite understand the functionality described there. There are literally a few sentences and a couple of examples. It looks like it was taken out of context. Most likely it is, since the setAppData function is just the tip of the iceberg and I need to dig through the information from the very beginning to figure out what’s what.

@Justin_B

Depending on how complicated your scene files are, it may be simpler to avoid SMTD altogether and steal the bits of Maxscript it uses for your own purposes. With that in mind, how are you submitting jobs currently? Just using SMTD for manually creating jobs?

Here is the easiest way to submit a job (3ds Max’s 3D scene) to Deadline, without using any custom scripts and other complications:

1. I’m manually opening the 3D scene I need using 3ds Max 2020.
2. I’m opening the Integrated Submitter (SMTD).
3. I specify the settings/parameters I need in 3ds Max itself (for example, image resolution) and in SMTD, or I do not specify anything at all (depends on my desire).
4. I click on the SUBMIT button, wait for the Sanity Check to end, and if everything is fine, then my job appears in Deadline Monitor and rendering begins on all my render nodes.

But as I have said many times before, this option does not suit me because of its time-consuming and inflexibility. Therefore, I want to use the same script (SMTD), but make it perform the entire process described above automatically. So, how do I do this?

For example, in my pipeline it should look something like this:

1. Me or my colleague sends the archive with the scene (with all assets and other parts belonging to the 3D scene) to some shared network drive/cloud in some specific folder. There, this scene is automatically unzipped by my other script.
2. At this point, for example, some kind of POST request is sent, to which my automation script (which I am trying to do) is triggered and submits the job independently.
   (By the way, to clarify, I mentioned the POST request just as an example. I mean I know that I can connect directly to the Deadline API and send the necessary requests directly)

I think now you understand my logic and what am I trying to achieve? What do you think, have I come up with something impossible or completely feasible?

[Justin_B]

I understand what you’re trying to do and it’s totally possible. You have a couple unknowns:

1.How to interrogate a 3dsMax scene file for its render settings
2.How to create a Deadline job compatible with the 3dsMax application plugin

The issue is that there isn’t a cut and dried example of how to do that, so I’m not sure what I can tell you that’s helpful that I haven’t already told you.

I’d start simple and expand the scope of the project, as this isn’t simple and little wins make big projects easier.

So if I was laying this out as an educational project for someone, here’s how I’d do it:

1. Submit a job from deadlinecommand using a non-functional example from the docs. A success here is a new job in the Monitor.

2. Submit a job that’s a duplicate of a manually submitted job, by providing it with job.info and plugin.info files from exporting job’s submission parameters. A success here is a new job in the Monitor that successfully renders just like the job that was submitted manually.

3. Take a look in the plugin.info file for the scene file and the output path. Take the key from the key=value pair and search for where that’s used within the 3dsmax.py file in DeadlineRepository10\plugins\3dsmax. Was the key you’d found correct? If you change the output path, and submit a job does 3dsMax respect that change? Success here is reverse engineering how the output path or scene file name is stored in the job’s submission data.

4. Now that you know the key for the scene file, go look in the DeadlineRepository10\submission\3dsMax directory for that key. I use ripgrep to look for text within files, use your preferred method. That’ll likely be where the job.info and plugin.info files are being written. Find the variable that’s being used to set the scene file and trace back to where that variable is set. That should have the maxscript code that returns the scene file name. Success here is using the maxscript console with your scene loaded and the code you’ve found to get the scene file path printed.

5. Now to pull data about a scene without manually opening Max. You can call MaxScript from a terminal, so it should be possible to load a scene and print information out about it. Success here would be using the MaxScript you learned in step 4 to headlessly get information about a scene file.

With all that done you’ll have the tools to interrogate 3dsMax scene files for their render settings and know how to figure out what the application plugin needs to render.

@Tiger

I think now you understand my logic and what am I trying to achieve? What do you think, have I come up with something impossible or completely feasible?

Totally feasible, just difficult. You can do it!