Draft assembly job produces an image with black lines

[smalldots]

The image output produced by draft assembly job looks like this when you look at it at 100% :

Lines sometimes are only horizontal, sometimes both horizontal and vertical…

At first we thought it’s a draft version problem, but then we moved to deadline 10 with draft 1.6.7, and still got the same problem, and it’s not consistent, it’s like it depends on how much tiles you make and the image resolution. In the case I shared above, the image is 6000x3377, but we get it in other resolutions such as 4000x2250 for example.

Here is the full output tiles and the result in exr for the example I showed above : we.tl/tJPtnoRams
This is the config file alone : we.tl/MW3WbEfNKP

I experimented with draft and I was able to assemble the tiles manually with this script :

[code] tileAssembler = Draft.TileAssembler()
#check for .exr files and non .exr files and exclude them from fileList
for f in fileListUnchecked:
if f.find(’.’) != 0 and f.rfind(’.exr’) != -1 :
if re.match(’(tile[\d]+x[\d]+[\d]+x[\d]+|region[\d]+_)’+workingFileName,f) :
readFile = Draft.Image.ReadFromFile( os.path.join(workingDir,f) )
tileAssembler.AddTile( readFile, 0, 0 )
fileList.append(f)

readFile = Draft.Image.ReadFromFile( workingFilePath )


tileAssembler.SetSize( readFile.width, readFile.height )
tileAssembler.SetChannels( readFile.GetChannelNames() )

originalName, ext = os.path.splitext(os.path.join(workingFileName))
newName = "{name}_compiled{ext}".format(name = originalName,ext=ext)

tileAssembler.AssembleToFile( os.path.join(workingDir,newName) )[/code]

The crazy thing is that with this script I can assemble the tiles without having the lines problem, also when we merge the tiles manually on a compositing software we don’t get the lines, so I think the problem is in the way the assembly plugin works in deadline.
Any ideas?
I can also continue using the draft script I made, but one of it’s downsides is it doesn’t conserve the metadata, maybe you can guide me on how to keep the metadata of the input in the assembled file.

[Bobo]

The problem is not in Draft itself, but most probably in the submitter defining the tiles and assembly control file.
Can you please specify which software (Maya, 3dsMax, Modo, etc.) you used for submission?

[smalldots]

I’m using maya, and rendering with arnold.

Here’s how the settings look like :

[Bobo]

Thanks, we are looking into it.

[Bobo]

I looked at the Maya code, and compared it to the Jigsaw code I wrote for 3ds Max. I believe that there is a rounding error in one place. If you want to test my theory yourself, you can try this:

• Navigate to the Repository of your Deadline installation
• Go into the sub-folder \submission\Maya\Main\
• Create a backup copy of the script file “MayaJigsaw.py”
• Open the file “MayaJigsaw.py” for editing and locate the function getRegions

It looks like this

    #Get the Jigsaw UI to return all of the regions and return an array of the ints with the appropriate positions
    def getRegions(self, invert = True):
        self.sockOut.sendall("getrenderregions\n")
        data = recvData(self.sockOut)
        renderHeight = cmds.getAttr('defaultResolution.height')
        renderWidth = cmds.getAttr('defaultResolution.width')
        
        widthMultiplier = renderWidth/(self.usingWidth+0.0)
        heightMultiplier = renderHeight/(self.usingHeight+0.0)
                
        regionString = str(data)
        regionData = regionString.split("=")
        regions = []
        if regionData[0] == "renderregion" and len(regionData) >1:
            regionData = regionData[1]
            regionData = regionData.split(";")
            for region in regionData:
                coordinates = region.split(",")
                if len(coordinates) == 4:
                    regions.append( int( (float(coordinates[0])*widthMultiplier) +0.5) )
                    regions.append(int( ( (float(coordinates[0])+float(coordinates[2])) *widthMultiplier) +0.5))
                    if invert:
                        value = int((float(coordinates[1])+float(coordinates[3]))*heightMultiplier +0.5)
                        regions.append(renderHeight - value)
                        value = int(float(coordinates[1])*heightMultiplier +0.5)
                        regions.append(renderHeight-value)
                    else:
                        regions.append( int( (float(coordinates[1])*heightMultiplier) +0.5) )
                        regions.append(int( ( (float(coordinates[1])+float(coordinates[3])) *heightMultiplier) +0.5))
        return regions

*I believe that the upper left corner’s coordinates are being rounded up/down incorrectly, and should look like this:

                if len(coordinates) == 4:
                    regions.append(int(float(coordinates[0])*widthMultiplier))
                    regions.append(int((float(coordinates[0])+float(coordinates[2]))*widthMultiplier))

• Save the modified script, and submit the same scene again.
• Does this solve the gap issue?

The idea is that the upper left corner of the tile should always be rounded down (int() results in an implicit floor(), dropping the decimal portion). By adding 0.5, we ensure that any floating point part above 0.5 results in the integer part going up to the next highest value (an implicit ceil()). This is a good idea for the bottom right corner to ensure a tile with a width of, say, 100.6 renders as 101 and not as 100. However, if the tile next to it also rounds up its upper left corner, they could end up producing a gap of a pixel. We would rather have an overlap of a pixel than a gap of a pixel.
Thus we want the upper left to always round down, and the lower right to round up to make sure we cover everything without gaps. Removing the +0.5 should ensure that.

[smalldots]

I still get the same black gaps even with those lines edited on the MayaJigsaw.py
I believe it’s more like an error of the assembler than tiling, cause as I said in my first post, when I use my assembly code the images look clean, and same when we assemble the images on any compositing package. It’s only when we use the built-in dependent assembly job we get those gaps.
Also as you might have noticed in the screenshot, I didn’t really use jigsaw UI, I used the built-in tiling in the deadline plugin UI, and put 5 tiles in X and 5 tiles in Y. But we had the same problem with jigsaw UI regions too.

[Bobo]

Thanks for the feedback, I am sure this will point our developers in the right direction!

[smalldots]

By the way regarding the gaps, this might help the developers find the solution. We noticed we only get the gaps when the quotient of resolution/regions inputs a non integer. So yesterday I tested sending a job with 4000x2260 in resolution divided into 5x5 regions, and it didn’t give me any gaps. So we only get the gaps on resolutions like 4000x2251 (I mentioned that I had the same problem on 4000x2250 but I was wrong, I noticed it was 2251 actually).

Meanwhile could you guide me on how I can at least transfer the metadata from my tiles to the tileAssembler output in my script above? since my script does assemble the tiles without the gaps problems. This way we can at least use my script to assemble the tiles we render while waiting for the developers to come up with a solution for the gaps problem in the deadline built-in draft assembly plugin.

[Bobo]

I just rendered 4000x2251 with 5x5 tiles and it assembled without gaps. Using Deadline 10.0.8.1 which I installed yesterday (10.0.8.3 is current as of today).
Now I have to find out whether anything was fixed in that build, or whether I did something wrong to get it right…

EDIT: Now rendering 6000x3377 to match your original report…
EDIT2: Rendered and assembled fine.
However, you are rendering in Arnold, and I tested with Software renderer first.
So now I will test with Arnold, and if it causes issues, then the problem is with Arnold specifically (your DTA control file seems to match my control file, so SMTD’s output is right, and that would mean there is an issue with the tiles themselves).
Will let you know what I will find…
EDIT3: There is a problem with the control file generated when the tiles are not cropped (as in Arnold). I am looking at the code defining the regions in the job vs. the regions in the DTA to see where the misalignment comes from…

[Bobo]

Looks like I finally got a correctly assembled image from Arnold after tweaking the control file generation code. However, I will have to test a lot more to ensure I did not break any of the other renderers’ assembling.

I will pass my fixes to the developers to review and confirm that I am doing the right thing…

[smalldots]

Thank you for all the tests you made Bobo,

One thing I was wondering about when you said “There is a problem with the control file generated when the tiles are not cropped (as in Arnold)”
Does this mean if “auto cropped” is enabled, we won’t have this issue ?
Anyway I’m testing that now and let you know if it’s that’s the case.

Edit: The problem occurs with both “auto cropped” checked and unchecked on arnold render globals. I’m using 6000x3377 for the resolution and 5x5 region rendering tiles.

[Bobo]

There is a coordinate system inversion (rendered image has the origin in the upper left, Draft has it at bottom left). We were calculating the inverted Y coordinate incorrectly when the vertical image resolution was not equally divisible by the tile count.

In your case, you have 3377/5=675.4, but we drop the decimal point and render 675 for the top 4 tile rows, and 677 for the 5th tile row to cover all 3377 pixels (0.45=2 pixels to compensate for!). But when we assemble, we calculate where the bottom left corner of the tile should go, and we once again use the rounded down 675 value, but end up 2 pixel rows too low. The correct way would be to take the tile row index times the tile height, and subtract from the total height of the image, e.g. 3377-(6751)=2702. Instead, we were inverting the tile row index, e.g. 675*(5-1)=2700. The same happens with all the other rows, and we get the 2 pixel rows offset for all of them. For the last row, now we must special-case it and place the bottom left corner of the tile at the 0 row, and calculate the same 677 height (which is 3377-(4*675)).

I have passed the issue to the developers, and had a code fix that worked for Arnold, but I have not tested if I broke anything else with any other renderer, or mode. So I don’t want to just give you a modified version that might affect something else…

[Bobo]

I have submitted a fix for this bug internally, but figured you might want to give it a try yourself and see if it works correctly now.
Attached is a fix which changes how Arnold tiles are assembled, but due to the changes I also had to modify V-Ray, MayaSoftware and all other renderer’s handling for both single frames and animations, so it took a while to implement and test.

To test,

• Navigate to your Repository share.
• Locate the \Submission\Maya\Main\ folder.
• Make a backup copy of the file “SubmitMayaToDeadline.mel” (e.g. rename to “SubmitMayaToDeadline.old”).
• Copy the attached version of the file “SubmitMayaToDeadline.mel” into that folder.
• Start Maya and try to submit a job with 3377 height and 5 tiles in Y.

SubmitMayaToDeadline.zip (65.3 KB)

[smalldots]

This problem seems to have been fixed I think since Deadline 10.0.10.3, at least we didn’t see the problem since we updated back then, however recently when we just updated to Deadline 10.0.18.1, we started getting the same issue again…
This is an image rendered in 6000x2394 and it has the black lines

This one is rendered at 6000x3380 and no black lines

Both of them were rendered in 5x5 tiles
So it’s safe to say that the problem is the same as before.
Here is the testing scene file : still_test_v16.zip (13.8 KB)

Should I roll back to deadline 10.0.10.3 again for more testing ? Any idea why this bug fix would disappear in the Deadline 10.0.18.1 ?

EDIT : reverting back to the 10.0.10.3 version of SubmitMayaToDeadline.mel seems to have fixed the issue, the above image doesn’t have lines anymore.

[eamsler]

Thanks for testing for this regression! I’ll move this over to the dev system.

The guess right now is that this is a rounding issue between the coordinate numbers.

[smalldots]

Just wanted to mention that this problem still persists in the 10.0.22.3 version of deadline too. Each time I update deadline I have to replace the SubmitMayaToDeadline.mel with the 10.0.10.3 version to not have this issue, which makes me lose other features of the last version I imagine.

[eamsler]

Thanks for chasing on this one! We’re actually doing some heavy re-work with regard to tile rendering in Maya. I’m not sure when it’ll be ready, but your feedback here was the catalyst for the work being one.

This all to say: We haven’t forgotten about you, and I don’t have a timeline for you yet.