proxy files

[capitanred]

We’re on a project here where we load millions of points with PRTLoaders which Krakatoa does really well so far.
Still, we did find place to imrproove our workflow quite a bit.

When loading big PRT-Files over the network, there is a big slowdown in the viewport display percentage since it still needs to go trough the whole file and load only the desired particles.
In some cases loading every Nth particle is not really an option, since particels move from left to right, but it’s nescessary to see the whole cloud in the viewport.
So the idea came up to cache a percentage of the original PRT-file out as a proxy. then this proxy PRT-File gets loaded for working in the viewport for fast loading and display. And before rendering a script replaces all the proxy files with the original High particle count ones for the final render.
Now, this works fine, but still needs to be setup by script, ran every time before and after render, to swap bewteen proxy and highRes files. Working on a lot of shots, with many effects, this can add up in time and problems.
An idea would be to integrate this into Krakatoa’s Caching System and Render function. So basically while caching particles, there would be an option to write proxy files on the fly.
And when proxys are loaded, on render Krakatoa could swap the file internally and render the HighRes files.

Another thing we saw is that Max becoems very slow loading the same cache into different PRT-Laoders. Even after all the caches has been loaded, moving the prt’s in teh viewport is slow. Referencing or Instancing the PRT’s speeds it up a lot.
Not so when rendering though. The memory usage is exactly the same aswell.

here is a log info form a test render between instancing, referencing and copying one PRTLoader 7 times:

INSTANCES:
PRG: -----4/24/2012 12:07:47 PM--------------------------
PRG: Rendering frame 1000
STS: Section “Collecting APM Volumes”:
STS: Total 00h 00m 00.000s Called 1 times Avg 00h 00m 00.000s
STS: Section “Evaluating Particle Objects”:
STS: Total 00h 00m 00.234s Called 1 times Avg 00h 00m 00.234s
STS: Section “Retrieving Particles”:
STS: Total 00h 01m 52.430s Called 1 times Avg 00h 01m 52.430s
PRG: Rendering 275610776 particles.
STS: Section “Lighting:Updating Light”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Lighting:Sorting Particles”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Lighting:Calculating”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Rendering:Matte”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Rendering:Sorting”:
STS: Total 00h 00m 17.909s Called 1 times Avg 00h 00m 17.909s
STS: Section “Rendering:Drawing”:
STS: Total 00h 00m 07.909s Called 1 times Avg 00h 00m 07.909s
PRG: Finished rendering frame: 1000
STS: Section “Renderer::Open()”:
STS: Total 00h 00m 00.016s Called 1 times Avg 00h 00m 00.016s
STS: Section “Renderer::Close()”:
STS: Total 00h 00m 03.369s Called 1 times Avg 00h 00m 03.369s
STS: Section “Renderer::Render()”:
STS: Total 00h 02m 22.164s Called 1 times Avg 00h 02m 22.164s
PRG:

REFERENCES:
PRG: -----4/24/2012 12:12:23 PM--------------------------
PRG: Rendering frame 1000
STS: Section “Collecting APM Volumes”:
STS: Total 00h 00m 00.000s Called 1 times Avg 00h 00m 00.000s
STS: Section “Evaluating Particle Objects”:
STS: Total 00h 00m 00.031s Called 1 times Avg 00h 00m 00.031s
STS: Section “Retrieving Particles”:
STS: Total 00h 01m 53.272s Called 1 times Avg 00h 01m 53.272s
PRG: Rendering 275610776 particles.
STS: Section “Lighting:Updating Light”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Lighting:Sorting Particles”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Lighting:Calculating”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Rendering:Matte”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Rendering:Sorting”:
STS: Total 00h 00m 18.206s Called 1 times Avg 00h 00m 18.206s
STS: Section “Rendering:Drawing”:
STS: Total 00h 00m 07.956s Called 1 times Avg 00h 00m 07.956s
PRG: Finished rendering frame: 1000
STS: Section “Renderer::Open()”:
STS: Total 00h 00m 00.015s Called 1 times Avg 00h 00m 00.015s
STS: Section “Renderer::Close()”:
STS: Total 00h 00m 02.044s Called 1 times Avg 00h 00m 02.044s
STS: Section “Renderer::Render()”:
STS: Total 00h 02m 23.474s Called 1 times Avg 00h 02m 23.474s
PRG:

COPIES:
PRG: -----4/24/2012 12:24:59 PM--------------------------
PRG: Rendering frame 1000
STS: Section “Collecting APM Volumes”:
STS: Total 00h 00m 00.000s Called 1 times Avg 00h 00m 00.000s
STS: Section “Evaluating Particle Objects”:
STS: Total 00h 00m 00.031s Called 1 times Avg 00h 00m 00.031s
STS: Section “Retrieving Particles”:
STS: Total 00h 01m 53.163s Called 1 times Avg 00h 01m 53.163s
PRG: Rendering 275610776 particles.
STS: Section “Lighting:Updating Light”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Lighting:Sorting Particles”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Lighting:Calculating”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Rendering:Matte”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Rendering:Sorting”:
STS: Total 00h 00m 18.003s Called 1 times Avg 00h 00m 18.003s
STS: Section “Rendering:Drawing”:
STS: Total 00h 00m 07.784s Called 1 times Avg 00h 00m 07.784s
PRG: Finished rendering frame: 1000
STS: Section “Renderer::Open()”:
STS: Total 00h 00m 00.016s Called 1 times Avg 00h 00m 00.016s
STS: Section “Renderer::Close()”:
STS: Total 00h 00m 03.822s Called 1 times Avg 00h 00m 03.822s
STS: Section “Renderer::Render()”:
STS: Total 00h 02m 22.538s Called 1 times Avg 00h 02m 22.538s

Since I don’t really know much about programming, I don’t know if there is any place to improove anything here. I just found it interesting to mention, since the viewport performance speeds up a lot.

Thanks

[anon58450692]

@capitanred

So the idea came up to cache a percentage of the original PRT-file out as a proxy. then this proxy PRT-File gets loaded for working in the viewport for fast loading and display. And before rendering a script replaces all the proxy files with the original High particle count ones for the final render.
Now, this works fine, but still needs to be setup by script, ran every time before and after render, to swap bewteen proxy and highRes files.

The PRT Loader supports loading multiple sequences of files in one object. It also supports tagging each sequence as visible in the viewport, at render time, or both. Typically people will add a proxy sequence tagged to only load in viewport and the render sequence to only load at render time. You can check out the details here: http://www.thinkboxsoftware.com/krak-prt-loader/#Viewport_and_Render_Controls.

@capitanred

Another thing we saw is that Max becoems very slow loading the same cache into different PRT-Laoders. Even after all the caches has been loaded, moving the prt’s in teh viewport is slow. Referencing or Instancing the PRT’s speeds it up a lot.
Not so when rendering though. The memory usage is exactly the same aswell.

The PRT Loaders don’t do anything fancy with instancing. If the same sequence is present in multiple loaders, the sequence will be fetched from source multiple times. It might end up caching it for the viewport if you use actual node instances but it will not do this at render time.

If you want to see the performance improvements of instancing, I recommend you use the PRT Cloner modifier. Please refer to the documentation here: http://www.thinkboxsoftware.com/krak-prt-cloner-modifier/ and here: http://www.thinkboxsoftware.com/kraka-using-prt-cloner-mod

[capitanred]

The PRT Loader supports loading multiple sequences of files in one object. It also supports tagging each sequence as visible in the viewport, at render time, or both. Typically people will add a proxy sequence tagged to only load in viewport and the render sequence to only load at render time. You can check out the details here: thinkboxsoftware.com/krak-pr … r_Controls.

cool, thats a good way to work, thanks!
still, there is no way to cache out a proxy and fullsize PRT file in one go exept sending to two machines.

I did give the PRT Cloner a try. I cached out one particle which I loaded into my distributer PRTLoader. With the modifier on top the viewport performance became very slow. I couldn’t move the PRT loader anymore without having to wait 10-15 seconds.
Render comparison between 1PRT Loader and 2PRTCloners against 3PRTLaoders instanced:

PRTLaoder and 2 PRT-Cloners:
PRG: -----4/24/2012 5:56:57 PM--------------------------
PRG: Rendering frame 1000
STS: Section “Collecting APM Volumes”:
STS: Total 00h 00m 00.000s Called 1 times Avg 00h 00m 00.000s
STS: Section “Evaluating Particle Objects”:
STS: Total 00h 00m 26.208s Called 1 times Avg 00h 00m 26.208s
STS: Section “Retrieving Particles”:
STS: Total 00h 00m 25.132s Called 1 times Avg 00h 00m 25.132s
PRG: Rendering 103354041 particles.
STS: Section “Lighting:Updating Light”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Lighting:Sorting Particles”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Lighting:Calculating”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Rendering:Matte”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Rendering:Sorting”:
STS: Total 00h 00m 04.914s Called 1 times Avg 00h 00m 04.914s
STS: Section “Rendering:Drawing”:
STS: Total 00h 00m 03.479s Called 1 times Avg 00h 00m 03.479s
PRG: Finished rendering frame: 1000
STS: Section “Renderer::Open()”:
STS: Total 00h 00m 00.046s Called 1 times Avg 00h 00m 00.046s
STS: Section “Renderer::Close()”:
STS: Total 00h 00m 00.749s Called 1 times Avg 00h 00m 00.749s
STS: Section “Renderer::Render()”:
STS: Total 00h 01m 01.293s Called 1 times Avg 00h 01m 01.293s

3PRT Loaders Instanced:
PRG: -----4/24/2012 5:58:25 PM--------------------------
PRG: Rendering frame 1000
STS: Section “Collecting APM Volumes”:
STS: Total 00h 00m 00.000s Called 1 times Avg 00h 00m 00.000s
STS: Section “Evaluating Particle Objects”:
STS: Total 00h 00m 00.015s Called 1 times Avg 00h 00m 00.015s
STS: Section “Retrieving Particles”:
STS: Total 00h 00m 41.559s Called 1 times Avg 00h 00m 41.559s
PRG: Rendering 103354041 particles.
STS: Section “Lighting:Updating Light”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Lighting:Sorting Particles”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Lighting:Calculating”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Rendering:Matte”:
STS: Total 00h 00m 00.000s Called 0 times
STS: Section “Rendering:Sorting”:
STS: Total 00h 00m 04.867s Called 1 times Avg 00h 00m 04.867s
STS: Section “Rendering:Drawing”:
STS: Total 00h 00m 04.212s Called 1 times Avg 00h 00m 04.212s
PRG: Finished rendering frame: 1000
STS: Section “Renderer::Open()”:
STS: Total 00h 00m 00.047s Called 1 times Avg 00h 00m 00.047s
STS: Section “Renderer::Close()”:
STS: Total 00h 00m 01.233s Called 1 times Avg 00h 00m 01.233s
STS: Section “Renderer::Render()”:
STS: Total 00h 00m 52.244s Called 1 times Avg 00h 00m 52.244s

so the retrieving particles part is indeed faster with the cloners. but the updating objects part is slow. the final rendertime is longer than rendering the PRT Loaders itself. The memory used is on both renders the same.
I really like the idea of the PRTCloner modifier, because it also lets one have different settings per PRT loaded. When instancing or referencing, there is always limitations in uniquenes of each object. It’d be also nice if it would be possible to apply the Cloner modifier to a point or some helper object, to grab its transform.

[anon58450692]

@capitanred

still, there is no way to cache out a proxy and fullsize PRT file in one go exept sending to two machines.

I’ll add this to the wishlist.

@capitanred

the final rendertime is longer than rendering the PRT Loaders itself

Not what I expected. I think this might be caused by dropping the viewport cache before rendering and needing to reload it from source afterwards, while the instanced PRT Loaders have special (read: annoying) viewport behavior that allows them to keep the cache. Nevertheless more caching controls are a common request and are on the wishlist.

[capitanred]

sweet, thanks for considering to add it to a later build

concerning the slow viewport performance on the cloner. when moving the distributor PRT, it seems to rebuilt and even reload the cache for every viewportcallback. I was having the same issue with the PRT FumeFX, which I posted over here
http://forums.thinkboxsoftware.com/viewtopic.php?f=22&t=7169