thinkboxsoftware.com/krak-fi … ainy-look/
this was sent to me after the Siggraph presentation I did, and early on with Daniel I was trying to figure out how to apply this strategy to the cinema 4d SR bridge. Could any of you Thinkbox genies share some info on translating this articles principles to our workflow?
LOL, I just spent 1:41:47 on Skype with Daniel talking about these things, and another hour typing. In fact, we discussed this exact topic.
I believe there might be a certain misconception that “Partitions” as we call them in Krakatoa MX and MY are a relic of the past, a necessary evil due to the inability of the host application (earlier Max 32 bit, then Maya in general) to handle a billion particles. The truth is that while the original reason for their existence was exactly that, once we started using them in production, we realized that there is more to this than just a workaround or a hack. So I will try to talk about this too.
But back on topic: That article was created because someone on a forum once posted about something rendered in Houdini Mantra “This looks grainy, like a Krakatoa rendering”. So obviously we had to do something about all the people posting grainy videos on YouTube 
The easiest way to think of Krakatoa in terms of real-world analogy is probably air brushing. There, you spray your paint on a canvas (a car, a body, any object) producing a mist of tiny droplets. Layer after layer they accumulate until you see a smooth result instead of individual particles.
Of course, sometime you might want the grainy look for artistic reasons. Be it pointillism
webexhibits.org/colorart/jatte.html
or because the director said so
thinkboxsoftware.com/storage … 2384559465
or because you are trying to recreate the Sandman from Spider Man
thinkboxsoftware.com/krakato … s/#Sandman
or are after a spongy look just because that’s what your character calls for:
features.cgsociety.org/newgaller … _large.jpg
But in many cases like when creating Plasma like in the opening of The Avengers
vimeo.com/51267399
or just creating simple wispy cigarette smoke, you want your particles to be invisible and rather merge into a continuous volume.
There is also a problem illustrated here: thinkboxsoftware.com/kmy-vol … rendering/
If you have a very high per-particle Density (the amount of density a particle contributes to the space it resides in), it starts blocking nearly all the light that passes through it, making all particles behind it dark. So you end up with a huge contrast between the lit particle and the immediate one behind it. We provide a separate control for the Lighting Pass Density so you can tweak how particles block light independently from how particles block the camera view, but in general it is desirable that the actual density of a particle is very low, e.g. 0.0001 or even less, and you add thousands of particles together in close proximity to contribute to a stronger density in the final pixel (which is represented by the Alpha channel of the image).
Even when rendering fully additively (only accumulating a color without calculating shadows), having many particles with little influence is very helpful to produce a fine result. An extreme example from our Siggraph 2011 tests was this rendering of 7 billion particles where each particle had an influence of only 0.000000001 to the final pixel! thinkboxsoftware.com/news/20 … /7bpf.html
So what we want is having a huge amount of particles to cover each pixel hundreds or thousands of times with a tiny value. We have to talk about two things here - why we want this, and how to achieve it. I think the above discussion and the link you posted explain a lot about why we want it. The How To is trickier because in the original version of the C4D bridge we are all testing here, the concept of Partitioning has not been really covered yet.
Partitioning was our way around the original limitation of 3ds Max 32 bit back in 2005/2006. We would simulate the same system several times, each time with slightly different random seeds in relevant operators like birth, position or velocity. Even just randomizing where a particle is born in the beginning of the same simulation produces a Partition where every particle ends up slightly off from the previous one, and when loaded together, the resulting cloud has double the density with two versions, 10 times the density with 10 partitions and so on.
In 3ds Max and Maya, we implemented the saving via scripting (MAXScript resp. MEL), so we are confident we could do something like this for C4D using Python. Right now, you would have to either produce the final amount of particles in one go (e.g. using X-Particles which can handle very high counts), or tweak the settings and save multiple versions of the same simulation manually using the Saver. In addition, we also have the Particle Repopulation option which takes a low-resolution simulation and creates new particles on a grid based on the influence of the base particles. The results are very useful for things like RealFlow liquids, but could result in loss of fine detail in smokey wispy situations, so it is not a universal silver bullet. But it allows you to produce millions of particles on demand at render time, so it is an important part of the arsenal.
And then on the loader side the Krakatoa File Source (aka PRT Loader) in KMX and KMY supports the more or less automatic loading of dozens of even hundreds of partition sequences with one or a few clicks. Currently in C4D Krakatoa File Source supports only a single sequence. Loading 10 sequences would require 10 loaders. It is not impossible, but it is a pain in proximity of the chair. 
So after spending less than 24 hours with C4D and Krakatoa like everyone else here, this was one of my main Wish List item.
I promised in the beginning to explain why Partitions are so useful. Here are some of the reasons:
When you start working on a project, you want to have fast iterations. So you are probably using 10 million particles per frame rendering at 1K to show the boss results quickly. The beauty of Krakatoa is that its speed is typically dependent on the particle count and not so much on the image resolution. And if you reduce the resolution, you can get away with less particles (see the original post above, it shows that in the article). So you can get a good idea of the final result using a fraction of the final particle amount.
Once the simulation is approved, you can keep your existing first partition and generate several more to create enough particles to cover 2K or 4K image with a similar density and high quality. And because you are producing iterations of, say, 10MP (million particles) each, you can keep on adding more and more until the deadline for rendering comes around, or you are happy with the quality (see air brushing analogy). Compare this to deciding to simulate 200 million particles in one go, waiting for hours or even days and then realizing you needed only 100 million to get a good enough result (wasted time!)
Then there is the parallelization of the processing - in both KMX and KMY (and hopefully in KC4D), the processing can be sent to multiple network machines. So if you need 100 million particles from 10 simulations, you can simulate 10 million on 10 network nodes using Deadline or whatever, and in the end get 100 million in 1/10th of the time. For large companies, this is quite important.
Then there is the loading part. In the KMX and KMY particle loaders, each sequence can be controlled individually to be visible in the viewport, in the renderer, in both or in neither. So you can load 10 partitions with 10MP each for a total of 100 million at render time, but load only the first partition of 10 million with every Nth set to 1% to show only 100,000 particles in the viewport, while still having a good overall representation of the whole cloud. Since PRT files contain a single Zip stream, all particles much be uncompressed to read even a fraction of them. So if you try to read 0.1% of a PRT with 100MP, you have to wait for all of them to be unzipped to get the ones you want. If you are reading 1% of 10MP, it takes 10 times less time to unzip the whole stream! (For speed, we also support loading the First N, but often that does not represent the whole simulation correctly because the order of particles is often age-based)
And then there is the ability to load faster (esp. on the local machine) from a fast SSD or Fusion-IO card, or even from an HDD drive. Having multiple partitions allows us to create a separate thread for each partition and process the zip streams in parallel. So reading 10 partitions on 10 CPUs can be up to 10x faster than loading 1 partition of 100 MP on one CPU (assuming the drive can keep up). You can watch the following video showing Krakatoa MX loading 180 million particles in 10 partitions x 18 million from a Fusion-IO card in 5.6 seconds vs. 77 seconds for sequential single-threaded reading.
Go to minute 4:11 for the relevant part:
thinkboxsoftware.com/news/20 … rview.html
There are some more practical benefits to using partitions. Imagine you have 200 frames animation in 1 sequence of 100 million particles and frame 182 becomes corrupt for some technical reason, or somebody just deleted it accidentally. You would now need to resimulate nearly everything to restore that frame. If you had 10 partitions with 10 million each, you can either resim 10 million to restore the bad/missing frame, or just disable the affected one partition and render the remaining 9 after increasing the Density by 10% - chances are the client won’t notice the difference! 
Long story short: I realized today that the Krakatoa C4D build we are testing does not provide partitioning tools yet, so we should look into creating some…
Here is a quick test of the Repopulation feature using C4D’s native particles.
I created an Emitter pointing up, a Wind pointing up, a Friction and a Turbulence.
Set the Birth Rate to 10,000 and produced 50,000 particles around frame 100.
Created a Krakatoa Emitter Source and picked the Emitter.
Here is what it looks like at 640x480 using Force Additive Mode and a blue color:
Then I enabled the Repopulation option and set it to Radius 0.1, 2 subdivs and 10 particles per subdiv.
This produced 39,387,887 particles in 27 seconds and finished rendering in about 41 seconds total:
But then I reduced the number of particles per subdiv to 5, producing half the particle count 19,693,969 in 16.177 seconds.
Total render time was around 25 seconds, about 9 seconds for the actual drawing of nearly 20 million points.
You can probably see no difference between 20 million and 40 million at this resolution…
If I would render at a higher resolution though, the additional particles would help.
Increasing the radius from 0.1 to 0.15 reduces the detail, but it also produces less particles with 2 subdivs and 10 particles per subdiv - instead of 39MP, we get only 21MP (21,633,024 in 16,446 seconds)
Render time is similar to our 3rd render since the particle count is also similar - we get it in 23 seconds.
Below are the original logs from the renders:
[code][Krakatoa | PROGRESS] Retrieving Particles
[Krakatoa | STAT] Section “Retrieving Particles”:
[Krakatoa | STAT] Total 00h 00m 00.016sCalled 1 times Avg 00h 00m 00.016s
[Krakatoa | STAT] Total number of particles to be rendered: 50,000
[Krakatoa | STAT] Total Memory Allocated to particles: 1 MB
[Krakatoa | PROGRESS] Rendering
[Krakatoa | PROGRESS] Rendering: Drawing 50,000 particles
[Krakatoa | STAT] Rendering 528 triangles from 1 meshes
[Krakatoa | STAT] Section “Rendering:Matte”:
[Krakatoa | STAT] Total 00h 00m 00.000sCalled 1 times Avg 00h 00m 00.000s
[Krakatoa | STAT] Section “Rendering:Sorting”:
[Krakatoa | STAT] Total 00h 00m 00.000sCalled 0 times
[Krakatoa | STAT] Section “Rendering:Drawing”:
[Krakatoa | STAT] Total 00h 00m 00.046sCalled 1 times Avg 00h 00m 00.046s
[Krakatoa | PROGRESS] Computing particle repopulation grid.
[Krakatoa | PROGRESS] 0% complete
[Krakatoa | PROGRESS] Retrieving Particles
[Krakatoa | STAT] Section “Retrieving Particles”:
[Krakatoa | STAT] Total 00h 00m 27.144sCalled 1 times Avg 00h 00m 27.144s
[Krakatoa | STAT] Total number of particles to be rendered: 39,387,887
[Krakatoa | STAT] Total Memory Allocated to particles: 1 GB
[Krakatoa | PROGRESS] Rendering
[Krakatoa | PROGRESS] Rendering: Drawing 39,387,887 particles
[Krakatoa | STAT] Rendering 528 triangles from 1 meshes
[Krakatoa | PROGRESS] 27% complete
[Krakatoa | PROGRESS] 34% complete
[Krakatoa | PROGRESS] 38% complete
[Krakatoa | PROGRESS] 42% complete
[Krakatoa | PROGRESS] 46% complete
[Krakatoa | PROGRESS] 50% complete
[Krakatoa | PROGRESS] 57% complete
[Krakatoa | PROGRESS] 63% complete
[Krakatoa | PROGRESS] 66% complete
[Krakatoa | PROGRESS] 71% complete
[Krakatoa | STAT] Section “Rendering:Matte”:
[Krakatoa | STAT] Total 00h 00m 00.000sCalled 1 times Avg 00h 00m 00.000s
[Krakatoa | STAT] Section “Rendering:Sorting”:
[Krakatoa | STAT] Total 00h 00m 00.000sCalled 0 times
[Krakatoa | STAT] Section “Rendering:Drawing”:
[Krakatoa | STAT] Total 00h 00m 10.593sCalled 1 times Avg 00h 00m 10.593s
[Krakatoa | PROGRESS] Computing particle repopulation grid.
[Krakatoa | PROGRESS] 0% complete
[Krakatoa | PROGRESS] 46% complete
[Krakatoa | PROGRESS] Retrieving Particles
[Krakatoa | STAT] Section “Retrieving Particles”:
[Krakatoa | STAT] Total 00h 00m 16.177sCalled 1 times Avg 00h 00m 16.177s
[Krakatoa | STAT] Total number of particles to be rendered: 19,693,969
[Krakatoa | STAT] Total Memory Allocated to particles: 525 MB
[Krakatoa | PROGRESS] Rendering
[Krakatoa | PROGRESS] Rendering: Drawing 19,693,969 particles
[Krakatoa | STAT] Rendering 528 triangles from 1 meshes
[Krakatoa | PROGRESS] 29% complete
[Krakatoa | PROGRESS] 36% complete
[Krakatoa | PROGRESS] 49% complete
[Krakatoa | PROGRESS] 58% complete
[Krakatoa | PROGRESS] 67% complete
[Krakatoa | PROGRESS] 75% complete
[Krakatoa | STAT] Section “Rendering:Matte”:
[Krakatoa | STAT] Total 00h 00m 00.000sCalled 1 times Avg 00h 00m 00.000s
[Krakatoa | STAT] Section “Rendering:Sorting”:
[Krakatoa | STAT] Total 00h 00m 00.000sCalled 0 times
[Krakatoa | STAT] Section “Rendering:Drawing”:
[Krakatoa | STAT] Total 00h 00m 05.445sCalled 1 times Avg 00h 00m 05.445s
[Krakatoa | PROGRESS] Computing particle repopulation grid.
[Krakatoa | PROGRESS] 0% complete
[Krakatoa | PROGRESS] Retrieving Particles
[Krakatoa | STAT] Section “Retrieving Particles”:
[Krakatoa | STAT] Total 00h 00m 14.664sCalled 1 times Avg 00h 00m 14.664s
[Krakatoa | STAT] Total number of particles to be rendered: 21,633,024
[Krakatoa | STAT] Total Memory Allocated to particles: 577 MB
[Krakatoa | PROGRESS] Rendering
[Krakatoa | PROGRESS] Rendering: Drawing 21,633,024 particles
[Krakatoa | STAT] Rendering 528 triangles from 1 meshes
[Krakatoa | PROGRESS] 27% complete
[Krakatoa | PROGRESS] 35% complete
[Krakatoa | PROGRESS] 43% complete
[Krakatoa | PROGRESS] 53% complete
[Krakatoa | PROGRESS] 62% complete
[Krakatoa | PROGRESS] 70% complete
[Krakatoa | STAT] Section “Rendering:Matte”:
[Krakatoa | STAT] Total 00h 00m 00.000sCalled 1 times Avg 00h 00m 00.000s
[Krakatoa | STAT] Section “Rendering:Sorting”:
[Krakatoa | STAT] Total 00h 00m 00.000sCalled 0 times
[Krakatoa | STAT] Section “Rendering:Drawing”:
[Krakatoa | STAT] Total 00h 00m 05.772sCalled 1 times Avg 00h 00m 05.772s
[/code]
I also found a few bugs in the process.
*When rendering native C4D particles, there seems to be a glowing ball of particles around the origin. I placed a matte sphere to occlude it, but I saw it yesterday too in a different Emitter case:
I wonder if C4D is producing particles that have not been placed at the emitter’s shape yet and have not been affected by the forces, thus being kind of “undefined” at age 0.
*When motion blur is enabled, the simulation is lost and I get just a glowing box of particles around the origin. So something in the way the time samples are evaluated might be interfering with the particle sim. I am no C4D guru, so I don’t know if this is a known effect or a problem with KC4D.
Ahh, thanks for all that info. The avengers plasma is precisely what Im trying to get closer too. With x-particles 2.5 and my machines the particle count is a non-issue I can easily work with 80-90million particles in one scene with the amount of ram I have available to me.
Problem, currently cinema 4d (without x-particles 2.5) is just unable to produce the amount of particles required to REALLY take advantage of the extreme layering.
Desired solution, being able to refine noisy particle renders into more whispy, additive renders.
With x-particles 2.5, tpartio, etc we do have the ability to handle large amounts of particle data. Even in Turbulence FD, we can create really impressive fluid sims to drive KSR’s render.
Ill report back with any renders that get closer to the ref link.
edit
oh wow particle repop works MUCH differently than it used to, take a gander at this.
[PROGRESS] Computing particle repopulation grid.
[PROGRESS] 0% complete
[PROGRESS] 53% complete
[DEBUG] Internally reserving memory for 1000000 particles.
[DEBUG] Using particle layout:
[DEBUG] channel_map {
[DEBUG] float32[3] Position;
[DEBUG] float16[3] Color;
[DEBUG] float16[3] Lighting;
[DEBUG] float16 Density;
[DEBUG] };
[DEBUG]
[PROGRESS] Retrieving Particles
[STAT] Section "Retrieving Particles":
[STAT] Total 00h 01m 34.594s Called 1 times Avg 00h 01m 34.594s
[STAT] Total number of particles to be rendered: 236,063,683
[STAT] Total Memory Allocated to particles: 6 GB
[PROGRESS] Rendering
[PROGRESS] Rendering: Drawing 236,063,683 particles
[PROGRESS] 52% complete
[PROGRESS] 55% complete
[PROGRESS] 59% complete
[PROGRESS] 64% complete
[PROGRESS] 67% complete
[PROGRESS] 71% complete
[PROGRESS] 73% complete
[STAT] Section "Rendering:Matte":
[STAT] Total 00h 00m 00.000s Called 0 times
[STAT] Section "Rendering:Sorting":
[STAT] Total 00h 00m 00.000s Called 0 times
[STAT] Section "Rendering:Drawing":
[STAT] Total 00h 00m 14.735s Called 1 times Avg 00h 00m 14.735sin the early version I had particle repop just made everything sorta swollen and not very interesting.
edit
starting to grasp the basics of the falloff/power relationship.
[PROGRESS] Computing particle repopulation grid.
[PROGRESS] 0% complete
[PROGRESS] 48% complete
[DEBUG] Internally reserving memory for 1000000 particles.
[DEBUG] Using particle layout:
[DEBUG] channel_map {
[DEBUG] float32[3] Position;
[DEBUG] float16[3] Color;
[DEBUG] float16[3] Lighting;
[DEBUG] float16 Density;
[DEBUG] };
[DEBUG]
[PROGRESS] Retrieving Particles
[STAT] Section "Retrieving Particles":
[STAT] Total 00h 01m 16.047s Called 1 times Avg 00h 01m 16.047s
[STAT] Total number of particles to be rendered: 189,246,817
[STAT] Total Memory Allocated to particles: 4 GB
[PROGRESS] Rendering
[PROGRESS] Rendering: Drawing 189,246,817 particles
[PROGRESS] 71% complete
[PROGRESS] 72% complete
[PROGRESS] 73% complete
[PROGRESS] 74% complete
[STAT] Section "Rendering:Matte":
[STAT] Total 00h 00m 00.000s Called 0 times
[STAT] Section "Rendering:Sorting":
[STAT] Total 00h 00m 00.000s Called 0 times
[STAT] Section "Rendering:Drawing":
[STAT] Total 00h 00m 14.079s Called 1 times Avg 00h 00m 14.079s
189million particles. nbd.
Yay!
What was the Falloff setting? The UI is a bit misleading since that value should be in the range from 0.0 (no falloff) to 1.0 (falloff everywhere).
If it was set to 0 (default), try setting it to 1, it might look even smoother and less spotty. Also, I don’t think you really need 190 million particles in this case 
A value of 0.5 will keep the influence constant for half the influence radius, then falloff in the second half of the radius. Basically low values keep the value constant throughout the influence radius of the particle, while high values produce more gradual falloff against the edges of the influence region.
Check out the bottom image in this article for an example:
thinkboxsoftware.com/news/20 … eview.html
no you’re right I can easily cut down the amount of particles. Its getting closer. Ill be up later tonight in my office playing around more. 24 3.0ghz CPUs and 64gb of ram should make the render time much faster than my i7 dell m6700.
Ill post the scene file for you to play with as well in the scene files thread.
OMG!
Looks like C4D emitter is allocating space for all particles in the beginning, and saved static particles sitting around until their turn comes to start rendering?!
Once I saved a PRT file, I realized every frame saved 50K particles, and when I loaded them, I got a box of particles ready to start moving. That’s disappointing 
Anyway, It also looks the the other bug is related - to produce motion blur from camera and matte objects, C4D has to go back half a frame or 1/4 a frame to take samples and this of course seems to invalidate the simulation, producing the box of particles with no forces. Wonderful. Not even saving to PRTs can save this.
I guess to solution is not to use ancient particle technology…
The standard particle emitter and thinking particles are a terrible source for particle data. It requires so many hacks to make it work that since x-particles inception I’ve stopped all dev on Maxon supplied solutions for particles.
Bobo,
This is invaluable information. Thank you so much for taking the time to go into this much detail for us Krakatoa Noobs!
Cheers
Tim
No problem, you are welcome!
Just wanted to mention that the “glowing ball at the origin” problem with old-style Emitters was fixed in Beta 3. We used the alive and visible bits of the particle data to skip particles that should not be rendered, so now Krakatoa only gets the actual alive particles out of the bunch…
hi Bobo this is to much information i wish one day meet you thank you genius and i wish other think box plugin come to c4d like max and maya this is will be huge and game changer thank you again for the information 
