We have opened the private beta for XMesh Maya. This has been a long time in coming and it is very exciting that it is finally here.
As we move through the beta and refine the plugin for release, I thought that it’s good time to start pointing our eyes a little farther down the road. To help us do that, we are asking you to fill out a short (3 questions!) survey about XMesh and how you use it.
One thing to note: it would be great if we could have the option to encapsulate all per-frame data into single files. We have severe speed issues with more complex geo spitting out 20 tiny files per frame, it loads fairly slow off the servers (that are usually optimized for larger reads, and not a lot of file open operations).
That being said, I would still prefer to have 1 file per frame, so not an “all in one” solution.
Yes this is on our roadmap and I suspect we will look at packaging the xmesh data just as you describe. It might be reasonable to have a single file in certain cases, but I think the best overall benefit would come from a single pack for a frame to address the problem of loading of many small files efficiently.
Our current priority is to take the Maya XMesh plugin to release. After that, we will look at our user feedback decide the next step. Even as we add support for new applications, that often triggers updates to the core in general. So its not an exclusive thing. But the feedback of what you guys need most and sooner will help us significantly!
im also interested in what peoples thoughts are on Alembic support? seems to fast becoming a standard format for moving between applications, much like Exr did.
i don’t believe alembic is as clearly defined nor as simple a case as openexr. I think openexr worked because it just ‘worked’…which is our goal with xmesh.
by setting the constraints to be narrower [geometry only] and defining and controlling the format ourselves, we can support edge cases and fix them and give facilities what they need, without them having to hack their own version of alembic together for their specific use cases.
i understand that it would be nice to support alembic, but which flavor? from which facility? for what target app? 3D data is not the same between applications and confuse things if we supported a format that wasn’t ready to be supported.
Or at least worked well enough. I haven’t seen any application implement the full EXR spec, which is quite extensive. But at least the most basic cases were well supported. It helped that what it was replacing (HUGE float TIF or HDR) was pretty crappy. Not unlike how XMesh replaces crappy things like FBX or OBJ sequences.
For XMesh (or Alembic), it’s essential that the mapping between applications is well handled. It’s all the little things like velocity units, or tangent basis, or how mapping seams are stored. What do you do about smoothing groups? Or material IDs? FBX is a complicated black box because it’s designed for interchange. XMesh will become a complicated black box once you have interchange (and it’s not just 3ds max roundtripping). Alembic isn’t a black box, but it is complicated for this same reason. Without someone like Thinkbox managing this for you, this would be a pain, and yes, with their black box they can hopefully provide better service.
Hypothetical question for anyone using this for interchange… If there were 4 or 5 available targets/hosts for XMesh, would you know the target at the time of export? Meaning, would prefer to say “Export XMesh for 3ds max” or “Export XMesh for Unity” or whatever, and Thinkbox used some internal mapping so the target was correct, or would you rather have the loader do the mapping based on some sort of metadata in the XMesh like “Exported from Maya”?
If a target was chosen on export, the saver would have all the data needed to make the most optimized files (like saving out smoothing groups instead of explicit normals) and there would be less overhead when reading the files in the target application (like if it was Unity, you would NOT want to doing any conversion/calculation steps at runtime).
If the target was not chosen for export, you’d be able to load the file into any of the host applications using XMesh and get the same (if Thinkbox does it right and the host application is able) results.
Basically, do you expect it to be more common that you would make a “generic” interchange XMesh, or would you be more likely to have a target, like “I’m sending this to Fusion”?
Back to Alembic, though, what about selling an Alembic conversion “host” separately? Like XMesh Alembic would allow you to convert Alembic to XMesh or vice/versa? Not sure what the pricing should be. Depends on the number of users and how much support (from wacky Alembic files?) they actually needed?
I want xmesh to be a lossless format - so no need to save to a format. we just store the historical [all of the material ids for instance] through the entire chain where possible.
so go max ->maya ->max and we tag the metadata along with the asset in the scene and hopefully carry it through, or at least provide it to the TD later in the chain
im still fairly new to this format, so spent some time testing existing Alembic plugins (exocortex.com/products/crate/des … obtainment )
and the native Alembic import/export out of Maya 2013.
had pretty good results using both plugins to bring data back and fourth between maya and max, so it seems at least in some ways there is some standardization of Alembic, at least in the case of what autodesk and the other available plugins.
Im really not sure what the value of Alembic is, but it does seem to cache quite quickly. will post some bench marks and comparisons as we test further.
Last time I checked the Max side of Alembic (in Fall 2012), the Digital Frontier plugin did not support Changing Topology, Material IDs, Vertex Colors, Mapping Channels 2 to 99, Vertex Velocities, Vertex and Face Selections. You could not save the render-time mesh, only the viewport mesh. The Loader did not support custom retiming curves (only time offset) or any kind of Proxy display.
During the same test, the Exocortex plugin did not support Vertex Colors, Mapping Channels 2 to 99, Selection channels, and had the same loader and saver limitations as the free DFX plugin, except it supported retiming curves. It is of course possible that some of these limitations have been addressed in the mean time, I will have to test again.
Performance-wise, in my benchmarks saving 100 frames of a 64 segments teapot with an animated Bend modifier and no topology changes took 61 seconds in DFX, 89 seconds in EC, 21 seconds in XMesh. The Loading of the same sequence in Max took 33, 61 and 27 seconds, respectively. The data size in MB was 344, 311 and 199, respectively.
Adding a new Mapping Channel 3 to the same object and animating the number of segments from 64 to 32 to produce changing topology over time broke the DFX plugin, EC failed to reflect the new mapping channel, but correctly saved the changing topology in 70 seconds vs. 53 for XMesh, loaded in 58 seconds vs. 36 for XMesh, and the data in MB was 430 vs. 187.
At that point, we did not have an official Beta for XMesh Maya, so I did not test the performance aspects with it. In my latest tests, XMesh for Maya saves as fast as Alembic for Maya, but we support bi-directional Edge Smoothing to Smoothing Groups conversion if you don’t want to use explicit Normals. You can still use Explicit Normals on both the Max and Maya side with XMesh, but since explicit normals cannot be deformed in Maya with regular deformers (except by Skin), some of our customers who tried to use Alembic to move static mesh models back and forth found it unusable for that purpose.
XMesh adds the benefit of separate files per frame, allowing multiple savers to process the same animation sequence on the network in parallel, or to perform recursive saving over the same sequence for advanced effects not possible with Alembic (or with plain 3dsMax in general). XMesh also supports saving and loading of the render-time geometry from Max, as well as the viewport geometry (as proxy for display purposes), preserves material assignments within Max and can create matching shading groups in Maya, supports vertex and face selections within Max and can store explicit velocities from changing topology, esp. from Frost and Thinking Particles/Volume Breaker which are notoriously tricky to handle.
But the main benefit of XMesh within 3ds Max is its ability to produce one mesh out of thousands animated objects. Max has always been slower with many small objects vs. one huge object, and XMesh allows you to combine many animated meshes into a single one, while Alembic stores one file, but needs one loader per entity to reload from that file. So if you had 10,000 small meshes animated in a Max or Maya scene and saved them to ABC, you would still need 10,000 Alembic Loaders to play them back in Max. In that case, XMesh would be orders of magnitude faster to play back… I think I will go benchmark that case now
Ok, I created a very synthetic benchmark that demonstrates the benefits of XMesh when working with large numbers of animated geometry objects in Max (have not tested the Maya side yet, but I will).
Historically speaking, XMesh exists because we had to solve exactly this problem in production - we had hundreds of thousands of animated mesh pieces (forming growing crystals), and Max (version 8 at the time) was giving us a hard time.
Here is a script which creates 10,000 boxes with animated random heights over 100 frames. No changing topology, just some vertex animation.
seed 12345
for y = 1 to 100 do
(
for x = 1 to 100 do
(
b = box width:10 length:10 height:(random 1 100) pos:[10*x,10*y,0]
with animate on at time 100 b.height = random 1 100
)
)
$Box*.material = standard()
max tool zoomextents all
Run this in an empty scene.
On my computer, the playback of the resulting 120,000 faces took 418 seconds.
The saving to XMesh took 50.8 seconds.
The playback of the resulting XMesh cache took 38 seconds (exactly 11 times faster).
Then I tried to save with the EC Alembic saver. It took over two minutes to even get to the saving progress bar. After 12 minutes it is still at 10% and going. I will post again once it has finished…
(UPDATE 1: After 30 minutes, Alembic is still at 22% and the ABC file size is already way over the whole XMesh sequence.)
(UPDATE 2: 60 minutes - 44%, 70 minutes - 50%, 90 minutes - 64%…)
(UPDATE 3: Saving finished in 2 hours and 29 minutes. Now trying to load the 10K boxes back into Max… Looks like another hour)
UPDATE 4:
Since it was Canada Day, I let the importing run unattended. When I left it had been running for 1 1/2 hours already, but the progress bar was not being redrawn, so I don’t know how far it was. When I came back many hours later, the boxes were shown in the viewport, but Max was acting up, drawing panels in all the wrong places, and the moment I tried to do Select By Name, it crashed. The backup scene saved in the process was unusable.
So I would claim that saving 10,000 animated boxes is not within the workflows of Alembic in 3ds Max.
Then I tried to use the Digital Frontier plugin, because my test setup did not involve changing topology, just animated vertices. The export took 2h2m30s and produced a file of 1,345,449,323 bytes (1.25GB) size. The importing took about a minute, but the playback was SLOW. It was so slow that the Play button of Max would not stay pressed and uncheck itself after each frame. I think the time to advance one frame was around 15 seconds. So I wrote a little script to force the time slider through 100 frames of animation, and stopped the time. It took 1657 seconds, so in fact the time per frame was around 16.5 seconds. This is 4 times SLOWER than the playback speed of the original 10,000 procedural boxes. So it worked, but just wasted a gig of disk space for slower playback.
For comparison, the EC ABC file was 561,897,014 bytes (535MB). The XMesh data for the same animation was 123,317,249 bytes (117MB), but a large part of it turned out to be User Metadata stored in each XMESH file describing the names of the 10,000 objects saved. I could (and will) provide a way to turn off the saving of the Metadata to conserve disk space in such cases. The actual data stored by XMesh was only 25MB (!). Even in the worst case scenario with all the Metadata, XMesh produced 10.9 times less data and its playback speed was 43.6 times faster when compared to the DF plugin. Without the Metadata, the file size difference between DF ABC and XMesh is more like 51 times.
This seems to be one of the cases where Alembic is the wrong tool for the job when using 3ds Max.
I am going to test and see what happens in Maya in the same situation.
Here are my first results using the DFX ABC file of the 10K boxes:
It loaded successfully in Maya, but took 17 minutes and 23 seconds to play back the frame range in wireframe mode ( that is 1043 seconds).
The XMesh loader played back the same animation in 33 seconds (wireframe mode) and 38 seconds (shaded mode). EXACTLY the same playback time as XMesh in 3ds Max with shaded mode.
Keep in mind that Alembic was designed to retain the entities and allows you to modify each one of them as a regular object after the caching, while XMesh was designed to capture the geometry to be rendered and assumes that it won’t be modified beyond the point of caching (although it IS possible to do so, at least in Max using procedural modifiers or Genome).
The example I posted isn’t in any way extreme - the crystal ship in Superman Returns did not use 10,000 animated boxes, it used hundreds of thousands of animated complex shapes forming crystal structures over time. XMesh was originally developed to help that particular workflow and it handles it really well.
I am in no way claiming that Alembic doesn’t have its place in the pipeline as an animation caching format. Just trying to show the cases where it fails short and highlight the benefits of having a Plan B (or Plan X, if you want )
Alembic might take a couple more years to become more mature and provide what YOU need, and not what sony&ilm needs (which does not even overlap between those two companies in some cases).
Their hdf5 based storage is about 2-5x larger than xmesh, and horrendeously slow. For example right now, we have 3-5 megabyte transform only caches (no pointcache in it, no geo, nothing, just a hierarchy of animated nulls) that take 1-2 mins (!) to load per model. They are now introducing a different internal storage library for this reason.
Even though we are using one of the 3rd party alembic plugins, we still had to build our own version for maya to be able to properly use alembic files handed off to use from other vendors. Compatibility varies greatly between different implementations. An alembic coming from maya’s base distribution will not work for rendering in max. You will be forced to convert them to exocortex’s version, and then, at that point, whats the reason for even using alembic in the first place?
Also, in terms of functionality alembic is in a limbo between xmesh and fbx. It supports more than xmesh does (for example, animated nulls, hierarchies and such), but also less, there are problems with normals, material IDs, basically anything max specific. I dont know if that changed since, but alembic v1 specs didnt even include custom uv channels (just 1 per object), so you kinda had to use a custom implementation to do that. It exports into 1 single file, which if you have a corrupted frame in a long/large fluid sim cache means you have to recache the whole thing, and not just replace the faulty frame.
It has less features than fbx, which was initially on purpose ( as in “lets focus on what people actually need, not what we think they might ever want” ), so it has no shader support and other various funky FBX functionality that never really worked.
Anyway, for most of the cases, what i want is 100% fool proof, round-trippable, non-retriangulating, vertex count & index & smoothing group maintaining, fast geo transfer between max, maya and nuke that works with changing topology.
As a phase#2 on that, it would be nice to provide animation curve caching functionality as well of course, thats nice, especially that most facilities end up having to write their own code for that. But still, fast geo & pointcache is the most important.
As part of my 10K Box Benchmark series, I simplified the test even further.
I created 10,000 cubes in Maya in the same distribution, but WITHOUT ANY animation whatsoever.
I saved them using the built-in Alembic implementation and got an ABC file with size of 163,569,517 bytes (159MB).
Same objects saved as one cache using XMesh for Maya produced 1,051,894 bytes (1MB).
You do the math.
We have used xmesh quite successfully for this moving animated characters from Maya into max for FX and rendering. In general this has worked quite well, though we had to write a few tools, as we were using the xmesh Maya alpha.
The biggest issue we ran into was when we wanted separate caches per object, this meant rerunning xmesh so many times that it became cumbersome in terms of time needed. As Bobo mentioned, the original implementation was for caching large geometry sets for lighting. But it seems as the tool matures its more useful to offer as many post editing options as possible, as well as a simpler way to define parts of the cache as individual editable objects without needing to rerun xmesh.
I believe what you can do is actually doing a cache process where you do the time loop yourself, and for each timestep you cycle through all objects you want to cache out and create their sample for the current frame. Im not 100% sure we are doing that right now with xmesh in maya (as we tend to do 1 mesh exports), but we are doing it with pointcaches, transform caches and all that.
something like (warning, bad pseudo code coming up):
for sObject in vCacheTheseObjects:
xmesh.startCaching( sObject )
for fTime in range(fStartFrame, fEndFrame):
cmds.currentTime(fTime)
for sObject in vCacheTheseObjects:
xmesh.pleaseSaveThisMeshForMe( fTime, sObject )
for sObject in vCacheTheseObjects:
xmesh.finishCaching( sObject )
Xmesh, and the Thinkbox toolset is at the heart of our data transfer pipeline. From prepping complex dynamic systems for rendering, to moving character animation between 3ds max and Maya, Xmesh is the tool of choice!
feel free to edit my quote so I sound less like a knuckle dragging neanderthal
)