Here’s a recent conversation from the Bevy discord:
Would love to hear what any Pijul developers might have to say about this.
1 Like
Hi! Thanks for the pointer. I wouldn’t mind talking to these folks, because the new Pijul was designed with them in mind!
I wrote a blog post about this very topic a while ago, you can point them to this if you want: Pijul
(or invite them here).
1 Like
By the way, I also totally agree that “the only way to know for sure is to see it in practice”.
In addition to the main incumbent PlasticSCM, another interesting solution in this space is Snowtrack:
Thanks! I hate to sound negative about a project, I think it’s great that people realise that version control is important and has applications beyond source code. However, their readme doesn’t mention what new insight that project is based on. Also, the descriptions of their issues with Git seem to be based on a misunderstanding of LFS (the mention “without LFS” is written on the first bullet point, but really applies to all others, except the issue with very large files on some setups on Windows).
This gives me the idea that SnowFS looks a bit like a reimplementation of Git LFS in TypeScript, sort of an “LFS done right” (and they’re right actually, the LFS tool itself doesn’t need to be fast, the main bottleneck is the network anyway), but then they also mention the idea of rewriting it in C, which would really be like “LFS done… like LFS”.
1 Like
I think the insight that informs and drives their design is their experience as graphics designers in need of a tool that didn’t exist for them. Thus, I expect their main innovation will be in their API design, and the GUI they build on top of it.
They’re self-described as experimental. What do you think about the prospect of them building their product as an extension of Pijul?
I’ve invited them to this topic and the #editor channel on Bevy so they can have the opportunity to speak for themselves 
Yes, I’m obviously open to collaboration. After I wrote that comment, I also saw their benchmarks, they seem much faster than Git LFS, I’m curious about where that comes from, and I’d like to see that compared to just rsync (which is another fast diff).
Sure, LFS was designed at Microsoft, but I guess the engineers who developed it were talking to the maintainers of Git (and we know tthe maintainers of GIt are really good).
Also, Pijul doesn’t yet know how to diff binary files.
Hi, I’m investigating this topic by really slowly building set of microservices in style of Perforce (not yet open source). Git is great for projects which are using only text files, I know that LFS exists but it’s not suited for game development from my personal experience. Perforce quickly becomes awkward with large projects because of it’s repository and branching handlers which forces companies to work on single repo and single branch through commit-based file locking where only you person that locked selected file can modify it. There is also functionality which allows you to map folders from repository to store them anywhere in your filesystem to easly manage it locally. For example, from some papers based on how Google manages their code, everything lives in single perforce repository and each project is it’s own virtual filesystem based on FUSE. Even if Perforce is considered industry standard, everyone hates it but there are no real alternatives than PlasticSCM
Source control in game development is not only about code management. From the view of artist (like me), wide range of tools must be suited for people which are not programmers, and thats the idea behind binary diffs and separated GUIs where they usually drag-n-drop textures, FBX or USD binaries etc. then commit changes and easly can go back to earlier versions. Textures are worst thing because you can operate only on single layer where binary diffing is important.
In all in-house engines and some public ones like UE5 & Unity, have built-in VCS support in editors for programmers to easly control code on which they operate. I’ve seen discussion about built-in Pijul support in upcoming editor for Bevy too.
At the end, I will look into implementing binary diffing and other features needed to make Pijul more friendly for building tools for Artists (Pixar & Dreamworks are using Perforce to manage art) and Game Developers.
1 Like
I’ve contacted the main author of Bevy, but they haven’t replied. There’s a lot to be done in that industry, and I believe good data formats are an important first step towards implementing this. So instead of trying to diff the formats, I’d first try to influence the file formats used by existing tools (I can help) to make a much better editor, for instance with 3D file merges.
Unfortunately, it doesn’t seem to me that a lot of people in that space understand this or are even actually interested in better tools, despite what we hear sometimes (like at the top of this page). But if you know anyone who could be interested, I’m still obviously open to talking to them.
I’m interested in Pijul and 3D art because I have plans for rendering engine which lets you create animated movies in style of Pixar’s renderman or Dreamworks’ moonray where 3D formats are important. Problem with them is that, glTF (json or binary as glb) is not that popular except some open source game engines. Industry used FBX for decades but now they started slowly switching to Pixar’s USD which is even more complex, it’s ASCII + Shader DSL + Binaries where everything is packed together into zip-like format. USD ASCII can also be easly written by hands because of it’s python-like syntax, Both, glTF and USD support extensions which makes it “easy” to extend on the application level which makes it possible to create advanced tooling inside the editor.
Edit: here is Remedy’s talk on USD as perfect 3D format ASWF_Games_Remedy.mp4 - Google Drive
The zipping isn’t a major problem, we can still diff that by unzipping + diffing, and add a zipping filter when outputting the file. This adds a big performance penalty, but is still doable.
I’d be more worried about the unzipped material not being very merge-friendly.
I believe another interesting approach is to just define a new format and add it to Blender. If it is superior for collaboration it will be used by at least some people.
I don’t think that you need another format? glTF created by Khronos is popular and seems to be version control friendly since it’s json with optional binary format glb which bundles textures and mesh data together. It’s extensible in the way where you don’t need to modify Blender source code, you can simple write plugin which handles custom extensions (there are many of them around).
I’ll look at these more specifically, but not any format (even JSON or binary) is easy to merge, which is the main thing I’m interested in. Any format is “Pijul-friendly” in the sense that you can store stuff in patches, but merging and conflicts are another story.
Today I came across SnowTrack.
Hey @pmeunier after few months of research and few talks with Pixar’s and Nvidia Omniverse devs, I’m able to say that USD is made to be merge friendly (ASCII + Random-access Binary) and generally the only VCS friendly format for 3D scenes. If you are interested in merging future patches with USD support, I could forward to you, a few companies which are interested in investing money in the VCS that does have a USD support.
There is official diffing tool example called usddiff, that uses Python API exposed by Pixar’s framework.
If pijul exposed python API, then I could easly create integration for artists to let them commit patches directly from DCCs and have a real-time visual feedback in other tools.
It’s not fully documented yet but there are few open PRs with docs and it’s getting full adoption in DCCs like Blender, Autodesk Maya, Foundry’s Katana (3D lights) + Nuke (VFX compositing + movie editing), Adobe’s Substance 3D and Game Engines like Unreal and Unity.
Personally, I have a WIP, stripped down implementation in Rust, which can be used as replacement for glTF or FBX. Most likely, there will be support only for USDA (ASCII) and USDC (binary), since the USDZ (ZIP) is complex and can be easly composed manually through official python script. Also worth noting that only USDA and USDC are needed for VCS while USDZ is the final archive which you compose for production usage, to be consumed by either renderfarm or game engine.
1 Like
I’m definitely interested. Feel free to share my email address: pe@pijul.org.