This week was spent finding a solution to the particle transition. I tried multiple solutions I thought of in Houdini, but none of them worked. After this I tried looking for resources online, looking for anything specifically related to Gaussian Splatting is very difficult as the technology is still very new, and as this was a very specific use case in just one of many DCCs there was 0 resources. I then turned to generic Houdini particle tutorials, I explored the entire SideFX learn database and couldn’t find anything similar to the effect I wanted. I tried multiple search terms on youtube and google and found one or two tutorials that were similar but they had flaws that meant they wouldn’t work with my project. I turned to GPT and Gemini for solutions, but they provided no help at all. Finally I went back to searching and found one source on creating a particle dissolve effect.
The man concept behind this effect is to have particles spawn on the points every frame, but having a mask that ensures there is a new set of points active every frame that do not repeat. The way this is achieved is by initially setting a group of points as active by giving them an attribute. This is then put into a DOP network that expands the selection by a desired amount. The DOP network can then get the data from the current frame, and compare it to the last frame, any points that are active on the current frame but weren’t on the previous frame get added to a transition group. This group is then used as the mask for the points.
When chosing a location to capture I looked at Alexandria Road, I have previously been to this location, so I knew that there wouldnt be many people walking around and the location fit the style of my project well. To capture the location I decided to use multiple different methods, as this technology is very new there isnt a “right” way to do it, but there is lots of them.
I chose 4 methods of capture.
iPhone Polycam
iPhone Luma 3D
360 Camera Video
DSLR RAW Images
First I used Polycam, I have used this app before so I knew the basics of how it works in regards to scaning and creating 3D assets. For this capture I walked around in a sort of spiral pattern trying to capture as many different angles, positions and raw data as I could. I had a conversation with a VFX supervisor that just finished giving a talk on the future of Gaussian Splatting within Virtual Production and asked for any advice on how to capture my scene. They said that in Polycam you have the ability to export all the captured photos that it produced while walking around capturing your scene, this data can then be passed into COLMAP and Postshot later on.
Then I captured using Luma 3D. This is one of the most popular apps for Gaussian Splatting and has lots of community uploaded datasets. I used this for my initial test and it didnt have the best quality, but I still wanted to try it for the location. The outcome was very messy and the worst of them all. There is also no option to export the photos to try and reprocess it later.
Then I captured using the 360 Camera. For this capture I started recording while holding the camera above my head to try and reduce my footprint within the shot. I then moved in a zig-zag pattern down the tunnel as I have heard this pattern to give good results when using 360 cameras.
Finally I started capturing using my DSLR camera. For this I decided to walk down the tunnel standing at one side and moving down while taking pictures of the opposite side, I tried to move slowly and make sure every new image had some part of the last image in it so that when I try and reconstruct the camera postions later it has an easier time placing them.
After capturing this data I imported their respective data into COLMAP and Postshot. Postshot is a software that can handle nearly all aspects of creating a 3D Gaussian Splat. You can import raw camera image data and it will work out the positions of the cameras, create the point cloud and work out how to turn it into a Gaussian Splat. However the camera position tracking in Postshot is very slow and isnt as advanced as some other open source options. This is where I use COLMAP, COLMAP is an open source software that takes an input of images and plots the position of each image in 3D space, then creates a rudimentary point cloud. COLMAP is a lot faster and I have heard many good things from various forums and people working and developing this technology.
The data I captured using Luma 3D was locked into that app and was of such a low quality I didnt persure it any further. The image data from Polycam was sent to my comptuer and processed through COLMAP and Postshot, it resembled the tunnel but still had a lot to be desired, this dataset also consisted of almost 3000 images so it took a long time to process. I then processed the 360 camera video data, I did this by putting the video file into another open source software, this software split the video up into frames, then split these frames into flat images, I used 6 images per 360 frame, this split it up into a “cube” and gave me a flat image from each side of this “cube”. I could then past these images into COLMAP to get the camera positions. Once I had the camera position data from COLMAP I could cobine this data with the images and drag them into Postshot, Postshot recognises there is camera position data and skips this stage, takes the camera positions and skips straight to training the Gaussian Splat. Finally I tried processing the images from the DSLR camera. I tried tracking the camera in COLMAP and Postshot but kept getting the same problem, because I took the photos direcly facing the wall and that wall being almost identical on both sides the software thought it was just one single wall. I tried tweaking various settings but could not get it to distinguish the two different sides. After my tests I saw that the DSLR had very high detail in small locations and the 360 video was good at rebuilding the whole location (with less quality). So I planned to go back to the location and shoot again with the DSLR, but using the movement pattern I did with the 360 camera.
Left: Original DSLR Capture pattern. Right: New DSLR Capture pattern.
With this new data captured, I imported the photos into Photoshop and batch edited the photos. I tried to make the photos as flat as possible so there was no over or under exposed areas, this allows for better tracking. I also used the AI Denoiser to try and give COLMAP and Postshot as clean of a data set as I could.
After the photos were processed I started reconstructing in COLMAP.
This took a very long time to process even with a high end GPU, but once it was done I finally had accurate camera position data and points. I exported this camera data into a file, this file can then be imported alongside the images into Postshot, skipping the reconstruction and using the already created data instead.
With this new approach I had a much better splat. There was still areas for improvement as the floors and walls had a lot of holes and visual artifacts. I think this happened as I picked a difficult scene to reconstruct, It features lots of repeating points of intrest and is a dark, somewhat confined area.
Postshot also has the ability to import Gaussian Splats into Unreal Engine via its own plugin. I will not be using this is my project but it was interesting to see how well it was implimented.
After the splat finished training I was able to export it as a .PLY into Houdini. For a basic test I used the new Gaussian Splat nodes in Houdini and could visualise my point cloud. As this now functions as a collection of points with attributes within Houdini I wanted to also test if I could move these points and see if the Gaussian Splat would still be rendered. I created an attribute wrangle node with some simple VEX code that moves these points based on a sine wave. As this worked well I moved onto more complex edits.
The next progression was POP particle simulations. I created a POP network and added a POP wind to move the particles around. By default the POP network continuously spawns particles at every point based on a value. As I wanted each single splat point to move I had to change this and make it so the network spawns a single particle at each point.
This worked but my desired effect is to have the points transition from Gaussians to particles. I began to experiment and find solutions to achieve this. My initial idea was to create a box that moves across the point cloud and give any points inside that box a group, this would be my mask. I could then set the POP network to only spawn particles on points in that group. However this did not work. As the POP network was set up to only spawn a single particle, it did this on the first frame, so when the box moves and adds new points to the mask, the initial particles have already been spawned and there will be no more created.
This week I started practically looking into the Gaussian Splatting implimentation added in Houdini 21.
I used the sample project I got from a video tutorial earlier and went through it breaking dow how it all works. This sample project looks at creating a 3D mesh based on the points, running a vellum simulation on this mesh, then sending that animation data back onto the point cloud. While this is very interesting it will not be useful in my project.
Instead I want to use some kind of particle simulation or large scale animation on the point cloud, I tested a very basic popnet setup to see how the Gaussian Splat would react to it and everything seemed to work as expected.
Then I looked into the rendering setup. While the vast majority of processing such as camera tracking and reconstruction is done before the pointcloud is made, they still need to be set up in order to be viewed correctly. As each point is made up of essentially a coloured circle all the lighting and materials are not needed and can just be viewed as is. However it is still possible to relight a Gaussian Splat and change elements of the render, I might look into these further down the project.
After experimenting in Houdini and seeing what tools were available to me, I was able to go from ideas of a storyboard to getting a first idea down. As this project is focused more on research & development than storytelling my storyboard is very basic and essentially just consists of the “real world” transforming into data and becoming the digital world that isnt quite perfect yet. this project is very valuable to me as a chance to look into this new technology that is very quickly intergrating itself into upcoming professional workflows and especially in the music video industry where my passion lies.
I plan to refine my storyboard by focusing on ~3 key shots I want in my final video. As this is more of a technical proof of concept and research project, the final video will most likely be around the 30-60 second mark, so instead of trying to create a deep cohesive story, I plan to focus on creating short high quality shots that highlight the techniques that were used to create them.
I will also capture a small scale guassian splat of a room. This will be using a DSLR and manually adding the photos to software. I will then explore the possibilities of manipulating the guassian splat and pointcloud within Houdini.
During the capture process I tried to cover as many area sof the room as possible. Since the capture technique is very similar to that of photogrametry, I used the same stratagey while taking the photos. When capturing a single object you want to orbit around the object at multiple heights, as this is a room instead of a single object, I reversed this and orbited the room facing from inside out.
Once I had all the photos It was time to process them. During my research I discovered the software Postshot, this is a dedicated Gaussian Splat training software. It takes all the raw images and reconstructsits position and orientation in 3D space.
