When chosing a location to capture I looked at Alexandria Road, I have previously been to this location, so I knew that there wouldnt be many people walking around and the location fit the style of my project well. To capture the location I decided to use multiple different methods, as this technology is very new there isnt a “right” way to do it, but there is lots of them.
I chose 4 methods of capture.
- iPhone Polycam
- iPhone Luma 3D
- 360 Camera Video
- DSLR RAW Images
First I used Polycam, I have used this app before so I knew the basics of how it works in regards to scaning and creating 3D assets. For this capture I walked around in a sort of spiral pattern trying to capture as many different angles, positions and raw data as I could. I had a conversation with a VFX supervisor that just finished giving a talk on the future of Gaussian Splatting within Virtual Production and asked for any advice on how to capture my scene. They said that in Polycam you have the ability to export all the captured photos that it produced while walking around capturing your scene, this data can then be passed into COLMAP and Postshot later on.
Then I captured using Luma 3D. This is one of the most popular apps for Gaussian Splatting and has lots of community uploaded datasets. I used this for my initial test and it didnt have the best quality, but I still wanted to try it for the location. The outcome was very messy and the worst of them all. There is also no option to export the photos to try and reprocess it later.
Then I captured using the 360 Camera. For this capture I started recording while holding the camera above my head to try and reduce my footprint within the shot. I then moved in a zig-zag pattern down the tunnel as I have heard this pattern to give good results when using 360 cameras.
Finally I started capturing using my DSLR camera. For this I decided to walk down the tunnel standing at one side and moving down while taking pictures of the opposite side, I tried to move slowly and make sure every new image had some part of the last image in it so that when I try and reconstruct the camera postions later it has an easier time placing them.
After capturing this data I imported their respective data into COLMAP and Postshot. Postshot is a software that can handle nearly all aspects of creating a 3D Gaussian Splat. You can import raw camera image data and it will work out the positions of the cameras, create the point cloud and work out how to turn it into a Gaussian Splat. However the camera position tracking in Postshot is very slow and isnt as advanced as some other open source options. This is where I use COLMAP, COLMAP is an open source software that takes an input of images and plots the position of each image in 3D space, then creates a rudimentary point cloud. COLMAP is a lot faster and I have heard many good things from various forums and people working and developing this technology.
The data I captured using Luma 3D was locked into that app and was of such a low quality I didnt persure it any further. The image data from Polycam was sent to my comptuer and processed through COLMAP and Postshot, it resembled the tunnel but still had a lot to be desired, this dataset also consisted of almost 3000 images so it took a long time to process. I then processed the 360 camera video data, I did this by putting the video file into another open source software, this software split the video up into frames, then split these frames into flat images, I used 6 images per 360 frame, this split it up into a “cube” and gave me a flat image from each side of this “cube”. I could then past these images into COLMAP to get the camera positions. Once I had the camera position data from COLMAP I could cobine this data with the images and drag them into Postshot, Postshot recognises there is camera position data and skips this stage, takes the camera positions and skips straight to training the Gaussian Splat. Finally I tried processing the images from the DSLR camera. I tried tracking the camera in COLMAP and Postshot but kept getting the same problem, because I took the photos direcly facing the wall and that wall being almost identical on both sides the software thought it was just one single wall. I tried tweaking various settings but could not get it to distinguish the two different sides. After my tests I saw that the DSLR had very high detail in small locations and the 360 video was good at rebuilding the whole location (with less quality). So I planned to go back to the location and shoot again with the DSLR, but using the movement pattern I did with the 360 camera.
With this new data captured, I imported the photos into Photoshop and batch edited the photos. I tried to make the photos as flat as possible so there was no over or under exposed areas, this allows for better tracking. I also used the AI Denoiser to try and give COLMAP and Postshot as clean of a data set as I could.
After the photos were processed I started reconstructing in COLMAP.
This took a very long time to process even with a high end GPU, but once it was done I finally had accurate camera position data and points. I exported this camera data into a file, this file can then be imported alongside the images into Postshot, skipping the reconstruction and using the already created data instead.
With this new approach I had a much better splat. There was still areas for improvement as the floors and walls had a lot of holes and visual artifacts. I think this happened as I picked a difficult scene to reconstruct, It features lots of repeating points of intrest and is a dark, somewhat confined area.
Postshot also has the ability to import Gaussian Splats into Unreal Engine via its own plugin. I will not be using this is my project but it was interesting to see how well it was implimented.
After the splat finished training I was able to export it as a .PLY into Houdini. For a basic test I used the new Gaussian Splat nodes in Houdini and could visualise my point cloud. As this now functions as a collection of points with attributes within Houdini I wanted to also test if I could move these points and see if the Gaussian Splat would still be rendered. I created an attribute wrangle node with some simple VEX code that moves these points based on a sine wave. As this worked well I moved onto more complex edits.
The next progression was POP particle simulations. I created a POP network and added a POP wind to move the particles around. By default the POP network continuously spawns particles at every point based on a value. As I wanted each single splat point to move I had to change this and make it so the network spawns a single particle at each point.
This worked but my desired effect is to have the points transition from Gaussians to particles. I began to experiment and find solutions to achieve this. My initial idea was to create a box that moves across the point cloud and give any points inside that box a group, this would be my mask. I could then set the POP network to only spawn particles on points in that group. However this did not work. As the POP network was set up to only spawn a single particle, it did this on the first frame, so when the box moves and adds new points to the mask, the initial particles have already been spawned and there will be no more created.