3D Gaussian Splatting in Unity

A photo captures a scene from a single point of view. In this post, we’ll turn 100 ordinary photos of a building into a 3D scene that you can fly around in Unity, in real time. The technique behind it is called 3D Gaussian Splatting (3DGS), and since the original paper came out in 2023, it has become a popular way to capture real places in 3D.

How It Works

In classic computer graphics, a scene is made of triangles that the GPU projects onto the screen. Gaussian Splatting works the same way, but instead of triangles, the scene is made of millions of small, fuzzy, semi-transparent blobs called Gaussians. Each one is “splatted” onto the screen, and together they blend into a picture.

Every Gaussian has a position, a shape (round, flat like a disc or thin like a needle), an opacity and a color. The color is stored as spherical harmonics, which let it change with the viewing angle, so splats can capture shine and reflections.

The Gaussians are placed and optimized through an iterative process:

1. Start from a rough point cloud of the scene, with a Gaussian on every point.
2. Render the scene from the viewpoint of one of the photos.
3. Compare the render to the real photo, and nudge every Gaussian to make the difference smaller.
4. Every few hundred steps, split Gaussians where detail is missing, and delete the ones that have become almost invisible.

The rough point cloud and the viewpoint of each photo come from a separate tool, so the whole process is a pipeline of four programs:

flowchart LR
    Photos[Photos] --> COLMAP[COLMAP]
    COLMAP -->|camera poses and points| Brush[Brush]
    Brush -->|raw PLY| SuperSplat[SuperSplat]
    SuperSplat -->|cleaned PLY| Unity[Unity]

The Photos

For a first attempt, it helps to use photos that are known to work, so we’ll work with the Gerrard Hall dataset. It has 100 photos of a brick building, taken from many angles around it.

Every step of the pipeline gets slower with more pixels, so we first shrink the photos to 1000 × 667 with a custom Python script:

import os
from PIL import Image

def batch_resize(input_dir, output_dir, max_size):
    """
    Resizes all JPEG images in the input directory and saves them to the output directory.

    :param input_dir: Folder containing original images.
    :param output_dir: Folder to save resized images.
    :param max_size: Tuple of (max_width, max_height).
    """
    # Create output directory if it doesn't exist
    if not os.path.exists(output_dir):
        os.makedirs(output_dir)

    # Loop through all files in the input folder
    for filename in os.listdir(input_dir):
        # Process only JPEG files
        if filename.lower().endswith(('.jpg', '.jpeg')):
            input_path = os.path.join(input_dir, filename)
            output_path = os.path.join(output_dir, filename)

            try:
                with Image.open(input_path) as img:
                    # .thumbnail() scales the image down to fit within max_size
                    # while perfectly maintaining the aspect ratio.
                    # LANCZOS provides the highest quality downsampling.
                    img.thumbnail(max_size, Image.Resampling.LANCZOS)

                    # Save the new image, optimizing the file size
                    img.save(output_path, "JPEG", optimize=True, quality=85)
                    print(f"Success: {filename} -> {img.size}")

            except Exception as e:
                print(f"Error processing {filename}: {e}")

if __name__ == "__main__":
    # Define your folders and desired maximum dimensions here
    INPUT_FOLDER = "./images"
    OUTPUT_FOLDER = "./resized"
    MAX_DIMENSIONS = (1000, 1000) # (max_width, max_height)

    batch_resize(INPUT_FOLDER, OUTPUT_FOLDER, max_size=MAX_DIMENSIONS)

COLMAP

Before training any Gaussians, we need Structure from Motion (SfM), which estimates the camera poses (position and orientation) from the images. For this, we use COLMAP. It finds keypoints in every photo, matches them across photos, and then solves for the camera poses and the 3D points at the same time. The end result is a “sparse” point cloud along with the camera information.

On macOS, COLMAP is available through Homebrew (brew install colmap).

First, we create a project folder called 3dgs-project and place the images in an images subfolder. To get the sparse point cloud, we run the following commands from the COLMAP CLI documentation:

colmap feature_extractor --database_path 3dgs-project/database.db --image_path 3dgs-project/images
colmap exhaustive_matcher --database_path 3dgs-project/database.db
mkdir -p 3dgs-project/sparse
colmap mapper --database_path 3dgs-project/database.db --image_path 3dgs-project/images --output_path 3dgs-project/sparse

• feature_extractor finds keypoints in every photo: small, distinctive patches, such as corners, that are easy to recognize across images. It stores them in an SQLite database called database.db.
• exhaustive_matcher compares the keypoints of every photo with those of every other photo, and keeps only the matches that fit a consistent geometry between the two images.
• mapper starts from two photos and adds the others one by one, refining all cameras and points as it goes. The result is the camera poses and the sparse 3D points.

COLMAP stores the result in sparse/0. The 0 stands for the first model, since COLMAP creates several if it can’t connect all the photos into one scene.

Brush

The official 3DGS implementation requires an NVIDIA GPU. Instead, we’ll use Brush, which is built on WebGPU and runs on almost any GPU, including the ones in Apple Silicon Macs.

Brush is written in Rust, so we first install the Rust toolchain with rustup, then clone the repository and build it:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
git clone https://github.com/ArthurBrussee/brush.git
cd brush
cargo run --release

cargo run --release compiles Brush and then opens the app. The first build takes a while (about 12 minutes on my machine). After that, the same command starts Brush in a few seconds.

In the Brush window, we open the 3dgs-project folder. Brush finds the images and sparse/0 folders inside it on its own and starts training right away. While it trains, you can watch the scene improve and fly around it. By default, Brush trains for 30,000 steps and saves a copy of the scene every 5,000 steps.

Training starts with one Gaussian for each of the ~40,000 points generated by COLMAP. The number grows quickly: by step 5,000, there are already a million Gaussians. We can stop training once the scene looks good enough. I trained for about 10,000 steps and ended up with 1.7 million Gaussians.

We can then export the scene as a .ply file.

SuperSplat

Next, we open the .ply file in SuperSplat, a splat editor that runs in the browser.

The building itself came out sharp, but it sits inside a thick shell of blurry trees, sky and stray Gaussians floating in the air. These need to be cleaned up to get a clear view of the building.

We can select and delete the stray Gaussians using Rectangle Selection (R) and Delete. After the cleanup, the building is much easier to see.

The brick walls, windows and doors look almost like the photos. The roof is the weak spot: the photos were taken from the ground, so the camera never saw it from above, and it’s covered in white smudges.

Finally, we export the cleaned scene with File > Export > PLY. This is the file that goes into Unity.

Unity

To display the splats in Unity, we use UnityGaussianSplatting.

git clone https://github.com/aras-p/UnityGaussianSplatting.git

After cloning the repository, we open projects/GaussianExample-URP from Unity Hub with Add > Add project from disk. The project was made with Unity 6.0, so we can either open it in that version or let Unity Hub upgrade it to a newer one. I upgraded it to Unity 6.5, which needs one small fix.

Fixing the Compile Error

In Unity 6.5, the project fails to compile with this error:

GaussianSplatRenderer.cs(187,74): error CS0619: 'ChangeGameObjectOrComponentPropertiesEventArgs.instanceId' is obsolete: 'instanceId is deprecated. Use entityId instead.'

To fix the error, make the following change to package/Runtime/GaussianSplatRenderer.cs:

--- a/package/Runtime/GaussianSplatRenderer.cs
+++ b/package/Runtime/GaussianSplatRenderer.cs
@@ -181,7 +181,7 @@ namespace GaussianSplatting.Runtime
                 if (stream.GetEventType(i) == UnityEditor.ObjectChangeKind.ChangeGameObjectOrComponentProperties)
                 {
                     stream.GetChangeGameObjectOrComponentPropertiesEvent(i, out var change);
-#if UNITY_6000_6_OR_NEWER
+#if UNITY_6000_5_OR_NEWER
                     var obj = UnityEditor.EditorUtility.EntityIdToObject(change.entityId);
 #elif UNITY_6000_3_OR_NEWER
                     var obj = UnityEditor.EditorUtility.EntityIdToObject(change.instanceId);

Back in Unity, the scripts should now recompile without errors.

Loading the Splat

1. Open the sample scene Assets/GSTestScene. Its render pipeline is already set up for splats.
2. Hide or remove the existing Gaussian Splat Renderer objects in the scene.
3. Create a folder named StreamingAssets under Assets, and copy the .ply file from SuperSplat into it.
4. Create an empty GameObject with Create Empty, and add the Gaussian Splat Renderer component to it.
5. Drag the .ply file from the Project window onto the component’s PLY / SPZ field.
6. Select the object and press F in the Scene view to zoom to it.

COLMAP doesn’t know which way is up, and it uses a different coordinate system from Unity, so the building shows up upside down and mirrored. Setting the object’s Scale Z to -1 undoes the mirroring, and a Rotation X of about -180 turns the building upright.

The building looks just as sharp as in SuperSplat, and it renders in real time: the Stats panel shows about 60 FPS.

Conclusion

In this post, we went from 100 photos to a 3D scene that runs in real time. COLMAP works out where the photos were taken, Brush turns them into Gaussians, SuperSplat clears away the stray Gaussians, and Unity renders the result.