Flax Facts #26 – Build System

How we build Flax?

Hello and welcome to the Flax Facts! It’s our dev blog series that showcases some of the engine development highlights. This time we are going to talk a bit about build system we use to compile and deploy Flax Engine. This will include many interesting insights and discussion related to this topic. Without further due, let’s jump into it!

msbuild

In the past Flax was running only on Windows and since we use C++ with C# as main languages (for Editor and Engine) the best choice was to stick to msbuild system that has great integration with Visual Studio which we use as our main development IDE.

However, during last years engine was growing (now has more than 500k lines of code). Some parts of the engine are not used across all platforms. Also, we wanted a more feasible solution rather than editing .xml files for build configuration for msbuild. During that time we developed other tools required for development such as Flax.Deps for dependencies building and pulling, Flax.Deploy for building final packages for the Flax Store. MSBuild is a great tool but we wanted more. We wanted to have more control over the product pipeline so we had to switch.

Flax.Build

Firstly after playing around with CMake, GN, Make, SCons, SharpMake we got enough ideas to create own system. The next step was to prepare a list of featured we need:

• Compiling and linking source files (C++ codebase)
• Generating project files for IDE (Visual Studio, XCode, etc.)
• Split engine into modules (custom rules for dependencies)
• Use C# for build rules configuration
• Multi-threaded building and fast incremental builds
• Automated deployment and dependencies building

After a month we got a new tool: Flax.Build!

How does it work? All starts with the *.Target.cs file that defines what to build and for which platforms/configurations/architectures. Here is an example for Flax Editor:

public class FlaxEditor : EngineTarget
{
    public override void Init()
    {
        base.Init();

        IsEditor = true;
        ProjectName = "Flax";
        ConfigurationName = "Editor";
        Platforms = new[]
        {
            TargetPlatform.Windows,
        };
        Architectures = new[]
        {
            TargetArchitecture.x64,
        };
        GlobalDefinitions.Add("USE_EDITOR");
        Win32ResourceFile = Path.Combine(Environment.CurrentDirectory, "Source", "FlaxEditor.rc");

        Modules.Add("Editor");
        Modules.Add("CSG");
        Modules.Add("ShadersCompilation");
        Modules.Add("ContentExporters");
        Modules.Add("ContentImporters");
    }
}

As you can see it defines the most basic information about the target and implements the list of root modules required for this target. Each module is a *.Module.cs file with source files list, build rules and other dependencies definitions. Here are example modules we use:

public class Editor : EditorModule
{
    public override void Setup(BuildOptions options)
    {
        base.Setup(options);

        options.PublicDependencies.Add("Engine");
        options.PrivateDependencies.Add("Main");
        options.PrivateDependencies.Add("Flax.VisualStudio.Connection");
        options.PrivateDependencies.Add("ShadersCompilation");
        options.PrivateDependencies.Add("MaterialGenerator");
    }
}

public class Main : EngineModule
{
    public override void Setup(BuildOptions options)
    {
        base.Setup(options);

        // Use source folder per platform group
        options.SourcePaths.Clear();
        switch (options.Platform.Target)
        {
            case TargetPlatform.Windows:
                options.SourcePaths.Add(Path.Combine(FolderPath, "Windows"));
                break;
            case TargetPlatform.XboxOne:
            case TargetPlatform.WindowsStore:
                options.SourcePaths.Add(Path.Combine(FolderPath, "UWP"));

                // Use Visual C++ component extensions C++/CX for UWP
                options.CompileEnv.WinRTComponentExtensions = true;
                options.CompileEnv.GenerateDocumentation = true;
                break;
            default:
                throw new Exception("Unsupported platform " + options.Platform.Target);
        }
    }
}

public class OpenFBX : ThirdPartyModule
{
    public override void Init()
    {
        base.Init();
        LicenseType = LicenseTypes.MIT;
    }
}

public class TextureTool : EngineModule
{
    public override void Setup(BuildOptions options)
    {
        base.Setup(options);

        switch (options.Platform.Target)
        {
            case TargetPlatform.Windows:
                options.PrivateDependencies.Add("DirectXTex");
                options.PrivateDefinitions.Add("COMPILE_WITH_DIRECTXTEX");
                break;
            default:
                throw new ArgumentOutOfRangeException();
        }

        options.PublicDefinitions.Add("COMPILE_WITH_TEXTURE_TOOL");
    }
}

Build rules

Modules can define public and private: dependencies, definitions, and include paths. Also, some modules are using normal build while others are included as header-files-only (eg. VulkanMemoryAllocator) or prebuild (eg. mono, PhysX). Every module has own compilation environment that inherits from the target environment and dependencies. For instance, if module XXX uses the private dependency to module YYY, then all public symbols are derived from the YYY module to module XXX.

Using a modular design helps with rapid development and improves overall build time.

Results

Finally, it’s time for the comparison. As stated before msbuild is a great build system but our Flax.Build has been created to match all our needs and fits better into the current ecosystem. But is it faster to build Flax now? Sure it is!

Configuration	msbuild	Flax.Build	Difference
Editor (Debug, Win64)	2:16 min	1:35 min	1.43x
Editor Incremental* (Debug, Win64)	13 s	15 s	0.86x
Game (Debug, Win64)	1:59 min	1:16 min	1.63x
Game (Release, Win64)	2:58 min	1:23 min	2.14x

*Incremental build after editing a single .cpp file.

All times were collected on Intel i7-8700K CPU with SSD drive. For reference, Unreal Engine 4 full build takes around 36 min on this machine.

As you can see the overall build time got reduced. We still need to work on faster incremental builds (improve incremental build cache loading and validation). However, the results are pretty satisfying. This huge reduction in build time is mostly related to the ability to exclude large portions of code for compilation (due to modular design) and improved code configuration (instead of using Config.h files or global defines we can inject per module local defines that affect only private compilation environment).

Summary

We are super happy to use Flax.Build tool. It helps to scale the project and expand into other platforms (Linux support incoming). It’s also easier to develop the engine in a team as there are no merge conflicts for project files.

Finally, anyone who worked or is working on bigger projects will agree that having less external dependencies is always good. This philosophy applies nicely to Flax as we try to don’t bloat engine with external libraries and implement many features internally.

Please let us what do you think about this solution! That’s all for now, see you soon!