Game Engine Dev, Explained for Non-Programmers: Building the Game
This series can be read out of order, but here are some navigation links for your convenience:
<<Introduction | <Previous post | Next post>
I released a demo video recently showing off a bunch of the engine’s basic features! Here it is in case you missed it:
I… might be a bit further ahead than these posts maybe had you believing. Can’t spend all my time on my blog after all! So, uh, sorry for spoiling the next few posts I guess.
Now that the engine code has a functional core in the entity system and game loop, I can start actually writing code that will, y’know, do stuff.
Like the test app I wrote, setting up a display system involves just a few calls to the framework1. I’ll talk about what goes into a fully-fledged display system more in a later post, but for now I’m just setting up a simple texture I can draw to and clear every frame (sort of like a digital canvas), then scaling that texture to an application window displayed to the user. Once that’s done, I can write code in the main loop or in entity components that draws stuff to that texture, and it’ll show up on screen!
At this point I’m still kind of limited though. The framework provides functionality for drawing ‘primitives’ like lines and triangles, but last I checked it’s not 1962 anymore, it’s at least 19742; time to get some sprites in! To do that, I’ll need to load some kind of image data into memory. I could just use the framework to load individual .png files directly, but once a game starts using a lot of sprites this can get quite slow for several reasons, so I’ll want to pack sprites into larger image files. Also, hm, how am I gonna organize all those files? These are animated sprites, each frame is its own image file! And anyone doing art is probably gonna be using a program like aseprite, are they gonna have to export and sort everything themselves? And where am I gonna put other assets like text files or sounds? Come to think of it, compiling the engine code manually every time from my command line is getting kinda tedious…
Ok, it’s about time to set up a build system.
What’s that?
Build systems come in many shapes and forms, but they can generally be defined as “automated processes that turn a program’s source code and other asset files into a set of files that can be run directly” (for simple programs, you could consider the “build system” as just being the compiler which turns your source code into a single .exe file that anyone can run, but, as we just saw, more complex programs like games tend to have more elaborate requirements). If you pop open your steam directory and open up any game’s install folder, you won’t (often) be able to see a game’s source code or view/edit its assets directly, what you’re seeing instead is the output of that game’s build system.
So what are my goals? Actually, pretty specific. The shortcomings of Game Maker’s build system remain a thorn in my side to this day, creating something that avoids making the same mistakes is a huge reason I’m developing my own engine in the first place. With that in mind, here’s the plan:
Any part of the game’s files that can be built individually should be. No need to rebuild every sprite when changing a line of code.
An automated file watching system to ensure that files are rebuilt as soon as a change is detected. This means the build system is always running in the background to ensure the game is ready to be launched as soon as possible. Normally, a build system would only start running the moment the user manually triggered it, forcing them to wait for it to finish before the program actually launched. This helps avoid that3.
Asset packing, which is basically just squeezing the raw data from multiple asset files into one big file that’s just a blob of ones and zeroes. This speeds up loading times in-game by reducing the amount of file-system requests I need to make, as well as slightly obfuscating the assets to make it harder for inexperienced users digging into the game files to mess with them.
Support for building .aseprite files. Aseprite is basically the industry standard for pixel art, and is what’s typically used to make the bulk of the art and animations for the sorts of games I’m developing. Unfortunately, like most image editing programs, it’s not standard enough that its project files can be directly read in the same way that .png files can. This usually means that aseprite project files need to be exported to a set of intermediary files in order to actually be legible. But automating this process as part of my build system (as well as exporting additional data such as frame timings) would allow me and other team members to work directly on aseprite files and see the results reflected in-game without any need to manually export things, a huge quality-of-life improvement.
A quick way to both run the current build and to initiate a new build from scratch. Having e.g. a keyboard shortcut to run the game makes it nice and easy to test changes on a whim. And initiating a build from scratch might take longer than running using any existing built files, but it’s a crucial ability to have given the automated file watching process might introduce errors or fail to keep up in rare cases.
Ability to ‘hot-reload’ assets at runtime. What this means is that it should be possible to change a non-code asset after the game has been launched and see the changes show up in-game. Now, there’s a limit to how robust this can be without putting in a ton of effort, but even a simple, slightly unstable version of this is incredibly useful as a developer-only tool.
Speed! The build system should be as fast as possible, wherever possible. This is a huge deal when it comes to iterating on a game, you want to be able to see changes you make reflected in-game as soon as possible. If it takes too long, your attention will wander and maintaining a tight iterative feedback loop in your own head becomes much harder. Say you wanted to tune a character’s walking and running speed to be just right. How much easier would that be if it took half a second for your changes to show up vs. 7 minutes? You might have noticed a lot of the features listed here are designed to cut down on the time a developer spends idly waiting; that’s all in service of the same goal.
Lofty goals! So how do I go about making something like this? It can’t quite be a part of the rest of the source code, that’s what we’re trying to build after all! In theory, I could also make all this using odin, it would just be a smaller, simpler set of programs that would be stored and compiled separately. This might even be one of the more performant options. But to be honest, it’s not exactly what the language was built for. A systems language makes sense when you’re trying to quickly manipulate a large, complex set of data in RAM. But most of the operations involved in a build system will either be requests to the filesystem or to other programs, which means the program instructions themselves won’t make up that large a portion of the overall execution time.
In that case, it makes more sense to look at a some sort of scripting language designed for these sorts of operations. The gain in ergonomics from e.g. not having to constantly recompile small programs (since interpreted scripts can just be run directly without needing to first compile them) or deal with odin’s more finicky text string manipulation seems like it would greatly outweigh the possible performance downsides. A popular language like python or lua could do the trick, but ultimately I decided to write most of my scripts in powershell, microsoft’s scripting language designed specifically for these sorts of tasks. And yeah, even at first glance it’s obvious that powershell makes it very easy to do file operations and to execute command line programs (it’s also probably as optimized as it gets for that sort of stuff… for an interpreted language). There’s definitely some syntax weirdness in other places, but it’s clearly the right tool for the job.4
Still though, the self-contained nature of each script makes it much easier to incorporate other tools into the build system. Jumping ahead a little, I ended up using a small lua script as part of the sprite exporting process to take advantage of aseprite’s scripting API, and the aformentioned asset packer ended up being written as a small odin program, since it involves manipulating a lot of raw data.5
And file watching?
So, tracking changes to files is a problem without an obvious approach. A while back, my thought was that I’d have to keep some kind of ledger that could be quickly checked against every time a build occured to see if anything had changed. Fortunately, after looking into it some more, I found out that there are a number of fleshed out file watching programs that can be customized to suit one’s needs. A file watcher is a program that runs in the background and periodically checks a specified directory for changes, and which can do stuff in response. I’m sure the details of how it does this efficiently are fascinating, but fortunately I won’t have to bother with them!
I decided to go with Watchman, a program developed at Facebook. It seems to be intended more for web stuff, but it’s simple and versatile enough to work well for me. By setting up a series of “triggers” (again, via a powershell script), I can have watchman pass information about individual changed files to my other scripts, to do with what they will.6
Waow!
That’s pretty much how the build system works, at a high level. To summarize, Watchman is initialized the first time the user builds the game, watches for changes in files in a project’s asset folder, and any changed files get processed by scripts and outputted into a dedicated ‘build’ folder. The build in this folder can then be run using a special script, tied to a keyboard shortcut, that makes sure everything’s done building correctly before opening the game’s executable.
For code, rebuilding is about as simple as calling the odin compiler whenever a change is detected in a .odin file7, but other types of assets each have their own special little process. Since each process closely tied to how that type of asset is used in-engine, I’ll talk more about them in detail when I get around to how that asset type is handled. And in fact, next time I’ll get back to the sprites, so look forward to it! But for now, you can check out the build system in action in the demo video linked above!
<<Introduction | <Previous post | Next post>
I really hope wikipedia got these dates right.
The more astute among you might realize that this tactic won’t help that much in cases where the user makes a change then wants to run the game immediately, but it’s often the case that some change will be made long before the user wants to actually run the game (such as when pulling a change from a remote repository or making several edits in a row to different assets). In cases like these, it prevents rebuilding work from piling up. It also eliminates the need to manually track and initiate rebuilds of specific individual assets.
Also, since powershell is part of the Windows OS, scripts can be run very easily, and it’s one less thing that needs to be installed for anyone else trying to use the engine. This does unfortunately lock me even more into working on Windows, but, frankly, being able to test on the actual OS that most end users will be on is the main reason I’m stuck here. If you don’t have this problem, do your part and switch to linux today!
How exactly am I packing my assets? There are a lot of solutions out there for asset packing file formats, but the problem seemed easy enough that I ended up deciding to just write my own dead simple format. This gets a bit technical, but essentially I’m just writing a stream of raw bytes to a file. For each type of asset, the program will search a provided directory for the requisite files. It then writes the type of asset (each type is a assigned a numerical ID) and the number of assets of that type. Then, for each file, it writes the size of the file in bytes along with the raw data from that file. Then, once the game is running, I can load all the packed assets from the file just by running the same process in reverse.
One convenient Watchman feature: when restarting the program, it will treat all existing watched files as newly changed, which means it will pass them all along to their corresponding scripts at once. This makes it very convenient to build the game again from scratch!
About as simple. I ended up deciding to autogenerate some code for components due to the limitations of Odin’s strict type system, as well as wanting to add a pinch of syntactic sugar of my own in component event code using special comments. Fortunately this turned out to be not that bad.