Why I Used a Custom Game Engine for my Indie Game
Part of a series of devlogs I’m making for the Diorama Break Kickstarter.
Long-time subscribers will remember the series of devlogs I did a couple years ago over the course of building my own custom game engine. A lot of my thinking on the subject is captured in that original post, but now, after over a year of actually using the engine and producing a successful game demo with it, I thought it would be worth revisiting.
Why Use a Custom Engine?
The “common wisdom” around this topic often boils down to this idea that creating a custom engine as a solo dev or small team will necessarily suck up half a decade without any actual game to show for it. While there are a few anecdotal horror stories to this effect, for the most part this is a myth. Use of third-party engines like Unity, Unreal, or Godot was nowhere near a dominant practice until the mid-2010s. Many of the most influential indie games of all time—games like Braid, FTL, and Terraria—were all made with custom engines in just 1 to 3 years. And thanks to the release of new and friendlier low-level languages and frameworks like Odin, Raylib, SDL3, etc. over the past decade and a half, this style of game development is easier and more approachable than ever!1 My own experience involved getting fed up with the limitations of Game Maker, giving myself half a year to replicate all the basic features I liked from it in my own software (and add a few I thought were sorely missing), and managing to get it all done in just about five months!
There are a few caveats of course. It’s not something I would recommend to total beginners; without experience actually using an existing engine you wouldn’t even really know what to build in the first place. Having at least one major project under your belt, ideally one made for commercial release, will give you indispensable insight into what you’ll actually need in terms of tools. 3D is also a much greater challenge compared to 2D. While I feel confident that I could tackle a custom 3D engine now, after my experience building my 2D engine, even at the onset I recognized that trying to do something 3D from the start would have added waaay more complexity and potentially spiraled into something unsustainable. And if you’re angling to compete with high-fidelity AAA graphics, it’s not clear to me that as a solo dev going custom is viable in at all the same way that it is for something 2D or stylized.
But I still haven’t really answered the question “why bother”? If third-party engines developed by all these well-resourced companies and communities are just sitting there for the taking, why put in all this extra work reinventing the wheel? At the core of it, it’s perhaps a sense of pride and self-reliance, wanting to have the agency, deep understanding, and control that building my own tools affords me. But I can also list a lot of practical advantages:
Third-party engines can be extremely bloated, with years of feature creep, abandoned functionality, and “legacy support” taking their toll on usability and the ease of learning the engine. When building your own tools, you can choose exactly what you do and don’t need, and you’ll have a full understanding of what’s available to you once you get to actually building your game.
Most third-party engines are built to be extremely general-purpose, in order to give all users the option to create almost any sort of game. That often means that the tools you’re given won’t be quite suited to your specific needs, which can range from them just being slightly annoying to work with, to requiring enormous additions and workarounds (which are often more work to integrate with existing engine bloat than it would be to just build the whole tool from scratch in your own clean ecosystem). More specialized engines like RPGMaker and Ren’Py exist, but they’re even more susceptible to the next problem…
If a third-party engine doesn’t support a feature or have a tool you need for your game, it’s often a serious problem. Building the tool from scratch and integrating it with the engine can range from extremely time-consuming to outright impossible, especially if the engine is closed-source. With your own engine, you built everything anyway, so one more feature is just that. Much, much easier.
Similar to the above point, if a third-party engine does technically support a feature but it’s broken in some way (either buggy or just unfinished), you’ll often be caught between trying to fix it yourself, hoping and waiting for the engine developers to fix it, or just working around it in a way that makes your game worse; all risky and annoying options. With your own tools, you can of course just fix them yourself as easily as you built them in the first place.
Many third-party engines will encourage the use of tools and scripting languages that offer very limited control of the exact behavior of your final program, especially when it comes to things like memory layout. This might make things easier when starting out, but will eventually spiral into you having performance issues you have no easy way to solve (or even properly diagnose). Certain designs will even be outright impossible to achieve without the ability to finely optimize your game; Noita is a game that simulates every single pixel on screen as a powder-like physics object, something that feels to me like it would be impossible or at least unbelievably onerous to do in an engine you didn’t have complete control over2.
This is more of an issue diffused across the industry, but widespread use of third-party engines, among other things, has caused an observable decrease in “diversity” in modern games. When you’re forced to build everything yourself, you’ll end up making a bunch of tiny idiosyncratic decisions about things like physics, lighting, rendering, etc. that cause your game to feel “unique” in subtle and not-so-subtle ways. When everyone is instead reaching for the same cookie-cutter solutions that engines offer for basic problems, designs and game feel are pushed more in the direction of what is easy to do in these engines, and many games end up with, for example, “Unity smell”. While it’s possible to get around this by carefully replacing and tuning every aspect of an engine, at what point does it not just become less effort to do everything from scratch?
Many (but not all) third-party engines demand fees, either fixed fees for access to exporters for certain platforms, or in the form of a portion of profits from successful enough commercial projects that rely on them. Not entirely unfair of them to want to be paid for their work, but avoiding these fees can be a compelling reason to stick with your own tools if you plan to develop a large commercial game.
How’s it Going?
Again, my old devlog series is probably the best place to go if you want to get more into the nitty-gritty of how I actually built the engine, but here’s how things have been going in the past year and a half for those who have read it:
In general, I’ve probably become even more handmade-pilled. The original engine had a few third-party libraries and tools in the pipeline that were so annoying that I’ve since stripped them out and replaced them with my own solutions3. I’ve definitely gotten the itch to try and replace every third-party library and tool once I have more time to work on the engine, though FMOD and SDL are still worth keeping in my opinion4.
Odin has also continued to be a blast to use, no major complaints there. Especially now that I’ve explored most of the features of the language, I’ve come to really appreciate the general philosophy it has of only trying to give you one great tool per problem you’re likely to encounter. Avoiding choice paralysis and footguns like that is great for my mental bandwidth, and I can always feel confident when I encounter a new problem that, if it can be solved with just Odin, there’ll be a straightforward way to do so (and this is indeed true of most problems I’ve encountered so far). Probably the biggest thing missing in my opinion is some way to automatically generate symbols (such as by reading asset directories), something I have to rely on external “metaprogram” scripts to do, but I definitely get why Bill can’t5, say, just quickly whip up a robust metaprogramming system to cover cases like this.
The biggest addition since I got all the basic engine features done has probably been the custom stage editor for Diorama Break. Having the ability to just whip up any feature I want for this tool has been great for producing something useful and ergonomic, though I do wish other members of the team would take advantage of this more and pester me about desired fixes and features. It’s a bit embarrassing to leave the tool in their hands for a while then try it out and notice a bunch of places where I left things horribly unfinished!! Anyway, there’s a lot I could talk about here, but I’ll leave it for a future devlog.
Overall, I don’t think I’ll ever be inclined to return to a third-party engine. The ease of use and sense of security all this control gives me has been fantastic, and especially now that we’re well over the initial development hump, the return on investment is really starting to show. I was able to implement our entire localization system in just a few days6, something which I’ve heard other developers say took them weeks if not months to do.
I also feel like I’ve grown tremendously as a programmer, maybe not as drastically as after the initial engine development period, but every day brings more improvement to my skills and confidence to just manifest whatever tool I need to knock down the problems in my path7. Sometimes it takes a little longer than expected, much to my producer’s frustration, but the results in Diorama Break’s demo hopefully speak for themselves. There’s still a ton to work on though, so let’s keep that Kickstarter momentum going!
Not to mention having LLMs available for research and personalized tutorials.
And indeed, the Noita team developed a custom engine expressly for the project, even naming it “Falling Everything” in reference to this central technical design goal.
Most notably, the texture packing system for sprites, which is now entirely custom.
Though I do want to upgrade to SDL3. There’s also no getting around using the Steamworks SDK, but that has been a surprisingly pleasant API to work with and I don’t really have any reason to want it gone.
And doesn’t want to.
And this is despite menus not at all being built with localization in mind (whoops)
Even ones you might not expect, like scripts to auto-collate some of the graphics for our Kickstarter page.