Developer Chat: Detaching Our Engine from Unity

From late 2014 to mid-2017 I worked at mobile game studio called Boss Fight Entertainment. While I was there I ended up writing a developer blog piece (in the hopes of making it a common occurrence). I tried to find that blog piece today, but couldn't. I managed to find an old link to it, but turns out the page is no longer a thing. Seems they updated their site and ended up losing all the previous blog entries. Thankfully, Wayback Machine has been on the march for years now and I was able to find the article on there. In the interest of archiving, I figured I would also re-post it here.

Note: I literally copy/pasted from the Wayback Machine, so all the embedded links are to archived pages.

From August 28th, 2015:

Developer Chat: Detaching Our Engine from Unity

Detaching Our Engine from Unity

Unity is all that and a bag of potato chips – until it isn’t. When you’re doing something technically interesting or complex, especially on a non-PC platform, you can quickly hit the point where the bag pops and the chips turn to crumbs.

Background

At Boss Fight, we currently use Unity3D for all our active projects. The projects are primarily mobile based and require various server related endpoints (simulating game state, collecting stats, etc). The way we do this today is via a set of non-Unity dependent C# code that can be compiled with newer versions of Mono or .NET proper. Effectively this is our Core + GameSim framework. Unity only ends up being our presentation (UX) framework. This is a fairly large task that we off load and the Unity tech holds quite nicely overall (e.g., rarely do we deal with mundane cross platform issues at a game level). The main point of this Core + GameSim architecture is that it allows us to share and run the same exact code on the client (device) and server (which is powered by Mono 3.x on Linux machines). This also permits us to easily execute and profile our non-Unity dependent code using other .NET runtimes, such as with Microsoft’s .NET framework.

Prior to Boss Fight Entertainment, the team worked on some very successful games. The more recent release, CastleVille, was implemented a little differently that our tech today. It still required server functionality to operate, but the clients were web based. Unlike our architecture today, the client was developed in Flash while the server was done in PHP. In order for most of the game logic to be shared it ended up getting programmed in a custom language called zdscript. However, all systems outside of the game logic couldn’t be shared, including the interpreter for said script. The many headaches and bugs this pattern generated led to the ‘pure-C# game code’ mentality that exists at our company today.

Unintentional Benefits

Boss Fight is getting very close to the world wide launch of our first game, Dungeon Boss. With that, we’ve been taking a lot of time to fine tune the engine and some of the tools in critical points of our pipeline. One of the critical components that exists in both is our “BFSerialization” framework, which we use instead of Unity’s built-in serialization API. With the BFSerialization framework we’re able to convert any object graph to and from binary for storage and communication purposes. This framework is used in both our server and client code, so it must be very fast and memory efficient.

There are other serialization solutions out there, but none of them neatly fit our problem space. For example, .NET’s serialization framework relies too much on reflection and embeds very strict type information into the output stream. With BFSerialization, we’re able to offload the majority of reflection-based computations through custom code generation. As input to the generator we interpret C# attributes on our serializable types in a project build step.

Our streams of serialized objects are constantly being transmitted to and from the server. We also use this framework for storing our game-defining data, or “proto-data”. Our engine is highly data driven, so we have a lot of this proto-data. Thus the performance of loading and saving objects is of a high concern. The slower our serialization is the longer the client takes to load and the more servers we have to pay for.

Even before I was tasked with looking into our serializer’s performance we knew it was sub-optimal. However, it wasn’t really made apparent until some big, new features were integrated close to our Soft Launch. While one part of the team was celebrating the fact that we had reached a new peak on our way up World Wide Launch Mountain, another part saw we were only a few steps away from a gaping chasm between us and the next slope. Specifically, they were noticing a fairly large increase in transaction times.

Everything occurs as a transaction between the client and game servers, evolving a character, receiving loot, etc. Some of these new features introduced new types and increased the number of transactions. So the original author of our BFSerializer did some preliminary scouting of this chasm in our metaphorical mountain and confirmed a sneaking suspicion: the time to deserialize game states and other payloads for processing was consuming far too much time.

Having recently helped on other lower level memory issues in our engine earlier in the year (more on that later), I was brought in to help us cross this obstacle and keep us on track to the next peak. So when it came time to profile this code I had a handful of options and tools in my pack: Unity’s profiler, building Mono with profiler support enabled, or get the code running in .NET where there’s a sea of mature tools.

Unity: The profiler in Unity can get the job done for many tasks, but it provides no means of comparing multiple profiler sessions, is frame based and will drop results after recording for so long, and doesn’t have very granular control over what stats are on or off in recordings.

Intensive profiling in general can involve trying many code configurations, but unfortunately Unity’s out of the box build system right now isn’t as versatile as in Visual Studio or Mono Develop.

• The automatic recompiling of project code whenever Unity regains focus can be annoying.
• There’s no simple means of setting your current build configuration (debug, checked, release, playtest, etc).
• You’re limited to using prebuilt DLLs or sticking code in one of the four “Assembly-CSharp” projects it automatically generates. Unity creates .meta files for everything, including the .cs files and the folders they reside in. I can imagine a step towards configuring code that exists in its own assemblies via overridable settings saved in these .meta files.

Mono: Mono has some profiler support, but doesn’t ship with it. (At least, the Windows distro for 3.x doesn’t). In order to get profiling to work, I would need to build Mono from source, and then use the mprof-report tool for raw analysis of the profiler’s output .mlpd files. There’s also Xamarin Profiler, which consumes said .mlpd files as well. Since I didn’t end up taking this route for this task I can’t offer any feedback on it.

.NET: Finally, there’s Microsoft’s .NET proper and a long list of commercial 3rd party tools to choose from for profiling: JustTrace, Dynatrace, Red-Gate’s ANTS, and dotTrace to name a few.

I chose to use dotTrace by JetBrains, as I had prior experience with it. dotTrace offers a matrix of tracing options, from typical sampling of call time, to line-by-line stats, or even a timeline capture. Whereas Unity’s profiler will collect all information (self/total time of a method, how many times it was executed, etc), dotTrace’s configuration options let you can chose to not track the number of times a method is called (ie, sampling vs. tracing). This setup lowers the skewing of run times in the results (as the profiling overhead ends up bleeding in).

Now, I know what you may be thinking: “you’re profiling code in .NET that is ran on Mono in production?” Yes, the .NET route was mainly to get an initial sense of where to begin improvements. There are obviously going to be platform and API differences in the context of performance. I figured if we needed to go that low level on performance that I’d eventually get a build of Mono running, but going with .NET tools felt like the path of least resistance.

The logs generated by dotTrace were extremely helpful in weeding out places where we were calling APIs in an inefficient manner. There was one case where we were going over an unsorted list of objects and processing their members if any have a certain attribute applied. The member info was reflected from the object each time. And as it turned out, most objects were of types that didn’t have the attribute! So instead what I ended up doing was sorting objects by their type. From there I could build the needed member info before even touching the objects. If we were really desperate we probably could have had the code generator build some static info regarding what types actually need this processing. However, these days we don’t try to throw too much into codegen.

You’re only as good as the tools you keep company. Had we invested all our money in Unity for climbing this mountain we would have had far fewer options in how we went about getting across this chasm (i.e., profiling our non-performant C# code). While the decision to leave Unity ties out of our Core + GameSim code didn’t stem from this problem, the choice was certainly a blessing in disguise. We’re at a point on our project’s mountain were we look down and see the tiny hole that once was a chasm representing a time suck in a very critical path in our game. If that time suck and its impact were weighed as a backpack full of extra gear to get up this mountain, the changes and improvements we made via these smarter tools have us free climbing instead. Or in plain speak: we’re experiencing >50% less overhead now with the improvements.

Dodging another hole in the Earth

As I was hinting near the start of this article, earlier this year we had no choice but to create our own custom memory tools for tracking “leaks” in the managed heap. Unity uses a very old version of Mono, which also uses the Boehm garbage collector (aka libgc). While they’re starting to move non-Editor builds to their IL2CPP technology, they still use the Boehm GC (albeit, a new revision). Said GC was originally designed for C programs and has no sane debugging or tracing functionality. It was written in the early 90s, so I guess there’s a considerable amount of legacy tech that may rely upon it which deters others from modernizing it. Getting around this specific hole was like entering a Survivorman mode and just taking everything around you to survive and get out of a mess in the wilderness. We made it back to civilization, but not without our fair share of scratches and bruises.

Conclusion

Hopefully you, the reader, can take some of this experience and consider it when working on your next game that’s based in Unity, even if it’s not strictly mobile. Unity as-is has a lot of fancy machinery, albeit some of it currently falls sort of its potential or can shoot you in the foot if you’re not careful. You even have to be mindful of some of the higher level C#-idioms you use in your Unity games (we’ll have to save more details for another blog post for now).

Halo Wars ModManifest.txt

I'm semi-back after a short hiatus. During my brief time away from the Halo Wars modding community, a patch to the game was released that added support to load a TXT file that describes other directories which contain game data. This makes it so you don't have to rebuild or even use ERA files for testing or distributing your game changes.

For example, if all you wanted to do was change data/leaders.xml to give you more Spartan population, you could use this new "ModManifest.txt" to point to a folder that contains the said file with your changes. Then when the game loads, it would consume this loose file in favor of whatever file may be in an ERA.

I originally posted the following in the Halo Wars modding Discord chat about a month ago, but I guess no one really paid attention or picked up what was going on:

In the UserRoot folder (the same folder gameconfig.dat exists) you can now add a new file:
“ModManifest.txt”

[Example]
C:\Users\<username>\AppData\Local\Packages\Microsoft.BulldogThreshold_8wekyb3d8bbwe\LocalState\ModManifest.txt
C:\Users\<username>\AppData\Local\Halo Wars\ModManifest.txt

The file format is simple, just a directory per line. You can use the semicolon character (;) to essentially comment out a directory if you want to disable it.
[Example]
e:\temp\TestMod

Now, the content of each of the directories will override files in the base game’s archive packages.

[Example]
Directory of E:\temp\TestMod\Art\unsc\vehicle\warthog_01

05/11/2017  12:00 PM    <DIR>          .
05/11/2017  12:00 PM    <DIR>          ..
08/11/2016  03:54 PM         5,592,528 warthog_01_df.ddx

You of course need to replace the "<username>" part of these paths with your account name. The first path is if you use the Windows Store copy of the game. The second path is if you own the Steam copy.

You could place your data/leaders.xml file in this theoretical "E:\temp\TestMod" folder too. Basically you need to remember that any files you wish to override via a directory in the Manifest must maintain the same directory tree as it appears in the ERA files.

How To: Modding building commands in Halo Wars

So, Halo Wars Definitive Edition is on PC and with it, anyone can easily mod the game. Verses the 360 version which requires a modded Xbox. Yuck.

I was asked on Twitter how one could change what things are trained or researched in a building/socket. So that's the focus today. I may do some other simple guides in the future, if I get the time. However, it will always be time taken away from writing code. Would be nice to instead guide a minion with the drive to explore without full assistance, who would then write/vlog the guides for the everyman/lady.

Setting up

1. You'll need to own and install Halo Wars Definitive Edition.
2. You'll need to dump the game package using UWPDumper. Download the lastest x64 build here.
3. With all the files dumped, you'll need to then uninstall the packed game.
4. With the game uninstalled, you can put this bat file in the folder with the dumped game files to install the game 'loosely'. If you run into any errors, check that you have side loading enabled in your Windows settings. If you continue to run into an error at this point, try deleting the .p7x file from the dump directory.
5. At this point you'll need the latest version of the PhxTools from github.
6. Run PhxGui.exe. It's a simple drag-n-drop tool.
7. Set the Game Version to DefinitiveEdition
8. Select the directory for ERA Expand Path, where files will be extracted.
9. Select the directory for ERA Build Output, where new ERA files will be placed.
10. Now, drag-n-drop the xgameFinal.exe (one of the dumped game files) into the app.
11. If the exe version is recognized, a back up will be made and the input will be modified to accept custom ERA files.
12. From here, you'll need to drag-n-drop root.era, and the contained files will be extracted to the path in step 8.
13. Go to the output path and open root.eradef file in a text editor that works with XML (you'll need to modify it later)
14. Now view the "data" folder
15. You want to find the files "objects.xml.xmb" and "techs.xml.xmb"
16. Drag-n-drop these files to have PhxGui create the .xml versions of these files.
17. Now find the files mentioned in step 15 in root.eradef and remove the ".xmb" extension so that the .xml files are referenced instead. Don't forget to save.
18. Drag-n-drop root.eradef into PhxGui to have it spit out a new root.era to the directory mentioned in step 9.
19. Assuming you didn't encounter any errors up to this point, you're ready to make changes to the files mentioned in step 15 to change the train/research options of buildings.

Changing build commands

Let's pretend you want to change what is trained out of the UNSC's barracks building. You'll want to open the objects.xml file in a text editor and search for an <Object> entry for unsc_bldg_barracks_01. Inside this element you'll find its <Command> elements. Refer to this excerpt for what you should be seeing.

There are various kinds of commands that a building can respond to. Research, Kill (self destruct), TrainSquad, etc. If you're wanting to mess with these different types, refer to other buildings. For now, let's just replace the command to train unsc_inf_marine_01 with unsc_veh_scorpion_01. These proto squads are defined in the squads.xml file (you'll need to convert the .xmb first), but you don't really need to change that file for command modding.

If the squad you were trying to train was say, a scarab or a unit that you don't normally have access to as the UNSC or leader you're playing with, you'd need to modify the techs.xml data. Specifically, the unsc_basic proto tech. See this doc for the stock unsc_basic data. Refer to it to figure out how to enable other proto squads for all UNSC leaders.

If you repeat step 18, after having saved any changes you made to the XML files, you'll have a root.era with your modifications in it. Back up the root.era that's in your game folder and then drop your custom root.era in.

Assuming there were no errors or mistakes up to this point, the next time you boot the game and play as the UNSC in Skirmish (campaign is a different beast), you should see the tank as the unit option in the barracks where the marines would normally be. It will still require the same amount of tech levels and cost as it normally would. Again, we're just changing the commands used to build units.

The End.

Sure, pictures or a video showing each step would have been nice, but I don't have the time to curate such a guide right now. So make like an explorer, and figure this brace new world out on your own and with others. Claim no land as your own, only territory for you and your fellow modder to brave share. No, I'm not communist, I just prefer to put the "kind" in mankind.

Wow, 15 years of Halo

Wow. Just realized that it's been 15 years now since I first got an Xbox and with it, Halo. I think it was bundled with a few other games too (Munch's Oddysee, an NFL game, Project Gotham Racing, and DoA3 I think), but I'm focusing on Halo today.

I think I actually got the console+games bundle before Xmas, but still in the month of December. I wouldn't have been able to really play it until winter break started anyway. I remember looking at the box thinking "hmmm, this looks cool...I wonder if it will be anything like Jet Force Gemini? I loved that game"

GoldenEye and Jet Force were the bread and butter of my N64 experience. Around that same time I would have probably been playing Driver 2. Earlier in 2001 I had moved from a different state and with that event I branched out to more single player games since I didn't have my old friends to play co-op with in legends like GoldenEye.

Anyway, I was looking at Halo's DVD case thinking "hmmm...those things [Grunts] kinda resemble the Ants in Jet Force. Why am I not playing yet??". So I finished setting up the Xbox and inserted the game. Splash screens roll by, main menus are briefly hit, and finally, a new campaign is started.

With the game's first cutscene I pretty quickly forgot the "is this like Jet Force" question/mindset. I was pretty amazed at what I was seeing, given that everything else I had played before was on an N64 or PS1. While I did play PC games too, the last interesting title I had played before then was a Star Wars Sith Lords game or Thief Demo from an EGM disc.

However, Halo's cinematics were just amazing and looked pretty. It had all my attention. I played through the first mission. More! Played through the second mission. Moar! Played through the third- "wait...are there no boss fights?". I realized that in the back of my mind I still had some hopes/expectations of what I experienced in Jet Force to appear in this game. One of those things being boss fights. The realization, for a moment, felt weird: "No boss fights?". But that quickly faded. "No boss fights!" It was back to shooting what I now knew as Grunts and Elites.

Now, if you're reading this you've more than likely played the first Halo. After all, it's been released and re-released four times now on four different platforms. So unless you are some PlayStation exclusive nerf herder, you probably already know that midway through the game a new foe is introduced: The Flood. At the time of course, none of us knew or anticipated this reveal.

Let me first paint a better picture of the environment I was playing this game in: I was in North East America, it was winter, cold (probably snowy), middle of the night, and I was in the basement of the house I lived in (practically another floor, since it was carpeted, mini bar, and with a sliding glass door to the the outside). Oh and the house was basically built into the side of a hill with thick woods. In a town where your neighbors were a hike away.

So yeah, I was already feeling quite isolated and susceptible to things too spooky. Then the game goes "here's The Flood, have fun!". My adrenaline was flowing. It was awesome. It probably helped that I was still middle school age too. No game or movie really hit me to such great affect after that. Well, expect maybe Signs. Only because I had scarred myself to the concept of "little green men" back in the 90s with the abduction/UFO shows that aired on TLC (you know...back when it didn't just air reality garbage) and Discovery Channel, plus due to some other events. I want to believe.

I'm fairly certain I finished the game in one sitting/night. While I had finished the game, I was by no means finished with it. I replayed it over and over. I would go to a friend's house and use GameSpy to tunnel mulitplayer matches. I discovered warthog jumping (warthog jumping revisited also needs mentioning).

It wasn't until I was in high school that the Action Replay came out for Xbox. Back in my N64 days I was a big GameShark user. I'd go online, find codes for games, then randomly start changing parts of the codes in hopes of getting something even cooler/different. I seem to recall the GameShark also coming with a VHS that explained basic hexadecimal, among other things. Sadly, I soon realized that the Action Replay was only a device for transferring game saves to/from the console and PC. I couldn't use it to modify a game's memory on the Xbox. But that wasn't the end of that.

Not wanting to let the investment go to waste, I used the device to transfer a Halo game save to the PC and opened it in none other than Notepad. At first anyway. The game used binary game saves, literally memcpy'ing the game state memory (always allocated at a fixed address) and flushing to disc. Before I would venture on to use Hex Workshop to view the bytes of the file, I scrolled what all I could read of the game save in Notepad. To my surprise, I saw help strings relating to the game's scripting engine (which was LISP inspired at the time, these days Halo and Destiny use Lua). This was due to how the engine "freed" datum in the game state memory. It would fill the datum blocks with random bytes from the executable itself. I didn't realize this was the reason until many years later when working on Halo Custom Edition.

All of this would eventually lead me to the Halo modding community and many, many other reverse engineering hijinks that would require their own post. It's kinda incredible to think that it's been 15 years now, longer than I had been alive by the time I first played Halo.

Incredible enough to make me take the time and write this blog post anyway. Perhaps I'll try firing up the Master Chief Collection and try playing Halo 1 again, this Xmas day? I say 'try' twice, because I'm uncertain if that game will function properly. It's too bad no one will do a technical post-mortem on that uber game. I'll probably get two or three missions finished before the nostalgia wears off and I realize I have better things to do with my time these days anyway :)

One simple change removed heap allocations from our generic IO extensions

The more I use generics, the more I miss C++ templates (we'll just ignore the horrendous compile errors we can get with them when not using Clang).

While I was profiling our Unity project the other day I was wondering why, exactly, parts of our serialization extensions were popping up in our per-frame heap allocations report. While we don't do much serialization ops in client code, the server nodes do, so I felt it worthwhile to investigate.

I normally only profile in a non-development "release" build (although, I'm not certain Unity compiles assemblies with the /optimze+ flag). So by default I'm not seeing file/line info for everything since Unity doesn't give the option to force include the .mdb files (which our profiler tries to use when tracing allocations and other bits).

After making another build, this time with "Development Build" checked and after modifying the .pdb path in Unity's prebaked executable, I was up and running our memory profiled builds with full file/line info for managed and unmanaged code. The reports were showing that at line 330 of our extensions file was to blame for the curious heap allocations.

But how can I easily fix this near the end of a milestone and without going and changing a bunch of callsites? You can't specialize generic code based on the constraints alone, so I couldn't have two Read<T> methods which only differed from "where T:class/struct". I really didn't want to go and make, plus reeducate others to use, a ReadValue and ReadReference method when our Write<T> extensions don't require such verbose code. A few other suggestions were thrown at me, but I wasn't biting.

But wait...what about default arguments? What if I instead added a parameter based on T that uses its default value? This would allow me to avoid having to update a huge number of callsites, but what about still having to allocate reference objects? Turns out checking null is enough for them, while value types don't compare equally to null, so we can avoid the obj = new T() statement altogether for value types and use default arguments to avoid any other code fixup!

Yeah, I hate it when people post code in picture form too, so I put up the bits on gist.github too.

There can be a hidden cost here, but I factor anything GC related to be the biggest, nondeterministic, hidden cost. Specifically, the larger the T value type is, the more code can get generated to deal with copy-by-value semantics for the stack and return value. Although it is possible the runtime or native compiler could work some magic under the hood to use a reference to a stack value. It all depends! I'm not positive what Unity's Mono (IL2CPP would be easier to figure out) is doing under the hood for JIT'd code, but I do know we're no longer generating unexpected temporary garbage in our serialization code!

List.Reverse calls Array.Reverse, which may or may not box

You'd expect List<T>.Reverse to not box (ie, allocate memory on the managed heap) when you're using structs, right? Well, you would be wrong. And you'd expect List<T>.Reverse to not box when using basic builtin types like short, byte, right? Sometimes, you may be right.

This is the Array.Reverse implementation in Unity's Mono (some branch of 2.x). It won't box values when the array (List<T> uses an array internally) is an int[] or double[], but will on everything else (see line 1261).

This is the Array.Reverse implementation in the latest version of Mono. It won't box arrays of builtin types, but your enums and custom structs will still result in boxed values.

This is the Array.Reverse implementation in MS.NET's reference source. At the time of this writing, it will try to call the externally defined TrySZReverse function before going on to just box all teh things. Apparently, according to this API issue, that function is a fast-path for reversing primitive types (but your structs, and I'd imagine your enums, don't fall into that category). That API issue is up for review, so it could be that MS.NET will have sane generic Reverse'ing in the future with no boxing!

So yes, you'd expect that your .NET runtimes wouldn't box when performing List<T>.Reverse or Array.Reverse<T> when T is a value type, but you'd mostly be wrong most of the time.

Leave the boxing to Mike Tyson and Rocky!

Be mindful of Dictionary's keyed with enums. Also, the power of per-frame GC analysis!

If you’re going to use a Dictionary with enums in Unity, or any value types rather*, ensure you provide a custom (hopefully static) IEqualityComparer instance for it to use, instead of letting it use the default implementation. I’m not %100 sure about IL2CPP, but in Mono the default comparer (when used with value types) causes a separate heap allocation every time you do ContainsKey or use the Dictionary indexer (you should also probably ask yourself why you’re using ContainsKey/get_Item, instead of TryGetValue, too). The size of each BoehmGC heap allocation, not factoring for things like alignment, is (sizeof(void*) * 2) + sizeof(enum-underlying-type), which in our case was a byte so on 32-bit machines each allocation was 9 bytes.

* You have no choice when it comes to AOT environments like iOS when it comes to structs as keys

At my day job on an unannounced project, we had a particular enum to describe a set of hard coded things (let's call the enum HardCodedThing). Then a certain set of our source game data (let's call the class ProtoSomething) had two dictionaries that were keyed by this enum for various reasons. In our game's sim we have a method, we'll call it UpdateSomethings(), which does a lot of HardCodedThing indexing. The code is ran when a sim is deserialized or when some other state is derived from the sim.

For the past week or so, I've been adding per-frame memory analysis to the custom memory tools that I worked with Rich Geldreich to implement. I marked two custom events (strings which the game runtime sends to the memory profiler at specific points for later reference) as the start and end frames (or rather, the frames which the events took place in). I was interested, well worried, as to why we had so many GC allocations in general when just idling in the game. It turns out our sanity checks for Animator components having parameters before trying to set them was generating garbage (detailed later). It also just so happened that the sim state deriving I mentioned earlier was taking place shortly before the marked last custom event, so heap allocations for the HardCodedThing enum were bubbling up.

The fix to avoid all these allocations was to simply add a Comparer that is then passed in to our Dictionary<HardCodedThing, TValue> instances. The implementation, using the disguised typenames used in this article, can be found here.

Adding the per-frame analysis utilities to our tool has been EXTREMELY helpful (I found some other GC-heavy APIs specific to Unity that I'm sure we're not the only ones ignorant of...should blog about them soon). We're not stuck scratching our heads by what's making the stock Unity Profiler show "GC Allocations per Frame: 9999 / 12 KB" or some general nonsense. We really don't even bother using Unity's Memory profiler. Reason being is that it's just a snapshot. Snapshots have limited use. It's also far easier and quicker to make a Standalone build vs a device build, since their tool is IL2CPP only. We instead have a transaction log of all Boehm/Mono memory related operations from the start of the process until the very end. Coupled with our custom events setup we can create near-deterministic transaction logs over multiple runs. In theory, we could probably even setup a BVT to ensure we're not leaking managed memory or churning GC memory more than we already expect to.

And when/if we do, we have a boat load of information at our disposal to diagnose problems. Part of the per-frame analysis was to first spit out a CSV file mapping types seen across a set of frames to how many allocations and frees they were associated with, along with their average garbage (if alloc:free is near 1:1, you have garbage heavy code using that type somewhere!). You can find an excerpt of the CSV report here. I mentioned that we had code sanity checking Animators for parameters we are trying to set earlier, and here you can see a glimpse of how they were showing up in our reports.

With this broad report, the next thing I added was a 'blame list' report. The blame list details, per type, the sets of backtraces which are spawning these allocations. We then sort and break down these backtraces by their allocations, so the biggest offenders of garbage are at the top. You can find an excerpt of the blame list report here.

Perf matters. Memory usage matters. Tools matter. While the studio I work at probably won't open source our memory tool's full pipeline anytime soon for various reasons (although our Mono modifications can be found here), I'm hoping to use this blog to publicly speak more about bad/good patterns we see/learn from experience and verified using the tools we've developed.