Sunday, June 11, 2017

Halo Wars ModManifest.txt

I'm semi-back after a short hiatus. During my brief time away from the Halo Wars modding community, a patch to the game was released that added support to load a TXT file that describes other directories which contain game data. This makes it so you don't have to rebuild or even use ERA files for testing or distributing your game changes.

For example, if all you wanted to do was change data/leaders.xml to give you more Spartan population, you could use this new "ModManifest.txt" to point to a folder that contains the said file with your changes. Then when the game loads, it would consume this loose file in favor of whatever file may be in an ERA.

I originally posted the following in the Halo Wars modding Discord chat about a month ago, but I guess no one really paid attention or picked up what was going on:

In the UserRoot folder (the same folder gameconfig.dat exists) you can now add a new file:

C:\Users\<username>\AppData\Local\Halo Wars\ModManifest.txt

The file format is simple, just a directory per line. You can use the semicolon character (;) to essentially comment out a directory if you want to disable it.

Now, the content of each of the directories will override files in the base game’s archive packages.

Directory of E:\temp\TestMod\Art\unsc\vehicle\warthog_01

05/11/2017  12:00 PM    <DIR>          .
05/11/2017  12:00 PM    <DIR>          ..
08/11/2016  03:54 PM         5,592,528 warthog_01_df.ddx

You of course need to replace the "<username>" part of these paths with your account name. The first path is if you use the Windows Store copy of the game. The second path is if you own the Steam copy.

You could place your data/leaders.xml file in this theoretical "E:\temp\TestMod" folder too. Basically you need to remember that any files you wish to override via a directory in the Manifest must maintain the same directory tree as it appears in the ERA files.

Sunday, February 19, 2017

How To: Modding building commands in Halo Wars

So, Halo Wars Definitive Edition is on PC and with it, anyone can easily mod the game. Verses the 360 version which requires a modded Xbox. Yuck.

I was asked on Twitter how one could change what things are trained or researched in a building/socket. So that's the focus today. I may do some other simple guides in the future, if I get the time. However, it will always be time taken away from writing code. Would be nice to instead guide a minion with the drive to explore without full assistance, who would then write/vlog the guides for the everyman/lady.

Setting up

  1. You'll need to own and install Halo Wars Definitive Edition.
  2. You'll need to dump the game package using UWPDumper. Download the lastest x64 build here.
  3. With all the files dumped, you'll need to then uninstall the packed game.
  4. With the game uninstalled, you can put this bat file in the folder with the dumped game files to install the game 'loosely'. If you run into any errors, check that you have side loading enabled in your Windows settings. If you continue to run into an error at this point, try deleting the .p7x file from the dump directory.
  5. At this point you'll need the latest version of the PhxTools from github.
  6. Run PhxGui.exe. It's a simple drag-n-drop tool.
  7. Set the Game Version to DefinitiveEdition
  8. Select the directory for ERA Expand Path, where files will be extracted.
  9. Select the directory for ERA Build Output, where new ERA files will be placed.
  10. Now, drag-n-drop the xgameFinal.exe (one of the dumped game files) into the app.
  11. If the exe version is recognized, a back up will be made and the input will be modified to accept custom ERA files.
  12. From here, you'll need to drag-n-drop root.era, and the contained files will be extracted to the path in step 8.
  13. Go to the output path and open root.eradef file in a text editor that works with XML (you'll need to modify it later)
  14. Now view the "data" folder
  15. You want to find the files "objects.xml.xmb" and "techs.xml.xmb"
  16. Drag-n-drop these files to have PhxGui create the .xml versions of these files.
  17. Now find the files mentioned in step 15 in root.eradef and remove the ".xmb" extension so that the .xml files are referenced instead. Don't forget to save.
  18. Drag-n-drop root.eradef into PhxGui to have it spit out a new root.era to the directory mentioned in step 9.
  19. Assuming you didn't encounter any errors up to this point, you're ready to make changes to the files mentioned in step 15 to change the train/research options of buildings.

Changing build commands

Let's pretend you want to change what is trained out of the UNSC's barracks building. You'll want to open the objects.xml file in a text editor and search for an <Object> entry for unsc_bldg_barracks_01. Inside this element you'll find its <Command> elements. Refer to this excerpt for what you should be seeing.

There are various kinds of commands that a building can respond to. Research, Kill (self destruct), TrainSquad, etc. If you're wanting to mess with these different types, refer to other buildings. For now, let's just replace the command to train unsc_inf_marine_01 with unsc_veh_scorpion_01. These proto squads are defined in the squads.xml file (you'll need to convert the .xmb first), but you don't really need to change that file for command modding.

If the squad you were trying to train was say, a scarab or a unit that you don't normally have access to as the UNSC or leader you're playing with, you'd need to modify the techs.xml data. Specifically, the unsc_basic proto tech. See this doc for the stock unsc_basic data. Refer to it to figure out how to enable other proto squads for all UNSC leaders.

If you repeat step 18, after having saved any changes you made to the XML files, you'll have a root.era with your modifications in it. Back up the root.era that's in your game folder and then drop your custom root.era in.

Assuming there were no errors or mistakes up to this point, the next time you boot the game and play as the UNSC in Skirmish (campaign is a different beast), you should see the tank as the unit option in the barracks where the marines would normally be. It will still require the same amount of tech levels and cost as it normally would. Again, we're just changing the commands used to build units.

The End.

Sure, pictures or a video showing each step would have been nice, but I don't have the time to curate such a guide right now. So make like an explorer, and figure this brace new world out on your own and with others. Claim no land as your own, only territory for you and your fellow modder to brave share. No, I'm not communist, I just prefer to put the "kind" in mankind.

Sunday, December 25, 2016

Wow, 15 years of Halo

Wow. Just realized that it's been 15 years now since I first got an Xbox and with it, Halo. I think it was bundled with a few other games too (Munch's Oddysee, an NFL game, Project Gotham Racing, and DoA3 I think), but I'm focusing on Halo today.

I think I actually got the console+games bundle before Xmas, but still in the month of December. I wouldn't have been able to really play it until winter break started anyway. I remember looking at the box thinking "hmmm, this looks cool...I wonder if it will be anything like Jet Force Gemini? I loved that game"

GoldenEye and Jet Force were the bread and butter of my N64 experience. Around that same time I would have probably been playing Driver 2. Earlier in 2001 I had moved from a different state and with that event I branched out to more single player games since I didn't have my old friends to play co-op with in legends like GoldenEye.

Anyway, I was looking at Halo's DVD case thinking "hmmm...those things [Grunts] kinda resemble the Ants in Jet Force. Why am I not playing yet??". So I finished setting up the Xbox and inserted the game. Splash screens roll by, main menus are briefly hit, and finally, a new campaign is started.

With the game's first cutscene I pretty quickly forgot the "is this like Jet Force" question/mindset. I was pretty amazed at what I was seeing, given that everything else I had played before was on an N64 or PS1. While I did play PC games too, the last interesting title I had played before then was a Star Wars Sith Lords game or Thief Demo from an EGM disc.

However, Halo's cinematics were just amazing and looked pretty. It had all my attention. I played through the first mission. More! Played through the second mission. Moar! Played through the third- "wait...are there no boss fights?". I realized that in the back of my mind I still had some hopes/expectations of what I experienced in Jet Force to appear in this game. One of those things being boss fights. The realization, for a moment, felt weird: "No boss fights?". But that quickly faded. "No boss fights!" It was back to shooting what I now knew as Grunts and Elites.

Now, if you're reading this you've more than likely played the first Halo. After all, it's been released and re-released four times now on four different platforms. So unless you are some PlayStation exclusive nerf herder, you probably already know that midway through the game a new foe is introduced: The Flood. At the time of course, none of us knew or anticipated this reveal.

Let me first paint a better picture of the environment I was playing this game in: I was in North East America, it was winter, cold (probably snowy), middle of the night, and I was in the basement of the house I lived in (practically another floor, since it was carpeted, mini bar, and with a sliding glass door to the the outside). Oh and the house was basically built into the side of a hill with thick woods. In a town where your neighbors were a hike away.

So yeah, I was already feeling quite isolated and susceptible to things too spooky. Then the game goes "here's The Flood, have fun!". My adrenaline was flowing. It was awesome. It probably helped that I was still middle school age too. No game or movie really hit me to such great affect after that. Well, expect maybe Signs. Only because I had scarred myself to the concept of "little green men" back in the 90s with the abduction/UFO shows that aired on TLC (you know...back when it didn't just air reality garbage) and Discovery Channel, plus due to some other events. I want to believe.

I'm fairly certain I finished the game in one sitting/night. While I had finished the game, I was by no means finished with it. I replayed it over and over. I would go to a friend's house and use GameSpy to tunnel mulitplayer matches. I discovered warthog jumping (warthog jumping revisited also needs mentioning).

It wasn't until I was in high school that the Action Replay came out for Xbox. Back in my N64 days I was a big GameShark user. I'd go online, find codes for games, then randomly start changing parts of the codes in hopes of getting something even cooler/different. I seem to recall the GameShark also coming with a VHS that explained basic hexadecimal, among other things. Sadly, I soon realized that the Action Replay was only a device for transferring game saves to/from the console and PC. I couldn't use it to modify a game's memory on the Xbox. But that wasn't the end of that.

Not wanting to let the investment go to waste, I used the device to transfer a Halo game save to the PC and opened it in none other than Notepad. At first anyway. The game used binary game saves, literally memcpy'ing the game state memory (always allocated at a fixed address) and flushing to disc. Before I would venture on to use Hex Workshop to view the bytes of the file, I scrolled what all I could read of the game save in Notepad. To my surprise, I saw help strings relating to the game's scripting engine (which was LISP inspired at the time, these days Halo and Destiny use Lua). This was due to how the engine "freed" datum in the game state memory. It would fill the datum blocks with random bytes from the executable itself. I didn't realize this was the reason until many years later when working on Halo Custom Edition.

All of this would eventually lead me to the Halo modding community and many, many other reverse engineering hijinks that would require their own post. It's kinda incredible to think that it's been 15 years now, longer than I had been alive by the time I first played Halo.

Incredible enough to make me take the time and write this blog post anyway. Perhaps I'll try firing up the Master Chief Collection and try playing Halo 1 again, this Xmas day? I say 'try' twice, because I'm uncertain if that game will function properly. It's too bad no one will do a technical post-mortem on that uber game. I'll probably get two or three missions finished before the nostalgia wears off and I realize I have better things to do with my time these days anyway :)

Thursday, April 14, 2016

One simple change removed heap allocations from our generic IO extensions

The more I use generics, the more I miss C++ templates (we'll just ignore the horrendous compile errors we can get with them when not using Clang).

While I was profiling our Unity project the other day I was wondering why, exactly, parts of our serialization extensions were popping up in our per-frame heap allocations report. While we don't do much serialization ops in client code, the server nodes do, so I felt it worthwhile to investigate.

I normally only profile in a non-development "release" build (although, I'm not certain Unity compiles assemblies with the /optimze+ flag). So by default I'm not seeing file/line info for everything since Unity doesn't give the option to force include the .mdb files (which our profiler tries to use when tracing allocations and other bits).

After making another build, this time with "Development Build" checked and after modifying the .pdb path in Unity's prebaked executable, I was up and running our memory profiled builds with full file/line info for managed and unmanaged code. The reports were showing that at line 330 of our extensions file was to blame for the curious heap allocations.

But how can I easily fix this near the end of a milestone and without going and changing a bunch of callsites? You can't specialize generic code based on the constraints alone, so I couldn't have two Read<T> methods which only differed from "where T:class/struct". I really didn't want to go and make, plus reeducate others to use, a ReadValue and ReadReference method when our Write<T> extensions don't require such verbose code. A few other suggestions were thrown at me, but I wasn't biting.

But wait...what about default arguments? What if I instead added a parameter based on T that uses its default value? This would allow me to avoid having to update a huge number of callsites, but what about still having to allocate reference objects? Turns out checking null is enough for them, while value types don't compare equally to null, so we can avoid the obj = new T() statement altogether for value types and use default arguments to avoid any other code fixup!

Yeah, I hate it when people post code in picture form too, so I put up the bits on gist.github too.

There can be a hidden cost here, but I factor anything GC related to be the biggest, nondeterministic, hidden cost. Specifically, the larger the T value type is, the more code can get generated to deal with copy-by-value semantics for the stack and return value. Although it is possible the runtime or native compiler could work some magic under the hood to use a reference to a stack value. It all depends! I'm not positive what Unity's Mono (IL2CPP would be easier to figure out) is doing under the hood for JIT'd code, but I do know we're no longer generating unexpected temporary garbage in our serialization code!

Tuesday, April 12, 2016

List.Reverse calls Array.Reverse, which may or may not box

You'd expect List<T>.Reverse to not box (ie, allocate memory on the managed heap) when you're using structs, right? Well, you would be wrong. And you'd expect List<T>.Reverse to not box when using basic builtin types like short, byte, right? Sometimes, you may be right.

This is the Array.Reverse implementation in Unity's Mono (some branch of 2.x). It won't box values when the array (List<T> uses an array internally) is an int[] or double[], but will on everything else (see line 1261).

This is the Array.Reverse implementation in the latest version of Mono. It won't box arrays of builtin types, but your enums and custom structs will still result in boxed values.

This is the Array.Reverse implementation in MS.NET's reference source. At the time of this writing, it will try to call the externally defined TrySZReverse function before going on to just box all teh things. Apparently, according to this API issue, that function is a fast-path for reversing primitive types (but your structs, and I'd imagine your enums, don't fall into that category). That API issue is up for review, so it could be that MS.NET will have sane generic Reverse'ing in the future with no boxing!

So yes, you'd expect that your .NET runtimes wouldn't box when performing List<T>.Reverse or Array.Reverse<T> when T is a value type, but you'd mostly be wrong most of the time.

Leave the boxing to Mike Tyson and Rocky!

Be mindful of Dictionary's keyed with enums. Also, the power of per-frame GC analysis!

If you’re going to use a Dictionary with enums in Unity, or any value types rather*, ensure you provide a custom (hopefully static) IEqualityComparer instance for it to use, instead of letting it use the default implementation. I’m not %100 sure about IL2CPP, but in Mono the default comparer (when used with value types) causes a separate heap allocation every time you do ContainsKey or use the Dictionary indexer (you should also probably ask yourself why you’re using ContainsKey/get_Item, instead of TryGetValue, too). The size of each BoehmGC heap allocation, not factoring for things like alignment, is (sizeof(void*) * 2) + sizeof(enum-underlying-type), which in our case was a byte so on 32-bit machines each allocation was 9 bytes.
* You have no choice when it comes to AOT environments like iOS when it comes to structs as keys
At my day job on an unannounced project, we had a particular enum to describe a set of hard coded things (let's call the enum HardCodedThing). Then a certain set of our source game data (let's call the class ProtoSomething) had two dictionaries that were keyed by this enum for various reasons. In our game's sim we have a method, we'll call it UpdateSomethings(), which does a lot of HardCodedThing indexing. The code is ran when a sim is deserialized or when some other state is derived from the sim.

For the past week or so, I've been adding per-frame memory analysis to the custom memory tools that I worked with Rich Geldreich to implement. I marked two custom events (strings which the game runtime sends to the memory profiler at specific points for later reference) as the start and end frames (or rather, the frames which the events took place in). I was interested, well worried, as to why we had so many GC allocations in general when just idling in the game. It turns out our sanity checks for Animator components having parameters before trying to set them was generating garbage (detailed later). It also just so happened that the sim state deriving I mentioned earlier was taking place shortly before the marked last custom event, so heap allocations for the HardCodedThing enum were bubbling up.

The fix to avoid all these allocations was to simply add a Comparer that is then passed in to our Dictionary<HardCodedThing, TValue> instances. The implementation, using the disguised typenames used in this article, can be found here.

Adding the per-frame analysis utilities to our tool has been EXTREMELY helpful (I found some other GC-heavy APIs specific to Unity that I'm sure we're not the only ones ignorant of...should blog about them soon). We're not stuck scratching our heads by what's making the stock Unity Profiler show "GC Allocations per Frame: 9999 / 12 KB" or some general nonsense. We really don't even bother using Unity's Memory profiler. Reason being is that it's just a snapshot. Snapshots have limited use. It's also far easier and quicker to make a Standalone build vs a device build, since their tool is IL2CPP only. We instead have a transaction log of all Boehm/Mono memory related operations from the start of the process until the very end. Coupled with our custom events setup we can create near-deterministic transaction logs over multiple runs. In theory, we could probably even setup a BVT to ensure we're not leaking managed memory or churning GC memory more than we already expect to.

And when/if we do, we have a boat load of information at our disposal to diagnose problems. Part of the per-frame analysis was to first spit out a CSV file mapping types seen across a set of frames to how many allocations and frees they were associated with, along with their average garbage (if alloc:free is near 1:1, you have garbage heavy code using that type somewhere!). You can find an excerpt of the CSV report here. I mentioned that we had code sanity checking Animators for parameters we are trying to set earlier, and here you can see a glimpse of how they were showing up in our reports.

With this broad report, the next thing I added was a 'blame list' report. The blame list details, per type, the sets of backtraces which are spawning these allocations. We then sort and break down these backtraces by their allocations, so the biggest offenders of garbage are at the top. You can find an excerpt of the blame list report here.

Perf matters. Memory usage matters. Tools matter. While the studio I work at probably won't open source our memory tool's full pipeline anytime soon for various reasons (although our Mono modifications can be found here), I'm hoping to use this blog to publicly speak more about bad/good patterns we see/learn from experience and verified using the tools we've developed.

Sunday, February 21, 2016

Exposing the Unity3D profiler window settings

Lately, I've been working on writing reverse engineering the internals of Unity's Profiler APIs and Window in hopes of doing my own viewer. This is part of the reason for the lack of new IL2CPP articles (although I do have some content I really want to cover, especially regarding virtual calls). Anyway, I want my own profile viewer.

A viewer that isn't limited to IMGUI patterns.

A viewer that isn't slow as hell (to the point of being unusable) when expanding CPU hierarchy nodes that are so many levels deep, especially in deep profiling (go jump off a bridge if you really think I have time to manage manual Sample traces, especially in code which is Unity agnostic, and the iteration times involved).

A viewer that can work on custom offline trace files. Which, maybe could provide ETL steps to spit out a .DTP version that is compatible with dotTrace. OK, so their formats don't appear to be open or documented, but you get the idea. One could even hack together a tool based off this Windows Heap profiler. Anything is possible when you have the raw data!

A viewer that is open source. Because one size does not fit all. If it did, I wouldn't be writing this blog post or reverse engineering the profiler!

Anyway, good progress is being made. I managed to get many of my LINQ utils working in the .NET framework used in Unity. Even uncovered some use cases that I didn't solve already in the process. It also helps that not all of the types used in the ProfilerWindow/APIs are marked internal. I expect to have some raw tools ready by early next month (probably in time for another round of profiling work at my day job).

But in the meantime, I'm serving appetizers!

Customizing the Profiler Window view

Yesterday, I happened to stumble across this programmer's tweet where he uses the PreferenceItem attribute to add custom Edit->Preferences options. Pretty cool! I then remembered how earlier I found (via ILSpy) that many of the Profiler Window's column's visibilities can be tuned by some EditorPrefs settings. Not long after, I was able to get the following settings working:

I can finally tune just how much BS is being displayed in the Profiler! Kind of a big deal, since I'm pretty sure part of the blame for slow down in the Profiler CPU Hierarchy is due to all the information it has to format into std::strings, which are then marshaled to C#, then processed in the window's IMGUI drawing code.

You can find the code for the editor here. You of course use it at your own risk. I will not be responsible for any strange states you may get your Profiler window into using it. On Windows, you could manually undo any craziness by regedit'ing the raw EditorPrefs key/values (just delete them).

It should be noted, however, that you can't dynamically add/remove columns with the Profiler window open. Before opening it, you must set your desired columns.

Also, a note about the code: I have a USE_SYSTEM_BITVECTOR macro. For public consumption, you'll want to keep this defined as it will cause the code to use the .NET BitVector32, instead of mine (which can do much more).

There's also some incomplete Chart prefs editors, in case anyone is that interested (I'm not, but did as much for completeness' sake).

Hope this helps others manage their...managed code profiling!