A Community discussion forum for Halo Custom Edition, Halo 2 Vista, Portal and Halo Machinima

Home  Search Register  Login Member ListRecent Posts
  
 
»Forums Index »Halo Custom Edition (Bungie/Gearbox) »Halo CE General Discussion »[WIP] Invader

Author Topic: [WIP] Invader (15 messages, Page 1 of 1)
Moderators: Dennis

Kavawuvi
Joined: May 24, 2018

Brrrrrrring Ha!


Posted: May 30, 2019 02:07 AM    Msg. 1 of 15       
Invader is a work-in-progress, cross-platform, open source project that builds cache files. I've been working on this for a while and I figured I'd share my progress.

How close to completion is Invader?
Invader is still fairly incomplete and won't work with many maps, especially maps with scripts. I do not recommend using it outside of testing for now, but it is working with more and more maps as progress is being made. This is the reason why I don't have binary builds of it available just yet.

Why make this when we have Tool?
The biggest drawbacks of tool.exe are that it's unsupported and closed source. We need it to do more than what it did when it came out, but the best we can do is hacking and patching the executable. This is very limiting when it comes to wanting to do more than what is allowed. Considering the MEK is replacing parts of the Halo Editing Kit, it makes sense to replace this enigmatic piece of software.

In layman's terms, tool.exe is a piece of software filled with black voodoo magic, and being software from the 90's, it sometimes requires various demonic rituals in order to function as intended. The MEK is the bane of evil and Invader is the light.

What are some features?
One useful feature of Invader is that it can build cache files in any tag order you want. This means that you can export the tag list of a multiplayer map, rip the map, modify the rip, and then rebuild the map using the same tag order as before. You can then run the map through Refinery's CRC spoofer, allowing you to use that map in place of the original map online. This is great for if you want to add or change content on the stock multiplayer maps while still allowing these maps to be played online.

Invader is also very fast. Here's a benchmark I ran last October.


Will Invader work on a Mac?
While I don't have any means of building Invader for macOS, it will most likely compile just fine considering it compiles perfectly fine for Linux and Windows. If so, this along with the MEK can be a useful step for Mac users being able to make maps without having to resort to Wine or having an installation of Windows. It's sure been long enough, after all.

How do I get Invader?
Currently, there are no binary builds of Invader as it is still too incomplete to start releasing such builds (see reasons above). Instead you must build it from source. You can get it here: https://github.com/Kavawuvi/Invader.

Presently, to build from source, you will need CMake (https://cmake.org/) as well as a C++ compiler. Windows users can use MinGW which you can get from MSYS2 for Windows (https://www.msys2.org/). I've not tested it with MSVC, but it will probably compile with it. Linux users can simply install GCC or Clang through their favorite package manager, and macOS users can get it through Xcode via the command-line tools.


BioGoji1989
Joined: Dec 24, 2017

Professional Idiot


Posted: May 30, 2019 11:53 AM    Msg. 2 of 15       
Quote: --- Original message by: Kavawuvi

In layman's terms, tool.exe is a piece of software filled with black voodoo magic, and being software from the 90's, it sometimes requires various demonic rituals in order to function as intended.


Hey! You leave our demonic rituals alone! We like sacrificing Minecraft and Destiny players to the cmd.exe god!


lolslayer
Joined: Mar 21, 2015

https://www.youtube.com/watch?v=uMHbAKvPJkU


Posted: May 31, 2019 03:54 PM    Msg. 3 of 15       
Sounds nice, are there already limitations of tool that Invader breaks?


Kavawuvi
Joined: May 24, 2018

Brrrrrrring Ha!


Posted: May 31, 2019 04:18 PM    Msg. 4 of 15       
Quote: --- Original message by: lolslayer
Sounds nice, are there already limitations of tool that Invader breaks?

Yes. Here is a short list:

  • Stock tool does not allow maps to be >384 MiB. Invader does not have this limitation and will happily output maps up to 4 GiB, and Halo will read the maps fine.

  • Stock tool does not allow some reflexives/tag blocks to exceed a certain limit such as netgame equipment. Invader does not have restrictions for most of these, and Halo will read the maps fine.

  • Stock tool only allows indexed, external tags in multiplayer maps despite the game not having this restriction. This results in broken bitmaps and sounds as well as English strings when playing an English UI/SP map on a non-English installation. Invader will use indexed tags for all types of maps, allowing the map to work on all languages of the game.

  • Invader does a small amount of deduplication to tag data, allowing you to put a little bit more tag data in the 23 MiB tag space. I'm planning on expanding on this feature later, but this may mean that you can build some maps that cannot possibly be built in tool.exe without removing tags.

  • Invader does not fail to open bitmaps.map, sounds.map, or loc.map if you have Halo open.


Invader also has a nicer, easier-to-read text output than tool.

tool.exe:
Couldn't read map file './toolbeta.map'
the meter definition ui\hud\cyborg shield does not specify a stencil bitmap group.
the meter definition ui\hud\cyborg body does not specify a stencil bitmap group.
WARNING: 1 clusters in structure_bsp levels\a10\a10_space have no background sound or sound environment.
WARNING: command list DUP107~look_at_halo could be ambiguously attached to BSPs #1 and #8
WARNING: command list DUP119~look_at_halo could be ambiguously attached to BSPs #1 and #8
culling uncompressed model vertices...done
culling uncompressed structure bsp vertices...done
culling uncompressed model animation data...done
building predicted resources for structures...done
structure bsp 'levels\a10\a10a' is 8.25M
structure bsp 'levels\a10\a10b' is 10.94M
structure bsp 'levels\a10\a10c' is 8.04M
structure bsp 'levels\a10\a10d' is 7.27M
structure bsp 'levels\a10\a10e' is 10.35M
structure bsp 'levels\a10\a10f' is 11.39M
structure bsp 'levels\a10\a10g' is 11.49M
structure bsp 'levels\a10\a10_space' is 1.90M
structure bsp 'levels\a10\x10hangar' is 3.62M
tag headers and names are 0.30M
streaming model vertex and index buffers...done
streaming tags.....................................done
writing vertex and index buffers...done 8.70M
writing tags...done (3724 tags for 9.79M)
total tag size is 21.28M (1.72M free)
compressing 190.96M...done
successfully built cache file.
Cache pack file bitmaps hits: 606 for 33.23M
Cache pack file bitmaps adds/misses: 438 for 46.79M
Cache pack file sounds hits: 428 for 7.36M
Cache pack file sounds adds/misses: 2514 for 53.64M
Cache pack file loc hits: 52 for 0.11M
Cache pack file loc adds/misses: 13 for 0.01M

Invader:
Scenario name:     a10
Tags: 3724 / 65535 (82.33 MiB)
BSPs: 9 (73.17 MiB)
levels\a10\a10a (8.24 MiB)
levels\a10\a10b (10.93 MiB)
levels\a10\a10c (8.03 MiB)
levels\a10\a10d (7.26 MiB)
levels\a10\a10e (10.34 MiB)
levels\a10\a10f (11.37 MiB)
levels\a10\a10g (11.47 MiB)*
levels\a10\a10_space (1.90 MiB)
levels\a10\x10hangar (3.61 MiB)
Tag space: 21.06 / 23.00 MiB (91.57 %)
Model data: 8.43 MiB
Bitmaps/sounds: 98.85 MiB
Indexed tags: 572
File size: 189.89 MiB
Time: 1015.425 ms


You can also make it not output anything at all, too. Outputting text to console takes a lot more time than it sounds.

That's all I can think of off the top of my head. I've got more plans for it if/when I finish it.

Edited by Kavawuvi on May 31, 2019 at 04:18 PM


PRPatxi
Joined: Oct 30, 2010

Dennis, free me from this suffering


Posted: Jun 1, 2019 02:59 PM    Msg. 5 of 15       
It's like 8 years since I last saw someone using switch statements in their code.


Kavawuvi
Joined: May 24, 2018

Brrrrrrring Ha!


Posted: Jun 4, 2019 01:38 AM    Msg. 6 of 15       
I got AI encounters to not T-pose and actually work. There are still some things to fix (model permutations and some AI behavior issues) and not all AI encounters are fixed, but the campaign is now playable.


Quote: --- Original message by: PRPatxi
It's like 8 years since I last saw someone using switch statements in their code.

I use what I feel makes sense at the time for what I'm writing. Switch statements are pretty useful for control flow.

Edited by Kavawuvi on Jun 4, 2019 at 05:16 AM


lolslayer
Joined: Mar 21, 2015

https://www.youtube.com/watch?v=uMHbAKvPJkU


Posted: Jun 5, 2019 04:57 AM    Msg. 7 of 15       
Great stuff, thanks for making such great tools for Halo <3


OrangeJuice
Joined: Jan 29, 2009

No patience? Then don't teach. So simple.


Posted: Jun 5, 2019 04:37 PM    Msg. 8 of 15       
It was fun seeing the earlier screenshots of it and the interesting ways it messes up halo, now this looks even nicer


Kavawuvi
Joined: May 24, 2018

Brrrrrrring Ha!


Posted: Jun 6, 2019 05:47 PM    Msg. 9 of 15       
Many of the AI encounters do not work properly still. Navigating this BSP tree thing has been quite difficult as it's not that well documented.

However, many multiplayer maps work quite well. It's not perfect, but it's getting pretty close to that and I'm really excited for when I complete this.

I may start releasing beta builds of Invader soon so people who don't want to compile Invader from source can test it.

That said, I don't want my topic locked (No double post rule FTW! /s), so when I start releasing beta builds, I'll only be able to post new builds on this topic whenever someone replies. Otherwise, you'll be able to find releases on the GitHub repository if they come out. I hope that's okay.

Quote: --- Original message by: OrangeJuice
It was fun seeing the earlier screenshots of it and the interesting ways it messes up halo, now this looks even nicer

Making a tool.exe replacement has been a long journey with many ups and downs. One of the main obstacles to doing this is the fact that there is so much stuff in a Halo map that isn't even exposed by Guerilla and is only set/generated on map build. Copying the tag data is most of the work, sure, but it's the little things that keep things from looking messed up.

I will also say this too: Relying on closed source software for creating maps and assets is not good for the long term health of the modding community. The Halo Editing Kit is, unfortunately, just that. There are also numerous other projects out there that are closed source and will most likely never be open source. Such software should eventually be replaced with open source software.

I hope that one day the entire Halo Editing Kit and every other closed source tool that people have released will be replaced. The MEK is such a great effort for this, and I want Invader to be another useful alternative for modders to use, as there is so much we can do with this.


BKTiel
Joined: Mar 18, 2014

strong independent bird needs no cage


Posted: Jun 7, 2019 01:41 AM    Msg. 10 of 15       
you are a good man


DeadHamster
Joined: Jun 8, 2014

https://discord.gg/Neu4EJM


Posted: Jun 7, 2019 07:19 AM    Msg. 11 of 15       
Do you plan on replacing tool's full function set or just the section of map building? Because there's a lot of changes that could be made in relation to raw resource compiling which is where tool really falls short. Allowing multiple formats to be compiled to sounds/bitmaps and changing how shaders are for selected for compiling models/BSP would be the first two.

Tool requires specific source formats of course, and for shaders Tool goes through all the tags inside the TAGS folder and finds the first shader with a matching name and assigns it. However if a user has too many tags/files to check, this process fails and tool is unable to compile the model. I actually keep a separate directory just to compile models and BSPs.


Kavawuvi
Joined: May 24, 2018

Brrrrrrring Ha!


Posted: Jun 7, 2019 10:23 PM    Msg. 12 of 15       
I haven't done benchmarks in a long time. This time, I decided to take Invader and tool.exe and make them build every single multiplayer map in the game. Here are the results (click to expand):



Takeaways:

  • Invader is faster than tool.exe (around 411% faster on Linux, around 213% faster on Windows)

  • Invader is around 15-17% faster when compiled for 64-bit depending on OS

  • Invader is around 82% faster when compiled for Linux but still quite fast on Windows

If you want to test the exact builds I used, you can download them here. I don't recommend using these for anything serious as it's pretty far from completion. Note that when Invader is released, only 64-bit builds will be released. Therefore, if you want Invader to run on a 32-bit operating system, you will need compile it yourself.

Quote: --- Original message by: DeadHamster
Do you plan on replacing tool's full function set or just the section of map building? Because there's a lot of changes that could be made in relation to raw resource compiling which is where tool really falls short. Allowing multiple formats to be compiled to sounds/bitmaps and changing how shaders are for selected for compiling models/BSP would be the first two.

Tool requires specific source formats of course, and for shaders Tool goes through all the tags inside the TAGS folder and finds the first shader with a matching name and assigns it. However if a user has too many tags/files to check, this process fails and tool is unable to compile the model. I actually keep a separate directory just to compile models and BSPs.

I do agree that there are a lot of things that can be done for replacing tool.exe. Invader currently just does map building, though I do plan on having other functionality besides map building that could be useful. tool.exe is a program with a lot of uses and features, and replicating everything will not be one easy task.

In fact, the program for building map files is invader-build.exe rather than invader.exe as I'm also working on other tools as part of Invader such as invader-indexer (tag indexing) and invader-scenario (scenario tag editing for use with Blender). Invader isn't just one thing here.
Edited by Kavawuvi on Jun 7, 2019 at 10:34 PM


Kavawuvi
Joined: May 24, 2018

Brrrrrrring Ha!


Posted: Jun 8, 2019 04:46 PM    Msg. 13 of 15       
So here's an update:

I fixed lightning tags (I think?) so those should work fine now.

I'm still trying to figure out navigating the collision BSP tree so I can place AI encounters accurately. For some reason Invader isn't correctly setting some encounters to the correct BSP on certain maps like d40 and b30, so I'm going to be spending a great deal of time fixing those. I've been spending several days on this, too, but I'm coming fairly close. Just not close enough.

If I fix these issues and I fix other AI behavior issues, then campaign maps should work perfectly or close to perfection. So this means that I have come very close to finishing Invader.

I'm also planning on adding deduping to more types of tag data. I did it to animations and tag paths for a while and it reduced tag space usage by a around 1 MiB in some maps. This may not sound like a lot if you don't make maps, but a problem with the stock engine is that you're only allowed 23 MiB of tag space, and this space is shared with the currently loaded BSP. Saving a MiB means saving an extra 4% of that tag space.

As you can imagine, being limited to 23 MiB of tag space can make it very easy to go over the tag space limit on some maps. Adding a complex animation tag, using a complex BSP, or even simply adding enough spawns for something could very well push the map over the edge in some cases. But having an extra 4% of space could be the difference between the map failing to build and the map building successfully. The best part? Deduping doesn't require Open Sauce.

If anyone wants to test this, here: https://drive.google.com/open?id=1p1X9x6MyJGbYyd0qavef-cFWB2S78KVG.

Quote: --- Original message by: BioGoji1989
Here's a free bump so that you can post another update later!

Keep up the good work! I hope to see this tool replacement in action in a final build someday.

Ooh, thanks! That's very much appreciated (Also, thank you DeadHamster. I saw your post but it got deleted).

Edited by Kavawuvi on Jun 8, 2019 at 04:49 PM


SBB_Michelle
Joined: Nov 4, 2015

Floof


Posted: Jun 9, 2019 05:43 AM    Msg. 14 of 15       
Nice space gains. I hope you can figure out the ai BSP attachment thing without too much more hassle.

Gotta love tool being compiled as a debug build making it slow.

I really think having an opensource map compiler will benefit us all at some point.


Kavawuvi
Joined: May 24, 2018

Brrrrrrring Ha!


Posted: Jun 11, 2019 02:38 AM    Msg. 15 of 15       
Here's another update.

- AI encounters now default to 0xFFFF if they don't have any spawns. This should remove some of the "0 BSPs" warnings that you may see, notably The Silent Cartographer (b30.map).

- I figured out how to calculate a certain hidden value in the biped tag (that is, the head node index). Before this, I assumed it was always 12, but some bipeds have it set to something different. This value affects aim assist as well as certain cutscenes. This notably affects the part where Captain Keyes says "Let's give our old friends a warm welcome." in The Pillar of Autumn (a10.map), as setting it to 12 instead of the correct value results in the camera zooming in on his ear instead of his face.



Due to personal reasons, I'm going to be posting most updates on Open Carnage so if you want this build, I recommend going there. Check there if you want any further news on Invader. I'll still stick around here to answer questions if anyone has any, though, and I'll also post here when I finish version 1.0 of Invader.

Thank you for your interest!

Quote: --- Original message by: SBB_Michelle
Nice space gains. I hope you can figure out the ai BSP attachment thing without too much more hassle.

It's mainly just a matter of trying to calculate whether or not something is inside of a BSP. I have it down to something that's semi-accurate but not 100% accurate.

Quote: --- Original message by: SBB_Michelle
Gotta love tool being compiled as a debug build making it slow.

Yeah, that hurts a bit. Invader as a debug build is a bit slower than the release build, but it's still considerably faster than tool.exe. I suspect this is due to me using a newer compiler, a newer CPU architecture (x86_64 vs. x86), and probably just more efficient code altogether. I also do fewer checks than tool.exe, and I do nearly everything in RAM, so that may contribute to a non-trivial portion of the speed improvements.

Quote: --- Original message by: SBB_Michelle
I really think having an opensource map compiler will benefit us all at some point.

Me too. I think it's pretty impressive what people have done with modifying the existing tool.exe, but I imagine that having an open source map compiler will make things a lot easier.

 

 
Previous Older Thread    Next newer Thread







Time: Thu July 18, 2019 8:54 PM 172 ms.
A Halo Maps Website