A Community discussion forum for Halo Custom Edition, Halo 2 Vista, Portal and Halo Machinima

Home  Search Register  Login Member ListRecent Posts
Viewing User Profile for: Kavawuvi
About Contact
Joined: May 24, 2018 01:51 AM
Last Post: Jun 11, 2019 02:38 AM
Last Visit: Today @ 04:20 PM
Your Age:
What Games do you play:
Brrrrrrring Ha!

Send Private Message
Post Statistics
Kavawuvi has contributed to 37 posts out of 469161 total posts (.01%) in 426 days (.09 posts per day).

20 Most recent posts:
Halo CE General Discussion » [WIP] Invader Jun 11, 2019 02:38 AM (Total replies: 14)

Here's another update.

- AI encounters now default to 0xFFFF if they don't have any spawns. This should remove some of the "0 BSPs" warnings that you may see, notably The Silent Cartographer (b30.map).

- I figured out how to calculate a certain hidden value in the biped tag (that is, the head node index). Before this, I assumed it was always 12, but some bipeds have it set to something different. This value affects aim assist as well as certain cutscenes. This notably affects the part where Captain Keyes says "Let's give our old friends a warm welcome." in The Pillar of Autumn (a10.map), as setting it to 12 instead of the correct value results in the camera zooming in on his ear instead of his face.

Due to personal reasons, I'm going to be posting most updates on Open Carnage so if you want this build, I recommend going there. Check there if you want any further news on Invader. I'll still stick around here to answer questions if anyone has any, though, and I'll also post here when I finish version 1.0 of Invader.

Thank you for your interest!

Quote: --- Original message by: SBB_Michelle
Nice space gains. I hope you can figure out the ai BSP attachment thing without too much more hassle.

It's mainly just a matter of trying to calculate whether or not something is inside of a BSP. I have it down to something that's semi-accurate but not 100% accurate.

Quote: --- Original message by: SBB_Michelle
Gotta love tool being compiled as a debug build making it slow.

Yeah, that hurts a bit. Invader as a debug build is a bit slower than the release build, but it's still considerably faster than tool.exe. I suspect this is due to me using a newer compiler, a newer CPU architecture (x86_64 vs. x86), and probably just more efficient code altogether. I also do fewer checks than tool.exe, and I do nearly everything in RAM, so that may contribute to a non-trivial portion of the speed improvements.

Quote: --- Original message by: SBB_Michelle
I really think having an opensource map compiler will benefit us all at some point.

Me too. I think it's pretty impressive what people have done with modifying the existing tool.exe, but I imagine that having an open source map compiler will make things a lot easier.

Halo CE General Discussion » [WIP] Invader Jun 8, 2019 04:46 PM (Total replies: 14)

So here's an update:

I fixed lightning tags (I think?) so those should work fine now.

I'm still trying to figure out navigating the collision BSP tree so I can place AI encounters accurately. For some reason Invader isn't correctly setting some encounters to the correct BSP on certain maps like d40 and b30, so I'm going to be spending a great deal of time fixing those. I've been spending several days on this, too, but I'm coming fairly close. Just not close enough.

If I fix these issues and I fix other AI behavior issues, then campaign maps should work perfectly or close to perfection. So this means that I have come very close to finishing Invader.

I'm also planning on adding deduping to more types of tag data. I did it to animations and tag paths for a while and it reduced tag space usage by a around 1 MiB in some maps. This may not sound like a lot if you don't make maps, but a problem with the stock engine is that you're only allowed 23 MiB of tag space, and this space is shared with the currently loaded BSP. Saving a MiB means saving an extra 4% of that tag space.

As you can imagine, being limited to 23 MiB of tag space can make it very easy to go over the tag space limit on some maps. Adding a complex animation tag, using a complex BSP, or even simply adding enough spawns for something could very well push the map over the edge in some cases. But having an extra 4% of space could be the difference between the map failing to build and the map building successfully. The best part? Deduping doesn't require Open Sauce.

If anyone wants to test this, here: https://drive.google.com/open?id=1p1X9x6MyJGbYyd0qavef-cFWB2S78KVG.

Quote: --- Original message by: BioGoji1989
Here's a free bump so that you can post another update later!

Keep up the good work! I hope to see this tool replacement in action in a final build someday.

Ooh, thanks! That's very much appreciated (Also, thank you DeadHamster. I saw your post but it got deleted).

Edited by Kavawuvi on Jun 8, 2019 at 04:49 PM

Halo CE General Discussion » [WIP] Invader Jun 7, 2019 10:23 PM (Total replies: 14)

I haven't done benchmarks in a long time. This time, I decided to take Invader and tool.exe and make them build every single multiplayer map in the game. Here are the results (click to expand):


  • Invader is faster than tool.exe (around 411% faster on Linux, around 213% faster on Windows)

  • Invader is around 15-17% faster when compiled for 64-bit depending on OS

  • Invader is around 82% faster when compiled for Linux but still quite fast on Windows

If you want to test the exact builds I used, you can download them here. I don't recommend using these for anything serious as it's pretty far from completion. Note that when Invader is released, only 64-bit builds will be released. Therefore, if you want Invader to run on a 32-bit operating system, you will need compile it yourself.

Quote: --- Original message by: DeadHamster
Do you plan on replacing tool's full function set or just the section of map building? Because there's a lot of changes that could be made in relation to raw resource compiling which is where tool really falls short. Allowing multiple formats to be compiled to sounds/bitmaps and changing how shaders are for selected for compiling models/BSP would be the first two.

Tool requires specific source formats of course, and for shaders Tool goes through all the tags inside the TAGS folder and finds the first shader with a matching name and assigns it. However if a user has too many tags/files to check, this process fails and tool is unable to compile the model. I actually keep a separate directory just to compile models and BSPs.

I do agree that there are a lot of things that can be done for replacing tool.exe. Invader currently just does map building, though I do plan on having other functionality besides map building that could be useful. tool.exe is a program with a lot of uses and features, and replicating everything will not be one easy task.

In fact, the program for building map files is invader-build.exe rather than invader.exe as I'm also working on other tools as part of Invader such as invader-indexer (tag indexing) and invader-scenario (scenario tag editing for use with Blender). Invader isn't just one thing here.
Edited by Kavawuvi on Jun 7, 2019 at 10:34 PM

Halo CE General Discussion » [WIP] Invader Jun 6, 2019 05:47 PM (Total replies: 14)

Many of the AI encounters do not work properly still. Navigating this BSP tree thing has been quite difficult as it's not that well documented.

However, many multiplayer maps work quite well. It's not perfect, but it's getting pretty close to that and I'm really excited for when I complete this.

I may start releasing beta builds of Invader soon so people who don't want to compile Invader from source can test it.

That said, I don't want my topic locked (No double post rule FTW! /s), so when I start releasing beta builds, I'll only be able to post new builds on this topic whenever someone replies. Otherwise, you'll be able to find releases on the GitHub repository if they come out. I hope that's okay.

Quote: --- Original message by: OrangeJuice
It was fun seeing the earlier screenshots of it and the interesting ways it messes up halo, now this looks even nicer

Making a tool.exe replacement has been a long journey with many ups and downs. One of the main obstacles to doing this is the fact that there is so much stuff in a Halo map that isn't even exposed by Guerilla and is only set/generated on map build. Copying the tag data is most of the work, sure, but it's the little things that keep things from looking messed up.

I will also say this too: Relying on closed source software for creating maps and assets is not good for the long term health of the modding community. The Halo Editing Kit is, unfortunately, just that. There are also numerous other projects out there that are closed source and will most likely never be open source. Such software should eventually be replaced with open source software.

I hope that one day the entire Halo Editing Kit and every other closed source tool that people have released will be replaced. The MEK is such a great effort for this, and I want Invader to be another useful alternative for modders to use, as there is so much we can do with this.

Halo CE General Discussion » [WIP] Invader Jun 4, 2019 01:38 AM (Total replies: 14)

I got AI encounters to not T-pose and actually work. There are still some things to fix (model permutations and some AI behavior issues) and not all AI encounters are fixed, but the campaign is now playable.

Quote: --- Original message by: PRPatxi
It's like 8 years since I last saw someone using switch statements in their code.

I use what I feel makes sense at the time for what I'm writing. Switch statements are pretty useful for control flow.

Edited by Kavawuvi on Jun 4, 2019 at 05:16 AM

Halo CE General Discussion » [WIP] Invader May 31, 2019 04:18 PM (Total replies: 14)

Quote: --- Original message by: lolslayer
Sounds nice, are there already limitations of tool that Invader breaks?

Yes. Here is a short list:

  • Stock tool does not allow maps to be >384 MiB. Invader does not have this limitation and will happily output maps up to 4 GiB, and Halo will read the maps fine.

  • Stock tool does not allow some reflexives/tag blocks to exceed a certain limit such as netgame equipment. Invader does not have restrictions for most of these, and Halo will read the maps fine.

  • Stock tool only allows indexed, external tags in multiplayer maps despite the game not having this restriction. This results in broken bitmaps and sounds as well as English strings when playing an English UI/SP map on a non-English installation. Invader will use indexed tags for all types of maps, allowing the map to work on all languages of the game.

  • Invader does a small amount of deduplication to tag data, allowing you to put a little bit more tag data in the 23 MiB tag space. I'm planning on expanding on this feature later, but this may mean that you can build some maps that cannot possibly be built in tool.exe without removing tags.

  • Invader does not fail to open bitmaps.map, sounds.map, or loc.map if you have Halo open.

Invader also has a nicer, easier-to-read text output than tool.

Couldn't read map file './toolbeta.map'
the meter definition ui\hud\cyborg shield does not specify a stencil bitmap group.
the meter definition ui\hud\cyborg body does not specify a stencil bitmap group.
WARNING: 1 clusters in structure_bsp levels\a10\a10_space have no background sound or sound environment.
WARNING: command list DUP107~look_at_halo could be ambiguously attached to BSPs #1 and #8
WARNING: command list DUP119~look_at_halo could be ambiguously attached to BSPs #1 and #8
culling uncompressed model vertices...done
culling uncompressed structure bsp vertices...done
culling uncompressed model animation data...done
building predicted resources for structures...done
structure bsp 'levels\a10\a10a' is 8.25M
structure bsp 'levels\a10\a10b' is 10.94M
structure bsp 'levels\a10\a10c' is 8.04M
structure bsp 'levels\a10\a10d' is 7.27M
structure bsp 'levels\a10\a10e' is 10.35M
structure bsp 'levels\a10\a10f' is 11.39M
structure bsp 'levels\a10\a10g' is 11.49M
structure bsp 'levels\a10\a10_space' is 1.90M
structure bsp 'levels\a10\x10hangar' is 3.62M
tag headers and names are 0.30M
streaming model vertex and index buffers...done
streaming tags.....................................done
writing vertex and index buffers...done 8.70M
writing tags...done (3724 tags for 9.79M)
total tag size is 21.28M (1.72M free)
compressing 190.96M...done
successfully built cache file.
Cache pack file bitmaps hits: 606 for 33.23M
Cache pack file bitmaps adds/misses: 438 for 46.79M
Cache pack file sounds hits: 428 for 7.36M
Cache pack file sounds adds/misses: 2514 for 53.64M
Cache pack file loc hits: 52 for 0.11M
Cache pack file loc adds/misses: 13 for 0.01M

Scenario name:     a10
Tags: 3724 / 65535 (82.33 MiB)
BSPs: 9 (73.17 MiB)
levels\a10\a10a (8.24 MiB)
levels\a10\a10b (10.93 MiB)
levels\a10\a10c (8.03 MiB)
levels\a10\a10d (7.26 MiB)
levels\a10\a10e (10.34 MiB)
levels\a10\a10f (11.37 MiB)
levels\a10\a10g (11.47 MiB)*
levels\a10\a10_space (1.90 MiB)
levels\a10\x10hangar (3.61 MiB)
Tag space: 21.06 / 23.00 MiB (91.57 %)
Model data: 8.43 MiB
Bitmaps/sounds: 98.85 MiB
Indexed tags: 572
File size: 189.89 MiB
Time: 1015.425 ms

You can also make it not output anything at all, too. Outputting text to console takes a lot more time than it sounds.

That's all I can think of off the top of my head. I've got more plans for it if/when I finish it.

Edited by Kavawuvi on May 31, 2019 at 04:18 PM

Halo CE General Discussion » [WIP] Invader May 30, 2019 02:07 AM (Total replies: 14)

Invader is a work-in-progress, cross-platform, open source project that builds cache files. I've been working on this for a while and I figured I'd share my progress.

How close to completion is Invader?
Invader is still fairly incomplete and won't work with many maps, especially maps with scripts. I do not recommend using it outside of testing for now, but it is working with more and more maps as progress is being made. This is the reason why I don't have binary builds of it available just yet.

Why make this when we have Tool?
The biggest drawbacks of tool.exe are that it's unsupported and closed source. We need it to do more than what it did when it came out, but the best we can do is hacking and patching the executable. This is very limiting when it comes to wanting to do more than what is allowed. Considering the MEK is replacing parts of the Halo Editing Kit, it makes sense to replace this enigmatic piece of software.

In layman's terms, tool.exe is a piece of software filled with black voodoo magic, and being software from the 90's, it sometimes requires various demonic rituals in order to function as intended. The MEK is the bane of evil and Invader is the light.

What are some features?
One useful feature of Invader is that it can build cache files in any tag order you want. This means that you can export the tag list of a multiplayer map, rip the map, modify the rip, and then rebuild the map using the same tag order as before. You can then run the map through Refinery's CRC spoofer, allowing you to use that map in place of the original map online. This is great for if you want to add or change content on the stock multiplayer maps while still allowing these maps to be played online.

Invader is also very fast. Here's a benchmark I ran last October.

Will Invader work on a Mac?
While I don't have any means of building Invader for macOS, it will most likely compile just fine considering it compiles perfectly fine for Linux and Windows. If so, this along with the MEK can be a useful step for Mac users being able to make maps without having to resort to Wine or having an installation of Windows. It's sure been long enough, after all.

How do I get Invader?
Currently, there are no binary builds of Invader as it is still too incomplete to start releasing such builds (see reasons above). Instead you must build it from source. You can get it here: https://github.com/Kavawuvi/Invader.

Presently, to build from source, you will need CMake (https://cmake.org/) as well as a C++ compiler. Windows users can use MinGW which you can get from MSYS2 for Windows (https://www.msys2.org/). I've not tested it with MSVC, but it will probably compile with it. Linux users can simply install GCC or Clang through their favorite package manager, and macOS users can get it through Xcode via the command-line tools.

Halo CE General Discussion » Proposal: How to share tags and maps May 23, 2019 11:27 PM (Total replies: 36)

Quote: --- Original message by: sparky
To properly share a tag:

    1. compile it into a map
    2. use a program like HEK+ [EDIT: Try using Refinery] to recursively extract the tag to a new folder
    3. .zip the folder
    4. upload the zip to Drop Box
    5. post the link here

Quote: --- Original message by: sparky
Please comment on this, for example argue for using RAR, argue for using something other than Drop Box, and telling me you can't upload to CE3 or HAC (anymore?). Also, someone share a link to CE3 and HAC map upload terms and forms. Thanks.

I would use the .7z file format over .zip for tags, too, as LZMA compression is much better, compression ratio-wise, than DEFLATE. You can optionally create a solid archive, too, improving ratios even further.

The obvious drawback is that it's not immediately available to you on macOS or Windows, but of course, neither is the Halo Editing Kit immediately available to you when you install Halo Custom Edition.

Quote: --- Original message by: BioGoji1989
Okay, so... who exactly are you replying to in this post?

Edited by Kavawuvi on May 23, 2019 at 11:29 PM

Quote: --- Original message by: DeadHamster
I'm all for the release of source code, by the creator of the content. Not by some salty little kid who's upset. So again. Were you going to be updating and maintaining this? Or did you just want to throw your little tantrum because somebody didn't release their source code right away?

He seems pretty happy to me.
Edited by Kavawuvi on May 23, 2019 at 01:21 PM

Quote: --- Original message by: DeadHamster
Laika- I'm sure you intend to release the source, but along with what Vaporeon and Michelle have said, I'm gonna recommend an archived source be uploaded somewhere just as a backup. Nobody is going to make a fork while you're still active, and it'll ensure that if you suddenly disappear as quickly as you've reappeared, then we aren't stuck here with a partially-broken tool that could potentially do amazing things. Also, there's a lot of smart people here. Someone could definitely help out and figure some things out.

In the end you're well within your rights to tell all of us to f*** off and keep the source to yourself. This is a huge gift to the community and we have to be grateful for it. But the standard now is that everything is released open-sourced, unprotected in as accessible a format as possible.

If you don't want to do that now, maybe send Masterz an archive of the source? He doesn't know how to do anything with it, the rest of the community knows it exists with him, and you don't have to worry about someone making a fork and messing everything up while you're still around.

Something I do with many of my incomplete projects now is that, if I don't feel the source is ready to be posted, I'll regularly send copies of the source code (usually on every day I work on it) to a friend who lives far away from me. I've been doing this with Chimera, Dark Circlet, Invader, Salamander, and other projects.

If I suddenly disappear off the Internet for a long period of time (e.g. I get hit by a bus, Trump drafts me into the military, my house burns down, etc.), they will release the unfinished source code with the hope that someone else will finish my work. Also, if I lose the source, they have those copies I've been sending them.

Edited by Kavawuvi on May 21, 2019 at 02:51 PM

Quote: --- Original message by: sparky
I am guessing that they did padding in the (interface) definitions -- which are like templates -- because of the static pointer calculations during map compilation and reading the data into memory. But contrarily, there is more padding than necessary for 32-bit integer memory address alignment. And it seems strange that the padding is unmodified, skipped, by the HEK programs that write to disk. Perhaps they were simply frivolous with skipped bytes to give themselves more leeway as they anticipated adding more metadata to the template definitions. Anyway, I'm not using template style, so it doesn't really matter much to me because I'm not trying to remake the game engine literally.

Most padding is actual padding. There are, however, plenty of values that Guerilla does not write to that tool.exe actually does write to. If these values are left at 0, unexpected things may happen such as dying instantly from fall damage or various effects not triggering on dedicated servers when they should.

1) Yes.
2) No. float_t can be used interchangeably with float.
3) As long as the size is correct when compiled for 32-bit and 64-bit, it's fine.
4) Padding is nice if you want to directly map data to a struct. If you don't need to do this, though, then that works.
5) Considering you're using C and not C++ and thus don't have access to the magical power of templates, that's fine.

Halo CE General Discussion » I fixed the sun Apr 8, 2019 10:21 PM (Total replies: 3)


This feature to Chimera I added fixes the sun, allowing it to be the correct size at resolutions above 1024x768. This means it won't look like a moon at 4K.
Edited by Kavawuvi on Apr 8, 2019 at 10:22 PM

Quote: --- Original message by: DeadHamster
I believe you actually meant 0x8, ending at 0xC. 0x8 contains the maps filesize in bytes written as a 32 bit int, whereas 0xC is all 0's.

Oh yeah, that's what I meant.

Anyway, cool. And yeah, you still need to make a map that's such a filesize to take advantage of it.

I'm working on a map building tool, myself (Invader), and I'm thinking I'll just have it just set the file size to 0 when it builds a map.

It's worth noting that setting the file size (32-bit integer at offset 0x8) in the map header to 0 will allow such large maps to work without needing mods like Open Sauce.

Halo only reads this value to check if it's <384 MiB. If it is, the map is (probably) valid.

Actually, it also checks if it's >128 MiB, and if it is, it leaks resource descriptors like crazy. It also does weird things like the name in the header not being read properly.

Setting this value to 0, therefore, is always a good idea even if the map would work otherwise.

Edited by Kavawuvi on Apr 1, 2019 at 03:03 AM

Halo CE General Discussion » Do you see Halo on switch? Mar 31, 2019 02:37 PM (Total replies: 1)

Is the Minecraft Halo pack not good enough?

This is because your second weapon is supposed to be the assault rifle you pick up after killing the three grunts.

Also, these scripted weapon pickups do not check how many weapons you are holding, thus if you picked up a plasma pistol, then when the assault rifle is given to you, you'll have a third weapon.

You can witness this by triggering a glitch where the grunts do not appear when they should, thus you don't obtain Keyes's pistol immediately. If you do this, get the assault rifle, pick up any second weapon, and then go back to trigger the grunt spawn, then Keyes's pistol will be given to you, resulting in you having three weapons. Also, if you already have a pistol, you'll have two pistols.

Edited by Kavawuvi on Mar 15, 2019 at 05:22 PM

Halo CE General Discussion » Serpent Mar 15, 2019 01:59 PM (Total replies: 1)

Here is a little fun project I did last month. This provides an alternative syntax for writing Halo scripting languages that is more similar to Lua than Lisp. Note that this doesn't add anything to the Halo scripting language, as I can't really make it do things you couldn't do before. I am planning on adding loops, though Halo's limited script stack size will mean you have to use these loops sparingly.

Here's an example Hello World script:
global string hello_world = "hello world"

startup start
developer_mode = 127
if (4 + 1) > 4
print("5 is greater than 4!")
elseif 5 / 0
print("wait, what?")

When parsed and converted into Halo script, it results in this script:
(global string hello_world "hello world")(script startup start (set developer_mode 127)(if (> (+ 4 1)4)(begin (print hello_world)(print "5 is greater than 4!"))(if (/ 5 0)(print "wait, what?"))))

The resulting script can then be compiled by Sapien into Halo Custom Edition. Unnecessary spaces and quotation marks are stripped and removed.

Again, this does not allow you to do anything you couldn't do in HSC before, but it may make scripting easier for people who are not used to Lisp. In the end, it's just a fun little project.

If you actually do want to use this, you will need Python. This might be a dealbreaker for some of you as Python can be really hard to install for some people, but if people are interested, I may make a more portable version later.

Source and further documentation: https://github.com/Kavawuvi/serpent

Edited by Kavawuvi on Mar 15, 2019 at 02:02 PM

Halo CE General Discussion » Map Compression Mar 10, 2019 04:13 PM (Total replies: 2)

Quote: --- Original message by: Banshee64
that's cool, how does it work? does a header flag tell the game it's compressed and it just reads the raw compressed data after the header? or is it done in blocks?

Actually, either work! The header contains a few fields: the block count and compression algorithm. Currently only uncompressed and LZMA are supported.

If the block count is 0, then all of the data after the block is considered one stream of data. This is like what Xbox does. This is what I use for maps that are under 8 MiB when uncompressed. Obviously uncompressed maps don't use blocks, too, so it will be set to 0 here.

If the block count is nonzero, then there is a table located directly after the header containing information about each block (decompressed size, compressed size, and file offset) with the number of entries equal to the block count. In decompression, this data is then decompressed and reconstructed into the original map file.

The benefit to doing blocks is that it can be easily split up into threads, and this massively reduces compression and decompression time while only slightly affecting the compression ratio.

Whether or not to use blocks as well as the block size to use is determined by the tool based on the file size, but it can be overridden. Larger maps use larger blocks, smaller maps use smaller blocks.

Edited by Kavawuvi on Mar 10, 2019 at 04:18 PM

Halo CE General Discussion » Map Compression Mar 10, 2019 03:04 PM (Total replies: 2)

Wow, a topic by me. I don't usually do that.

Using LZMA compression, I've managed to reduce the file size of various maps. This isn't the same as putting a map in a .7z file, as the header of the map still remains uncompressed. Rather, it's more comparable to compressed Xbox maps. That said, it does take a little longer to load a compressed map than an uncompressed map. Having more CPU cores and threads can potentially mean better decompression times, but this will vary from map to map, as will compression ratio.

For example, the entire Refined campaign takes up 2.649 GiB normally, but as compressed maps, it takes up only 1.003 GiB. A more extreme example is With Whiteness which, when uncompressed, is 759.7 MiB, but compressed results in a file size of 65 MiB. For decompression time, the largest map in the Refined campaign, d40.map, takes around 1.67 seconds to decompress when allowed a thread count of 8 or more, but with fewer threads, the time goes up significantly. I'll include this map and several other maps in a table. As for With_Whiteness, because most of the data is just unused 00's, the decompression time is very fast: 0.89 seconds at 12 threads, 1.26 seconds at 8 threads, 1.71 seconds at 4 threads, and 3.12 seconds with 2 threads.

Not everything works with this, however. Open Sauce maps, most protected maps, and maps with embedded Chimera scripts cannot be used with this, and most of this is, unfortunately, technically beyond my control.

Here is a table with some benchmarks:

The benefit to this is that large map folders can use less space without having to be stored in an archive like a .rar or .zip. Also, you do not have to cut back on image quality nearly as much. The loading time penalty is minor, and it can be made even less apparent by caching recently used maps.

Edited by Kavawuvi on Mar 11, 2019 at 12:28 AM

Time: Tue July 23, 2019 11:46 PM 250 ms.
A Halo Maps Website