MosesofEgypt has contributed to 292 posts out of 462973 total posts
(.06%) in 1,517 days (.19 posts per day).
20 Most recent posts:
Well good news is that everything you guys have mentioned doesnt seem to be a problem with my extractor. I havent tested recorded animations or the biped shadows, but i tested everything else you guys mentioned and i see no problems. I've also got everything completely extracting. I just need to make a usable extraction queue widget and fix a bug with 4byte colors and then you guys can try it out.
Could you name the issues you've had with the hek+? It'd really help me if i knew what to look out for. I have no doubt there will be bugs for a while, so any that you know of with hek+ will be ones i should specifically try to avoid.
The tool is called Refinery, and it's a map extractor/deprotector for Halo xbox/pc/ce/demo and Stubbs the Zombie on xbox/pc. Basically every type of Halo 1 map(including open sauce yelo maps). The ui isn't totally done yet, and some of its more specialized functionality wont be implemented for a while(deprotection using hashcaches and fixing tag classes), but I feel like I'm far along enough with it that a beta release should be ready it in a few weeks or so. And yes EmmanuelCD, it will REQUIRE python.
There are a few bugs I have to fix in the extraction that I know of, but otherwise I can extract every tag(seemingly) just fine.
I've got the framework set up for fixing mangled tag classes, but it'll require a decent amount of time and careful checking to make sure I type in all the offsets correctly. Once this is ready for everyone to try I'd REALLY, REALLY like to get feedback on it. If something seems wrong, I need to know. It will be a while before you guys should trust it to perfectly extract stuff. Also, it wont be able to extract from yelo resource caches cause I dont know how those are used. I might add it later if I can, but for now the open sauce extraction will be limited to regular .map and .yelo maps.
It will be in the MEK repository for download when it's ready. You'll need to run the mek_installer and tell it to upgrade as well, since I'm making changes to the underlying libraries. Cheers!
Edited by MosesofEgypt on May 23, 2017 at 09:12 PM
Edited by MosesofEgypt on May 23, 2017 at 10:08 PM
Edited by MosesofEgypt on May 26, 2017 at 01:21 PM
Just added a tool to convert an entire tagset to/from 60/30 fps(save for a few exceptions). The tool is named Halo_Tagset_Fps_Changer. Check the readme for information.
Seriously, check the readme. I will point anyone to this post who complains about something that's explained in the readme, even if they majorly screwed something up or dont understand what it did. It's that important. This tool can and will edit a vast majority of the tags in any folder you point it to and change lots of values all over the place. I haven't been able to fully test it, so for a few months you guys should use it with caution.
Also, you WILL need to update the mek in order to use this.
Edited by MosesofEgypt on May 5, 2017 at 01:25 AM
It's missing stuff because update and install have been replaced with MEK_Installer. I don't know why the installer isn't working, but if you rename it from MEK_Installer.pyw to MEK_Installer.py it should give you a printout of any errors on startup. Also, I'm editing the readme now to change the install and update instructions. I get so caught up in making changes that I sometimes forget to do things like updating readmes.
Sorry about the late reply, I was fixing a friend's GBA SP.
Edited by MosesofEgypt on Apr 24, 2017 at 06:46 PM
Update and let me know if the problem persists. I've made some changes, and the osv4 stuff works for me, so unless something was added that I'm unaware of then I believe this should fix anything that was broken.
EDIT: If it still persists, tell me what the mozzarilla.log says, or the console window. the log can be found in the mozzarilla directory wherever you had it installed.
Edited by MosesofEgypt on Apr 24, 2017 at 04:42 PM
If you don't understand why I will lose respect for you then you aren't thinking about it hard enough. I have my reasons, but nothing I say is going to change the mind of someone ready to spew an essay to try and prove their point. I'm not going to waste my breath on that.
If you actually try to charge for modding utilities you make, you will lose my respect as a modder, programmer, and researcher. That's the last I have to say about the subject.
Edited by MosesofEgypt on Apr 22, 2017 at 02:57 PM
I'll be honest and admit that i read about a two thirds, then skimmmed, but i don't believe you are correct about the reverse engineering aspect(re). Re is forbidden by the EULA, so any knowledge gained from it is considered by microsoft to be illegally obtained, as you broke your EULA that expressly forbids it. I'm not going to claim to be a lawyer, but neither should you by claiming that users WILL be able to make and sell HCE resources using your scratch built programs. It is still up to the court to decide, which can go either way from what i've read. It would be dangerous for users to start selling their halo assets without talking to a lawyer first, and even then microsoft may decide to enforce their eula anyway.
I've thought about this same thing with my programs, but i'm not brazen enough to make a post claiming that you can do it. RE is still highly debatable in court, especially when you did your own RE and thus did not employ clean room design. Again, i'm not going to claim that i am definitely correct, but you shouldnt lead users to believe that you are either.
I'm not reverting the commit since I've already applied others after it, but I am overwriting the change it with a new commit. Also, I'm not sure it's worth it to tweak Korns Guerilla. I just don't really see a big benefit in doing so when Mozz exists and your editor will too. If you want to fix that stuff, go right ahead. I wouldn't even know where to begin.
You should look at reclaimer for the max reflexive counts as well. I obtained all of my max sizes my either adding entries to guerilla till it capped, or if the limit was too high(around 128) I hex edited a reflexive count/rawdata size in a tag until it gave me an error about the size being too large. The nice thing is that if it decided the number was higher than the limit, it wouldnt even try to parse the rest of the file, so I could just keep tweaking the numbers without having to actually add to the tag. I even got the sbsp limits.
bsp = Struct("bsp",
reflexive("bsp3d nodes", bsp3d_node, 131072),
reflexive("planes", plane, 65535),
reflexive("leaves", leaf, 65535),
reflexive("bsp2d references", bsp2d_reference, 131072),
reflexive("bsp2d nodes", bsp2d_node, 65535),
reflexive("surfaces", surface, 131072),
reflexive("edges", edge, 262144),
reflexive("vertices", vertex, 131072),
material = Struct("material",
rawdata_ref("uncompressed vertices", max_size=4864000),
rawdata_ref("compressed vertices", max_size=2560000),
detail_object = Struct("detail object",
reflexive("cells", detail_object_cell, 262144),
reflexive("instances", detail_object_instance, 2097152),
reflexive("counts", detail_object_count, 8388608),
reflexive("z reference vectors", detail_object_z_reference_vector, 262144),
sbsp_body = Struct("tagdata",
reflexive("collision materials", collision_material, 512, DYN_NAME_PATH='.shader.filepath'),
reflexive("collision bsp", collision_bsp, 1),
reflexive("nodes", node, 131072),
reflexive("leaves", leaf, 65535),
reflexive("leaf surfaces", leaf_surface, 262144),
reflexive("surface", surface, 131072),
reflexive("lightmaps", lightmap, 128),
reflexive("lens flares", lens_flare, 256, DYN_NAME_PATH='.shader.filepath'),
reflexive("lens flare markers", lens_flare_marker, 65535),
reflexive("clusters", cluster, 8192),
rawdata_ref("cluster data", max_size=65536),
reflexive("cluster portals", cluster_portal, 512),
reflexive("breakable surfaces", breakable_surface, 256),
reflexive("fog planes", fog_plane, 32),
reflexive("fog regions", fog_region, 32),
reflexive("fog palettes", fog_palette, 32, DYN_NAME_PATH='.name'),
reflexive("weather palettes", weather_palette, 32, DYN_NAME_PATH='.name'),
reflexive("weather polyhedras", weather_polyhedra, 32),
reflexive("pathfinding surfaces", pathfinding_surface, 131072),
reflexive("pathfinding edges", pathfinding_edge, 262144),
reflexive("background sounds palette", background_sound_palette, 64, DYN_NAME_PATH='.name'),
reflexive("sound environments palette", sound_environment_palette, 64, DYN_NAME_PATH='.name'),
rawdata_ref("sound pas data", max_size=131072),
reflexive("markers", marker, 1024, DYN_NAME_PATH='.name'),
reflexive("detail objects", detail_object, 1),
reflexive("runtime decals", runtime_decal, 6144),
reflexive("leaf map leaves", leaf_map_leaf, 256),
reflexive("leaf map portals", leaf_map_portal, 524288),
You should check the Reclaimer library that mozz relies on. I've put all the hard limits in for all rawdata structs for both regular halo and open sauce(though I admit I somehow typed the max_size for the comment data as 2x larger).
recorded_animation = Struct("recorded animation",
SInt8("raw animation data"),
SInt8("unit control data version"),
BSInt16("length of animation", SIDETIP="ticks"), # ticks
rawdata_ref("recorded animation event stream", max_size=2097152),
source_file = Struct("source_file",
rawdata_ref("source", max_size=262144, widget=HaloScriptSourceFrame),
rawdata_ref("scenario editor data", max_size=65536)
rawdata_ref("script syntax data", max_size=380076)
rawdata_ref("script string data", max_size=262144)
for open sauce the script syntax and string data is increased
rawdata_ref("script syntax data", max_size=570076)
rawdata_ref("script string data", max_size=393216)
Edited by MosesofEgypt on Apr 18, 2017 at 12:40 AM
Just updated it again. Some of the images weren't extracting because some of the bitmaps have a different size for some of their structs. Turns out this structure I've been trying to figure out its meaning is almost certainly a versioning structure. It specifies the version of the struct, the number of them, and their byte size. This is sorta bad news for me since it means that I've gotta account for a bunch of different versions of - ya know what, I'll just paste that part of the skype chat in here:
so looks like the carbine cant be extracted
couldn't decompress it
turns out the tag was missing the extra 4 bytes in the body that were added when
they went from halo1 to halo2.
the tags structure is actually different than it should be!
this is interesting. this means that they had different versions for the tags and
something in the tag said which version it was.
that's what that tbfd is.
it really is a tag_block_field_definition.
the last 4 bytes are the size of the block. that carbine has it set to 108(the
number of bytes in halo 1 bitmaps) and most of the other halo 2 tags have it at 112.
it IS a versioning structure!
curiously.. the carbine reticule was the last one to be added to the sprite sheet.
maybe the carbine was the second to last weapon added?
hence the discrepancy?
this is gonna be kinda annoying.
i did a scan of the tags you sent me and logged all the unknown bytes in the tbfd
structures to try and figure out what they mean.
i'm thinking they're either an enumerator or flags.
and that they might specify which structure version the reflexive is for.
the thing is though, if they had a bunch of different structures, then i'd need to
know how to read each one and set up a switch to be able to properly read all tags.
i really hope they all got converted to the same final version when compiled into
a map, or else extracting tags would be a god damn pain.
i think it would though, since opening that carbine tag and resaving it upgraded it
to the new version.
but yeah, i was originally really scratching my head at what the hell the purpose
of those tbfd structures could be. now I know. This way they could work with any
version of tags without having to constantly upgrade them to incorporate any
changes to the engine.
a variant of this is probably still used in the current engine. that's prolly why
the new versions of their editors can make and work with all these old tags.
Edited by MosesofEgypt on Apr 7, 2017 at 01:37 PM
Just upgraded TeXource to be able to extract from H2Vista bitmaps as well.
Oh crap, it ISNT extra header information, they've changed the way their reflexives are stored. hmmmm. I can work with this.
So turns out that incorporating halo 2 into mozz is much more of a possibility in the past month than it ever has been.
The source code for prometheus was released a month ago, which includes mostly complete definitions for all the halo 2 tags. The definitions are missing a small chunk of field names though, and I'll still have to figure out the extra data in the tag header since proms dev team designed their own tag header that's completely different than h2vista's. But yeah, This is much more possible now. Give credit to the prometheus team for all that work and kornman for opening this possibility up to me.
Edited by MosesofEgypt on Apr 6, 2017 at 12:22 AM
I've written my own triangle strip generator in python that generates a single, contiguous strip of any mesh given to it using degen triangles to link the strips:
I've used it to write my own model compiler for Gauntlet Dark Legacy. It allows me to create new models from obj files. I could probably write a model compiler that parses jms files and creates render models, but only once I understand the structure of the model files.
I could honestly probably also write a converter that converts sounds from HCE into H2V tags since I'm gonna take a wild guess that they didn't change the encodings or bitrates since they still had to play on xbox.
Animations are another story. They could have changed god knows how many things when upgrading that crap. They might have even decided to go with some kinda interpolated frame data rather than explicit frame by frame data.
First off, everyone should run the upgrade function of the installer since I've fixed a chunk of significant bugs, some of which I introduced a while ago because I was in a rush when committing changes and didnt have time to properly check them.
Anyway, about the H2EK, I intend to at some point in the future, but it'll basically require me to do what I did with Halo 1 and map out every tag's structures, find out which ones are shared among all of them, put those ones in the shared commons, and figure out how things are different. As an example the header is 80 bytes rather than 64 and the non-filepath strings are now unlimited length rather than 31 char max.
I actually included an empty file in the h2ek folder, but I don't think it got uploaded with the pypi distribution. It's filename was "the stuff in here is a placeholder until i start on the h2 tag set". If enough people actually care about using Mozz for H2 then it'll give me more incentive to do it. As of right now it's one of those "It'd be really cool to do this, but I dont have the time so it's going near the bottom of my mental todo list". It'll be a lot easier if someone is able to provide me with the most recent version of kornman's h2 guerilla and any dll's required to run it. The version I've got is a beta and is pretty unstable, but I've been able to use it for some stuff.
Edited by MosesofEgypt on Apr 5, 2017 at 01:05 PM
Quote: --- Original message by: OrangeJuice
I got 3dsmax5
working on 64
-bit windows10. . . Natively.
It has no bridge tool. Or pelt-unwrapper. Or seam-unwrapper.
I did get DX9-renderer(max5 is from win95 era), Blitzkrieg 5, and Render To Texture working though :)
lol. Time for a self-imposed mapmaking challenge ! ! ! ! http://i.imgur.com/exR5eiG.jpg Edited by OrangeJuice on Apr 5, 2017 at 09:10 AM
Pfft, 5. I had to use 4.2 to make models and animations for warcraft 3. Here's an inverse kinematics test I did before I started animating in it.
I was running it on XP though.
I just checked the cyborg.biped that comes with the hek from a fresh install. The values are in little endian. I guess these were getting extracted incorrectly by whoever did the rest of the tag collection. I guess i can remove that part from my comment in the next update.
Also, about the spherical coordinates stuff, that's something i learned in calc 3. They could be spherical coordinates, but I'll leave it to you with your live memory editing expertise to discern their purpose. I think the last 2 might be 2 sint16's for a few reasons: the previous values seemed to come in pairs, the default value of both would be -1, -1, they dont make sense as a float or int32 because of how large they'd be.
About the crc again, i don't believe i'll change it unless i see a compelling reason to. For me it's almost the same as if the checksum is a set of null bytes because i dont see its effect. If it serves some important purpose(like speeding up verifying the integrity of a tag before it is compiled or something) then i'll implement it. I've got a feeling though, that it was probably only used by bungie's dev tools to make sure they werent working with corrupt tags since they couldnt afford to let data corruption slide.
Edited by MosesofEgypt on Apr 4, 2017 at 10:50 PM