A Community discussion forum for Halo Custom Edition, Halo 2 Vista, Portal and Halo Machinima

Home  Search Register  Login Member ListRecent Posts
  
 
»Forums Index »Halo Custom Edition (Bungie/Gearbox) »Halo CE General Discussion »[BETA RELEASE] Refinery 0.7

Page 1 of 3 Go to page: · [1] · 2 · 3 · Next
Author Topic: [BETA RELEASE] Refinery 0.7 (76 messages, Page 1 of 3)
Moderators: Dennis

MosesofEgypt
Joined: Apr 3, 2013


Posted: May 31, 2017 10:34 PM    Msg. 1 of 76       
All right everyone, here it is. Refinery is NOT finished(hence the version 0.7), but it can do most of what you guys need it to do.

Here's a list of what Refinery can currently do:

* Load and extract tags and data from Halo 1 Xbox, PC, CE, PC Demo, and Yelo maps
* Load and extract tags and data from Stubbs the Zombie maps on PC and Xbox
* Load and extract tags and data from CE(not PC) resource maps(loc.map, bitmaps.map, and sounds.map)
* Load Halo 2 maps and extract data from Halo 2 Vista maps.


* Display all pertinent information contained in the map's header, tag index,
scenario bsp headers, and Open sauce header(if it exists).
* Has 3 ways to view the map contents: hierarchy, class, and hybrid
Hierarchy sorts the tags by folder, like a regular directory structure.
Class sorts all tags by their class(bitmap, gbxmodel, weapon, etc)
Hybrid combines the two. It sorts everything by class, then inside
each class is a directory structure of all tags of that class.

* Extract all Halo 1 tags
* Extract all Open sauce tags
* Extract the cheape.map from Yelo maps
* Extract most Stubbs the Zombie tags(excludes vege, imef, terr, bipd, vehi,
and obje tag classes)

* Decompress and save Xbox maps
* Save any loaded non-resource map(rearranges structures and renames protected
tags so the map can be opened by other map tools)

* Allows you to rename tags and folders of tags in any loaded non-resource map(tags
have to at least partially exist in the map to rename them, i.e. they cannot be
in a resource map.)
* Allows previewing any tag in the map as if loaded in Mozzarilla.
Changes made to the tag will NOT be saved to the map. Exporting works in the
window, but rawdata will be in little-endian and other meta-specific details
will not have been tweaked, so don't expect to be able to properly import ANY
exported block into a tag.
* Allows extraction by tag class as well as by folder or individual tags.


Here's what is currently NOT implemented that I plan to add at some point in the future:

* Recursive renaming(you rename a biped/weapon/etc and it'll intelligently put most
tags it references in the same folder).
* Using hashcaches of collections of tags to detect and rename any matching ones in the map.
* Using heuristics to rename tags based on gleamed information(weapon/vehicle names in
the strings list, shaders/bitmaps/physics/collision/animations that go with them, etc).


If you guys can think of anything else that you'd like to see(no bitmap/model/bsp preview please) then I'm open to suggestions.


Here are all the known bugs that I can think of right now(I plan to fix them):

* Tag explorers might not be sorted if you edit them(if you rename an entire folder
and it disappears from the tree, it will be added to the bottom instead of alphabetically)



So now that you've read all that, here's a few tips on using Refinery:

* You can double click an item in the map explorer(or select multiple and hit enter)
to either rename it/them, add it/them to the queue, or preview the tag(only if
selecting individual tags).
* Renaming a folder to one that already exists puts all tags in that folder into the new one.
* Renaming is case insensitive, and everything gets converted to lowercase.
It's just easier this way.
* If the project_yellow_globals tag doesnt extract, go into the settings and check the
"Use old project_yellow_globals definition" button.
* If you just want stuff from the bitmaps.map, sounds.map, or loc.map just load that map up
instead of an actual map.
* If you only want one type of tag(all bitmaps) then change the explorer to "class" view,
select the tag class(bitm in this case), and add it to the queue.
* You can edit the details of something in the queue by double clicking it.


Have some pretty pictures:




Let me know of any bugs you guys come across. I will try my darndest to fix them, especially if they are extraction bugs.


DOWNLOAD(Requires Python 3.4 or higher)
https://bitbucket.org/Moses_of_Egypt/mek/downloads/


Click "Download repository" and extract the contents somewhere. Read the readme.txt for installation instructions. Refinery comes with the MEK. Whenever I have an update for it(which wont be very often at all) you'll likely need to re-download the MEK and paste the Refinery.py and refinery folders from the download over the old ones.
Edited by MosesofEgypt on Sep 26, 2017 at 04:30 PM
Edited by MosesofEgypt on Sep 26, 2017 at 09:50 PM


MatthewDratt
Joined: Sep 11, 2010

CE3 - Takedown - HaloCE3.com


Posted: May 31, 2017 11:11 PM    Msg. 2 of 76       
Cool


HandofGod
Joined: Jan 10, 2014

Halo Mac/MD Player & Modder


Posted: Jun 1, 2017 09:41 PM    Msg. 3 of 76       
Nice


Super Flanker
Joined: Oct 5, 2012

The length of your life depends on my aim.


Posted: Jun 3, 2017 12:14 PM    Msg. 4 of 76       
Wonderful


MosesofEgypt
Joined: Apr 3, 2013


Posted: Jun 3, 2017 01:18 PM    Msg. 5 of 76       
Patronising? I want some actual feedback.


Banshee64
Joined: Dec 4, 2012

oify


Posted: Jun 3, 2017 03:18 PM    Msg. 6 of 76       
I really like it so far, A lot of these features were already possible but it's nice to have them updated and all in one place. The layout is nice, and the integration with your other tools is very useful. Do you plan on expanding it to say Halo 2 maps? I know the only real benefit of that would be ripping, but a lot of halo 2 tag extraction hasn't been done yet.


Ling Ling Lang
Joined: Jul 26, 2016

Are you finding Ling Ling?


Posted: Jun 3, 2017 04:24 PM    Msg. 7 of 76       
Quote: --- Original message by: MosesofEgypt
Patronising? I want some actual feedback.


I've only done some minor testing (ripping b30 and a few other levels + viewing some assets in sapien) but everything so far looks awesome. Packaging all your tools together in a repository is a really good idea as well, both for distribution and awareness.

I'm still having the same problems I had with Mozz on W7. I'm not in a protected directory or anything, but I still can't just launch a Python file by double clicking it (it just opens a cmd for a brief second before closing). I can use cmd or a batch file to run Refinery (and your other apps) just fine. Am I missing something obvious?


DeadHamster
Joined: Jun 8, 2014


Posted: Jun 3, 2017 06:18 PM    Msg. 8 of 76       
Quote: --- Original message by: Ling Ling Lang
I'm still having the same problems I had with Mozz on W7. I'm not in a protected directory or anything, but I still can't just launch a Python file by double clicking it (it just opens a cmd for a brief second before closing). I can use cmd or a batch file to run Refinery (and your other apps) just fine. Am I missing something obvious?


No you're not, Python's Non-Console version crashes, whereas the Console version launches. At least, it does for me on Win 7, haven't upgraded in a bit though.


TheHaloKnight
Joined: Jun 11, 2015


Posted: Jun 3, 2017 08:08 PM    Msg. 9 of 76       
Can anybody confirm that bipeds have shadows on ripped BSPs with this utility ?
Edited by TheHaloKnight on Jun 3, 2017 at 08:08 PM


DeadHamster
Joined: Jun 8, 2014


Posted: Jun 3, 2017 11:07 PM    Msg. 10 of 76       
Quote: --- Original message by: TheHaloKnight

Can anybody confirm that bipeds have shadows on ripped BSPs with this utility ?
Edited by TheHaloKnight on Jun 3, 2017 at 08:08 PM


MosesOfEgypt can confirm that the shadows do not show. He specifically confirmed this in the first post.


MosesofEgypt
Joined: Apr 3, 2013


Posted: Jun 4, 2017 03:31 PM    Msg. 11 of 76       
Not planning to implement Halo 2, but if it wouldn't take too much work then it might be something worth investigating later.

Also, idk what the problem is with W7. I don't have these issue when I run it on my laptop with windows 7, so I can't really debug the problem. Try the console version of it since that runs with a console window open. If you have that same issue with any of the tools, change the extension from pyw(python windowless) to py. The extension forces it to run with or without a console window.

Finally, I'd like some help figuring out why there aren't shadows. Vehicles have them, but bipeds dont. Same goes for the rest of the bugs. If any of you can tell me what the issue is, It'd be really helpful.


DeadHamster
Joined: Jun 8, 2014


Posted: Jun 4, 2017 04:06 PM    Msg. 12 of 76       
For biped shadows Ive noted that removing a lightmap reference does not remove a level's lighting in Sapien. If you could verify thats true that might be a good place to start. Perhaps theres tagdata that's stripped during compilation, like the scripts are.


not giraffe
Joined: Jul 17, 2014


Posted: Jun 4, 2017 07:03 PM    Msg. 13 of 76       
Quote: --- Original message by: MosesofEgypt
Finally, I'd like some help figuring out why there aren't shadows. Vehicles have them, but bipeds dont. Same goes for the rest of the bugs. If any of you can tell me what the issue is, It'd be really helpful.

Sprinkle's realtime game editor saves bsp tags correctly.
http://forum.halomaps.org/index.cfm?page=topic&topicID=45788

You could compare tags.


MosesofEgypt
Joined: Apr 3, 2013


Posted: Jun 5, 2017 03:21 PM    Msg. 14 of 76       
I'll keep that in mind. I'm currently trying to debug this issue with the audio. At least I can take solace in the fact that Halo PC a10 raises an unknown exception when I try to compile it with tags ripped with hek+, but compiles without any errors when ripped with refinery.


Ki11erFTW
Joined: Jul 4, 2009

You've seen nothing yet.


Posted: Jun 7, 2017 01:15 AM    Msg. 15 of 76       
Relevant post below


MosesofEgypt
Joined: Apr 3, 2013


Posted: Jun 7, 2017 01:37 AM    Msg. 16 of 76       
I've been working on fixing bugs with Refinery, and have gotten to the point that I can extract a10 from Halo PC and recompile it for Halo CE with only a handful of bugs. The current bugs are as follows:

* bipeds dont have shadows(doesnt seem to occur on all recompiled maps, but does on a10)
* keyes is about one foot too far to his left in one scene in the intro cutscene.
see picture below
* grenades dont chain detonate for some reason.
Flags in equipment and damage effect are properly set and the damage type is set
as grenade. Not sure what is going on.
* shaders on certain scenery are borked(not my fault, ce's rendering for
shader_environment is broken on scenery)

This is an overlay of a screenshot of the keyes scene I'm talking about. The Halo PC one is set to 50% transparency and overlaid on top of one from my recompiled a10.


Killa uploaded the recompiled map below so you guys can see what refinery can do right now. I can honestly say that in it's current v0.8.2 form it is most likely a more reliable and accurate tag extractor than the HEK+ at this point.

I literally just had to tell refinery to extract the tags to a folder, put tool in the folder, and run "tool build-cache-file levels\a10\a10". No tag editing required. Widgets, cutscenes, ai, scripts, everything works as it should(other than what i've mentioned above)


EDIT: If anyone would be so kind, I'd really appreciate a rip of the original campaign bsps from Halo PC using Sprinkle's real time editor; it wont run on my 32 bit machines. I should be able to fix the shadowless bipeds issue by comparing tags like giraffe said. Help me finish this guys, we're so close to having perfect tag extraction! When Refinery hits version 1.0 it'll mean I'm confident the extraction is 100% perfect(excluding super obscure bugs I would never think to test for).
Edited by MosesofEgypt on Jun 7, 2017 at 05:05 PM


Ki11erFTW
Joined: Jul 4, 2009

You've seen nothing yet.


Posted: Jun 7, 2017 01:58 AM    Msg. 17 of 76       
A10 extracted from Refinery:

https://www.mediafire.com/?mcim55dl1l228e8
Edited by Ki11erFTW on Jun 7, 2017 at 01:58 AM


MosesofEgypt
Joined: Apr 3, 2013


Posted: Jun 8, 2017 03:14 PM    Msg. 18 of 76       
Everyone should re-download the mek and run the updater because i just fixed all the issues i talked about above. As far as i can tell, there are only some bugs ripping OS tags. When those are fixed, refinery will hit version 1.0 and be considered capable of perfectly ripping tags. As of right now i believe it can perfectly rip non-OS tags.

P.S. Someone leave a comment so i can post more updates.
Edited by MosesofEgypt on Jun 8, 2017 at 03:15 PM


R93_Sniper
Joined: Feb 13, 2011

When in Doubt, RUN!


Posted: Jun 8, 2017 04:31 PM    Msg. 19 of 76       
Leaving comment so Moses can do his thing


Vergil
Joined: Jun 13, 2011

you're just mad cuz you're angry


Posted: Jun 8, 2017 04:52 PM    Msg. 20 of 76       
Windows XP pictures for a Halo Tool?

Brings me back.


MosesofEgypt
Joined: Apr 3, 2013


Posted: Jun 10, 2017 04:40 AM    Msg. 21 of 76       
Just finished playing through a10, a30, a50, b30, and b40 on normal. No bugs to report. I sent all 909MB of perfectly(as far as I can tell) ripped original campaign tags to killa so he can upload them for anyone who wants them. Someone should take the tags, fix the covenant shield door shaders(they look like garbage on pc) and restore all the missing multipurpose bitmaps and recompile them into better pc ports of the maps. We can rip the original bitmaps from all the tags that shipped with the hek using TeXource, so we can make all the weapons look as good as possible. I won't be doing anything halo related for quite a while, so consider this sorta a parting gift for the time being. Refinery will remain at v0.9.0 until I'm sure there are no ripping bugs(which you guys will need to notify me of). Cheers!
Edited by MosesofEgypt on Jun 10, 2017 at 04:42 AM


ghost901
Joined: Mar 4, 2010


Posted: Jun 10, 2017 04:56 AM    Msg. 22 of 76       
Ran into the energy sword bug.


All tags are stock extracted with Refinery.


It seems to occur on that one instance.
Edited by ghost901 on Jun 10, 2017 at 06:00 AM


lolslayer
Joined: Mar 21, 2015

https://www.youtube.com/watch?v=uMHbAKvPJkU


Posted: Jun 10, 2017 07:23 AM    Msg. 23 of 76       
For how long will "quite a while" be?


MosesofEgypt
Joined: Apr 3, 2013


Posted: Jun 10, 2017 02:08 PM    Msg. 24 of 76       
Likely months. Gonna try helping my friend with a game he's working called fantasy crescendo. I'm not a touhou fan, but he needs some help. Also, that bug i believe is caused by my definition not having the proper enum sizes for certain fields in the glow tag. Tweaking stuff now, though my pc just crashed lol. Should be ready for you to run the mek updater to get the fix in like, an hour. I originally saw that and in my tired stupor thought "eh, i think i remember it looking stupidly big like that" lol. So what are you thinking about the extraction overall though? I think Refinery beats HEK+ at this point lol.

EDIT: Actually, looks like my definition is correct as far as sizes go. Only thing that wasn't was the value for the NONE enum setting, which should have been -1 instead of 0. I'm checking halo pc to see if the glow is that stupidly big on there too, cause I don't see anything wrong with either of the sword's glow tags.


EDIT2: Okay, yeah, wow. So i'm guessing that glow is one of their oldest tags, cause the attachment enumerators aren't just byteswapped when they're converted to meta, they subtract 1 from them. The enum values in the tag are the same as you'd see throughout the rest of the library wherever you see function_input:
NONE=0, a_out=1, b_out=2, c_out=3, d_out=4
When converted to meta though, the glow tag uses these values:
NONE=-1, a_out=0, b_out=1, c_out=2, d_out=3

I tested this by looking at what the energy swords values got converted to when I extracted it. All values were -1, but when it got converted to meta in the recompiled CE map they were all set to -2, which isn't even a valid setting. This basically means I just need to add 1 to each of the values when I extract the meta.


EDIT3: Just tested ingame, and that fixed the bug. Turns out those 2 energy sword glow tags are the only ones in the entire campaign, and not every map has the energy sword in it. In any case, download refinery from the mek to get the fix. Version is bumped to 0.9.1.
Edited by MosesofEgypt on Jun 10, 2017 at 06:03 PM


lolslayer
Joined: Mar 21, 2015

https://www.youtube.com/watch?v=uMHbAKvPJkU


Posted: Jun 11, 2017 10:08 AM    Msg. 25 of 76       
We'll miss you moses :(


Sod
Joined: Apr 10, 2017


Posted: Jun 20, 2017 12:19 PM    Msg. 26 of 76       
I'm running MEK_Installer and have selected a place to install to - C:\Users\User\Desktop\MosesofEgyptmek
and I've tried C:\Program Files\mek

But I keep receiving the same errors in the MEK Installer

--------------------------------------------------------------------------------
pip install mozzarilla
Install failed.

--------------------------------------------------------------------------------
pip install arbytmap
Install failed.

---------- Finished ----------

I downloaded Python 3.6.1 and selected "Add python to PATH".

Sorry if I'm messing up something incredibly stupid, I'm a complete noob at all this.


MosesofEgypt
Joined: Apr 3, 2013


Posted: Jun 20, 2017 01:17 PM    Msg. 27 of 76       
Sorry, it's my bad. Something's wrong with the portable install, so I just now pushed an update that comments out the buttons that allow you do so it. For now, don't select the install location and dont check "Portable install". The portable install might work for some people(worked for me) but it's not for some others, and I'm not sure why.


Super Flanker
Joined: Oct 5, 2012

The length of your life depends on my aim.


Posted: Jun 21, 2017 05:08 PM    Msg. 28 of 76       
👌


Sod
Joined: Apr 10, 2017


Posted: Jun 24, 2017 01:13 AM    Msg. 29 of 76       
Quote: --- Original message by: MosesofEgypt
Sorry, it's my bad. Something's wrong with the portable install, so I just now pushed an update that comments out the buttons that allow you do so it. For now, don't select the install location and dont check "Portable install". The portable install might work for some people(worked for me) but it's not for some others, and I'm not sure why.


Thanks for the help :D


MosesofEgypt
Joined: Apr 3, 2013


Posted: Jul 16, 2017 01:21 AM    Msg. 30 of 76       
Refinery is updated with partial zteam tag deprotection. Click the deprotection button to start it. Currently it can de-mangle most of a maps tags, making them extractable. I'm not sure how the rest of the tags are referenced to where Halo can find everything it needs(even with mangled class fourcc's), but if I find out I'll be able to implement deprotecting most of the rest of the tags. There are certain types of tags that can't be deprotected though, such as tag_collection and ui_widget_collection tags. Since those types of tags aren't needed once in a map, the protection makes them irreparable.

I tested by deprotecting selentic's Revelations map, and it gets it about 80% deprotected. This zteam deprotection was helped and inspired by 002's work on Deathstar Deprotector. The main differences are that mine deprotects tiny bit more of the classes, prints info about what couldnt be deprotected, and can work with open sauce maps. Can't thank 002 enough for the help though. I wouldn't have put the effort into it if I had to figure it out myself with how much time it would have taken to figure out some of the more tricky parts.

All that said, I can't guarantee that the deprotection is bug free. If I typed an offset wrong, Refinery could be overwriting the wrong areas of the map. I haven't extensively tested it, so I'm very open to feedback right now. It should be relatively bug free though, since me typing in a wrong offset will usually result in a seek error that would have been caught while I was programming.

Re-download refinery from the MEK and run the upgrader.
Edited by MosesofEgypt on Jul 16, 2017 at 01:22 AM


Isxz
Joined: Apr 14, 2013

MrChromed


Posted: Jul 19, 2017 04:27 PM    Msg. 31 of 76       
This is gonna be a weird question, but: can I use this program in a virtual machine? Since for some reason, Python doesn't install correctly in my PC, I thought if it was possible to use this with VirtualBox, running a 32-bits OS like XP or Seven.


MosesofEgypt
Joined: Apr 3, 2013


Posted: Jul 19, 2017 05:39 PM    Msg. 32 of 76       
I have no experience with virtual machines, but i dont see why not. Tbh I think you might just want to save yourself the hassle in the long run and try to debug the install problem if you haven't already. Maybe try redirecting it to install to your C drive root rather than appdata. It's what i did with mine.


ghost901
Joined: Mar 4, 2010


Posted: Jul 23, 2017 06:45 PM    Msg. 33 of 76       
Is there something I'm doing wrong here?


MosesofEgypt
Joined: Apr 3, 2013


Posted: Jul 23, 2017 07:06 PM    Msg. 34 of 76       
Run the mek_installer and tell it to update. That'll fix it. If it doesnt, clear your python pip cache and run it again.


ghost901
Joined: Mar 4, 2010


Posted: Jul 23, 2017 07:32 PM    Msg. 35 of 76       
Quote: --- Original message by: MosesofEgypt
Run the mek_installer and tell it to update. That'll fix it. If it doesnt, clear your python pip cache and run it again.

It's still not working I'm afraid.

 
Page 1 of 3 Go to page: · [1] · 2 · 3 · Next

 
Previous Older Thread    Next newer Thread





Time: Tue October 17, 2017 5:32 AM 157 ms.
A Halo Maps Website