A Community discussion forum for Halo Custom Edition, Halo 2 Vista, Portal and Halo Machinima

Home  Search Register  Login Member ListRecent Posts
  
 
»Forums Index »Halo Custom Edition (Bungie/Gearbox) »Halo CE General Discussion »Maximum Halo Script File (.hsc) Size in bytes

Author Topic: Maximum Halo Script File (.hsc) Size in bytes (17 messages, Page 1 of 1)
Moderators: Dennis

sparky
Joined: Jun 27, 2009


Posted: Apr 17, 2017 10:08 PM    Msg. 1 of 17       
A scenario can have each comment block be up to 8192 bytes including the invisible null terminator byte. How many bytes can an .hsc file be? Does it have the same 8 KB file size limit?

I'm adding some string usage types to facilitate data checking and I have these so far:



I am wondering if anyone can help by simply telling me if an .hsc file has a maximum size in bytes that it can be in order to be compiled into a scenario. The reason I am asking is because Blam scripting is the one area of Halo with which I have the very least experience. I have never compiled a working script into a scenario and never used a script that was compiled into a map file.


EmmanuelCD
Joined: Jan 7, 2015

Using fox avatar since 1893


Posted: Apr 17, 2017 10:29 PM    Msg. 2 of 17       
Not sure, my largest hsc was about 5 kb.


sparky
Joined: Jun 27, 2009


Posted: Apr 17, 2017 11:18 PM    Msg. 3 of 17       
Since you know how to do it, could you please put some dummy data into an .hsc and compile it into a scenario?


DeadHamster
Joined: Jun 8, 2014


Posted: Apr 17, 2017 11:24 PM    Msg. 4 of 17       
Quote: --- Original message by: sparky
How many bytes can an .hsc file be? Does it have the same 8 KB file size limit?]


No.





MosesofEgypt
Joined: Apr 3, 2013


Posted: Apr 18, 2017 12:37 AM    Msg. 5 of 17       
You should check the Reclaimer library that mozz relies on. I've put all the hard limits in for all rawdata structs for both regular halo and open sauce(though I admit I somehow typed the max_size for the comment data as 2x larger).


recorded_animation = Struct("recorded animation",
ascii_str32("name"),
SInt8("version"),
SInt8("raw animation data"),
SInt8("unit control data version"),
Pad(1),
BSInt16("length of animation", SIDETIP="ticks"), # ticks
Pad(6),
rawdata_ref("recorded animation event stream", max_size=2097152),
SIZE=64
)

source_file = Struct("source_file",
ascii_str32("source name"),
rawdata_ref("source", max_size=262144, widget=HaloScriptSourceFrame),
SIZE=52
)

rawdata_ref("scenario editor data", max_size=65536)
rawdata_ref("script syntax data", max_size=380076)
rawdata_ref("script string data", max_size=262144)


for open sauce the script syntax and string data is increased

rawdata_ref("script syntax data", max_size=570076)
rawdata_ref("script string data", max_size=393216)

Edited by MosesofEgypt on Apr 18, 2017 at 12:40 AM


Masters1337
Joined: Mar 5, 2006

halonaminator's unfortunate idol


Posted: Apr 18, 2017 08:43 AM    Msg. 6 of 17       
I have a few that are so large I have to remove comments in order to get sapien to read them. I can send them over to you if you wish.


sparky
Joined: Jun 27, 2009


Posted: Apr 18, 2017 10:46 AM    Msg. 7 of 17       
Does the game engine recognize Unix line endings, or are CRLF Windows line endings necessary? I know that Guerilla does not interpret Unix line endings but interprets Windows line endings. If you could convert the line endings with a program such as Notepad++, you would use fewer bytes; this would be helpful if you want it to remain legible and if the Lua syntax that Blam uses does not need CRLF line endings.

Masters1337: Please test the file size of an .hsc script file or let me know how to do it: do I just add the script file to a "scripts" folder in the scenario's tags subdirectory and issue the "compile scripts" command from within Sapien? I'll be testing the limit as 1 << 18, which is 0x00040000, which is 262144 as maximum bytes, and I will be using Lua comments to fill the data.

EDIT: Kirby's scripting tutorial PDF says to use the corresponding subdirectory of data. I'll test this and report back, thanks for the help.
Edited by sparky on Apr 18, 2017 at 11:17 AM

256 KB is the maximum .hsc script file size. (262144 bytes). If it is 262145 bytes, you get the error in Sapien, "Maximum source file size exceeded."
Edited by sparky on Apr 19, 2017 at 08:46 AM

There are some caveats.

You can have several scripts (sapien and "a hobo" will compile up to 6 .hsc script files in a directory -- it will silently ignore additional script files after the first 6 files), and the total sum bytes of all scripts compiled must be at most 256KB (262144 bytes). The scenario will be able to be opened by Guerilla. Kornman's version of Guerilla will not open a scenario unless every script file used has at most 8192 bytes, because Kornman's version of Guerilla displays the script text in a text box similar to the comment text box and those text boxes only support showing at most 8192 bytes -- so Kornman's version of Guerilla will crash if you try to open a scenario tag that has a script file over 8192 bytes. But it will open fine in the default Guerilla.
Edited by sparky on Apr 19, 2017 at 05:16 PM

Line endings don't matter to the syntax (although I haven't tested for situations where it might) and Kornman's version of Guerilla does not show line endings properly unless they are Windows CRLF format. Changing to Unix line endings will save some bytes.
Edited by sparky on Apr 19, 2017 at 05:20 PM

---

OK so further testing reveals that MosesOfEgypt was correct: the maximum size of the Comments data in .scenario is 16384 bytes.
Otherwise, you get:
tag data 'editor_comment_data_definition' too large.

When you compile using tool.exe.
However, an additional limitation is imposed by Guerilla: Guerilla will crash if you try to open any (probably not exclusively .scenario) tag where it needs to load more than 8192 bytes in a text block. Since Kornman's version of Guerilla shows the script information in a text block, that means the same limitation is given for script data with that program.

  • Maximum Comment Data: 0x4000 (1<<14) (16384) bytes (16KB)
  • Maximum Script Data for Sum of All Scripts (that Sapien or a hobo will compile): 0x40000 (1<<18) (262144) bytes (256KB)
  • Maximum Script Files that Sapien or a hobo will recognize per directory: 6
  • Maximum Multi-line Text Box Data that Guerilla and Kornman's Guerilla will permit without crashing: 0x2000 (1<<13) (8192) bytes (8KB)


So you can see that the interface imposes artificial limitations on the size of data. Another reason why there needs to be an improvement to the HEK.
Edited by sparky on Apr 19, 2017 at 06:43 PM

Also note that Kornman's version of Guerilla "enables" 512 total entries in the Scripts block and 8 total entries in the Source Files block of a scenario tag.
Edited by sparky on Apr 19, 2017 at 06:52 PM

So, in Guerilla's interface...
  • Normal text fields (for 1 byte, 2 byte, and 4 byte numbers) are limited to 30,000 characters. Guerilla prevents additional text from being entered.
  • String text fields (for labels) are limited to 31 characters. Guerilla prevents additional text from being entered. These always store 32 bytes.
  • String text fields (for tag pathname strings) are limited to 254 characters. Guerilla crashes if additional text is entered. These store up to 255 bytes (presumably, especially given some directory path length limitations in Windows XP).
  • String multi-line text fields (for comments in .scenario and -- in Korman's version -- for script text in .scenario) are limited to 8191 characters. Guerilla crashes if additional text is entered. These typically store many more bytes, as previously described.
  • String multi-line text fields (for .string_list tags) are limited to 30000 characters. Guerilla prevents additional text from being entered. These store up to 4096 bytes; Tool will not compile extra bytes and Guerilla will not save additional bytes.


Related tests:

"tool strings" will compile "ascii/multibyte" strings (ASCII or UTF-8) in this format:
hi this is string A
###END-STRING###
hi this is string B
###END-STRING###

As long as each string entry is at most 8191 characters, which in this case translates to 8192 bytes. Actually, it will say that it compiled properly, but in reality, it will not compile the strings into the tag unless they are each at most 4095 characters, which in this case translates to 4096 bytes.

"tool unicode-strings" will compile "16-bit unicode" strings (UTF-16/UCS-2 with Little Endian or Big Endian Byte Order Markings -- they result in the same little endian byte patterns, essentially as 1-byte characters including the string terminator with 1-byte terminators) in the same format as mentioned -- and will actually compile the string entries into the tag if they are each at most 16383 characters, which in this case translates to 32768 bytes, even though Tool will say that it successfully compiled a string that is at most 32767 characters which would have translated into 65536 bytes. Modifying the data to be more than 32768 bytes results in it being unable to be opened in Guerilla, with the error that the string data is too large.

In sum, .string_list entries can be up to 4096 bytes (4095 characters) and .unicode_string_list entries can be up to 32768 bytes (16383 characters). Tool falsely claims to compile entries twice those lengths: .string_list entries up to 8192 bytes (8191 characters) and .unicode_string_list entries up to 65536 bytes (32767 characters).

...which means that the Reclaimer source code is accurate regarding these limits.
Edited by sparky on Apr 19, 2017 at 09:12 PM


MosesofEgypt
Joined: Apr 3, 2013


Posted: Apr 19, 2017 10:48 PM    Msg. 8 of 17       
You should look at reclaimer for the max reflexive counts as well. I obtained all of my max sizes my either adding entries to guerilla till it capped, or if the limit was too high(around 128) I hex edited a reflexive count/rawdata size in a tag until it gave me an error about the size being too large. The nice thing is that if it decided the number was higher than the limit, it wouldnt even try to parse the rest of the file, so I could just keep tweaking the numbers without having to actually add to the tag. I even got the sbsp limits.



bsp = Struct("bsp",
reflexive("bsp3d nodes", bsp3d_node, 131072),
reflexive("planes", plane, 65535),
reflexive("leaves", leaf, 65535),
reflexive("bsp2d references", bsp2d_reference, 131072),
reflexive("bsp2d nodes", bsp2d_node, 65535),
reflexive("surfaces", surface, 131072),
reflexive("edges", edge, 262144),
reflexive("vertices", vertex, 131072),
SIZE=96
)

material = Struct("material",
[...snip...]
rawdata_ref("uncompressed vertices", max_size=4864000),
rawdata_ref("compressed vertices", max_size=2560000),
SIZE=256
)

detail_object = Struct("detail object",
reflexive("cells", detail_object_cell, 262144),
reflexive("instances", detail_object_instance, 2097152),
reflexive("counts", detail_object_count, 8388608),
reflexive("z reference vectors", detail_object_z_reference_vector, 262144),
SIZE=64
)

sbsp_body = Struct("tagdata",
[...snip...]
reflexive("collision materials", collision_material, 512, DYN_NAME_PATH='.shader.filepath'),
reflexive("collision bsp", collision_bsp, 1),
reflexive("nodes", node, 131072),
[...snip...]
reflexive("leaves", leaf, 65535),
reflexive("leaf surfaces", leaf_surface, 262144),
reflexive("surface", surface, 131072),
reflexive("lightmaps", lightmap, 128),

Pad(12),
reflexive("lens flares", lens_flare, 256, DYN_NAME_PATH='.shader.filepath'),
reflexive("lens flare markers", lens_flare_marker, 65535),
reflexive("clusters", cluster, 8192),
rawdata_ref("cluster data", max_size=65536),
reflexive("cluster portals", cluster_portal, 512),

Pad(12),
reflexive("breakable surfaces", breakable_surface, 256),
reflexive("fog planes", fog_plane, 32),
reflexive("fog regions", fog_region, 32),
reflexive("fog palettes", fog_palette, 32, DYN_NAME_PATH='.name'),

Pad(24),
reflexive("weather palettes", weather_palette, 32, DYN_NAME_PATH='.name'),
reflexive("weather polyhedras", weather_polyhedra, 32),

Pad(24),
reflexive("pathfinding surfaces", pathfinding_surface, 131072),
reflexive("pathfinding edges", pathfinding_edge, 262144),
reflexive("background sounds palette", background_sound_palette, 64, DYN_NAME_PATH='.name'),
reflexive("sound environments palette", sound_environment_palette, 64, DYN_NAME_PATH='.name'),
rawdata_ref("sound pas data", max_size=131072),

Pad(24),
reflexive("markers", marker, 1024, DYN_NAME_PATH='.name'),
reflexive("detail objects", detail_object, 1),
reflexive("runtime decals", runtime_decal, 6144),

Pad(12),
reflexive("leaf map leaves", leaf_map_leaf, 256),
reflexive("leaf map portals", leaf_map_portal, 524288),
SIZE=648,
)


sparky
Joined: Jun 27, 2009


Posted: Apr 20, 2017 08:53 AM    Msg. 9 of 17       
Are you going to revert this commit back to 16384? It does not seem right to be limited by Guerilla's interface.
https://bitbucket.org/Moses_of_Egypt/reclaimer/commits/a1242a39aa5252624955ee9783be7c626c949736?at=default

Thanks for the information; I will review it with my retribution.h.

EDIT: https://github.com/LiquidLightning/infernal/commit/e14090e9f0115cf2e4febaeb4656da45af32c27d
When I do my own tests, I'll let you know if any results differ from yours. Hopefully, there are no substantial inconsistencies among the other HEK programs. It's too bad that Guerilla's multi-line text field is limited to interpreting 8192 bytes and crashes if data is otherwise, even though more data is supported by the game engine. At least this is not the case with string_list tag text blocks but only with scenario tags' comment blocks and (with Kornman's Guerilla) also with script text blocks.

Here is what I have now with string types. I was thinking of doing what most other Halo editors have done, including Kornman with Prometheus, and specifying metadata groups like from/to. Kornman actually released the source code to Prometheus, but I've been waiting for him to say something on the Halo Maps Forum about this. Megasean shared the link on my Halo Discord server; I've been glancing through that to see what others have done in their approaches to reverse engineering. It seems like most of the text is generated from the HEK or game executables or files. I noticed several minor textual mistakes. Anyway...

https://github.com/LiquidLightning/winferno/blob/master/infernal/infDataTypes.cs

public enum InfDTStringUsage
{
tagclass, // 4 (fixed, no terminator) (4 characters)
profilename, // 24 (fixed, 11 unicode + 1 (interpreted?) unicode terminator) (1-11 characters)
label, // 32 (fixed with interpreted terminator) (0-31 characters)
tpns, // 255 (variable with terminator) (0-254 characters)
stringlist, // 4096 (variable ASCII or UTF-8 with terminator) (0-4095 characters)
unicodestringlist, // 32768 (variable 2-byte UTF-8 with terminator) (0-16383 characters)
comment, // 16384 (variable with terminator) (0-16383 characters;Guerilla and kornman's Guerilla multi-line text field will crash if over 8191 characters)
scriptsource // 262144 (variable with terminator) (0-262143 characters;kornman's Guerilla multi-line text field will crash if over 8191 characters)
}


Since these are new programs that are intended to replace Guerilla, it seems best for us to warn users about things that will prevent compatibility with Guerilla but to only provide limits according to Blam in general.
Edited by sparky on Apr 20, 2017 at 09:10 AM
Edited by sparky on Apr 20, 2017 at 10:56 AM
Edited by sparky on Apr 20, 2017 at 11:02 AM


Super Flanker
Joined: Oct 5, 2012

The length of your life depends on my aim.


Posted: Apr 20, 2017 09:16 AM    Msg. 10 of 17       
I have no idea what any of you just typed.....


sparky
Joined: Jun 27, 2009


Posted: Apr 20, 2017 11:05 AM    Msg. 11 of 17       
Do you think that it is worth doing a patch of Kornman's Guerilla, or doing a somewhat-inclusive patch of Guerilla to fix some of these interface problems? Or is that not worth it? It would seem to be a rather simple step for now.

Most notably, this text clipping issue should be fixed:


Edited by sparky on Apr 20, 2017 at 11:09 AM


SOI_7
Joined: Mar 23, 2012

Tatsumaki best waifu


Posted: Apr 20, 2017 12:23 PM    Msg. 12 of 17       
Quote: --- Original message by: sparky

Do you think that it is worth doing a patch of Kornman's Guerilla, or doing a somewhat-inclusive patch of Guerilla to fix some of these interface problems? Or is that not worth it? It would seem to be a rather simple step for now.

Most notably, this text clipping issue should be fixed:

http://i.imgur.com/SnWGaVs.png
Edited by sparky on Apr 20, 2017 at 11:09 AM


You can actually fix that running Guerrilla in Windows Vista compatibility mode


MosesofEgypt
Joined: Apr 3, 2013


Posted: Apr 20, 2017 12:48 PM    Msg. 13 of 17       
I'm not reverting the commit since I've already applied others after it, but I am overwriting the change it with a new commit. Also, I'm not sure it's worth it to tweak Korns Guerilla. I just don't really see a big benefit in doing so when Mozz exists and your editor will too. If you want to fix that stuff, go right ahead. I wouldn't even know where to begin.


sparky
Joined: Jun 27, 2009


Posted: Apr 20, 2017 04:01 PM    Msg. 14 of 17       
Quote: --- Original message by: SOI_7
Quote: --- Original message by: sparky

Do you think that it is worth doing a patch of Kornman's Guerilla, or doing a somewhat-inclusive patch of Guerilla to fix some of these interface problems? Or is that not worth it? It would seem to be a rather simple step for now.

Most notably, this text clipping issue should be fixed:

http://i.imgur.com/SnWGaVs.png
Edited by sparky on Apr 20, 2017 at 11:09 AM


You can actually fix that running Guerrilla in Windows Vista compatibility mode


I tried without success.


Caesar
Joined: Jul 1, 2013


Posted: Apr 20, 2017 05:52 PM    Msg. 15 of 17       
Sparky it has been almost five years and yet u still didnt show us anythang amazing.




U said there will be 32 players. U said that and dont lie.


Masters1337
Joined: Mar 5, 2006

halonaminator's unfortunate idol


Posted: Apr 20, 2017 07:59 PM    Msg. 16 of 17       
here is the hsc for SPV3 B30:e. adding a few extra lines to it even if commented out will result in sapien not be able to read it.

https://www.dropbox.com/s/iv4maeep9vbgiqe/spv3_b30_evolved.hsc?dl=0


Super Flanker
Joined: Oct 5, 2012

The length of your life depends on my aim.


Posted: Apr 21, 2017 10:06 AM    Msg. 17 of 17       
Quote: --- Original message by: Asters1337
here is the hsc for SPV3 B30:e. adding a few extra lines to it even if commented out will result in sapien not be able to read it.

https://www.dropbox.com/s/iv4maeep9vbgiqe/spv3_b30_evolved.hsc?dl=0

 

 
Previous Older Thread    Next newer Thread





Time: Sun June 25, 2017 2:58 PM 235 ms.
A Halo Maps Website