sparky has contributed to 1791 posts out of 462306 total posts
(.39%) in 2,859 days (.63 posts per day).
20 Most recent posts:
Much more has been written. Do you read books? If so, how long is the average book you read?
The exception in the DMCA applies.
Also, I think that reverse engineering is actually a requirement for normal use of the HEK. For example, there are a lot of threads on this forum that ask what a certain error means, and the information they gain from this forum to help them avoid that error is gained through some minor or major form of analytical reverse engineering. Even experimentation with different parameters and files is a form of reverse engineering, to see what the program works with and what it does not work with.
DeadHamster, I do not think Microsoft, Bungie, or Gearbox Software would sue someone over a 20-year-old antiquated game and its game engine that they no longer support or have any genuine interest in dealing with, especially when it falls under fair use through the DMCA's interoperability exception -- which is squarely what is happening here by making a new program that is interoperable with the HEK and/or Halo. Especially since the HEK is free and they wouldn't be losing any money anyway. To spend money on litigation when you are not even losing any money but rather having your product continue in lifespan while the defendant's actions are valid under fair use is simply asking for a losing battle and a public relations nightmare on a silver platter.
Edited by sparky on Apr 23, 2017 at 07:41 AM
Edited by sparky on Apr 23, 2017 at 07:46 AM
Quote: --- Original message by: MosesofEgypt
If you actually try to charge for modding utilities you make, you will lose my respect as a modder, programmer, and researcher. That's the last I have to say about the subject.
My argument is that people have the right to earn money for their work. If you see this as disrespectable, then how do you earn a living besides mooching off of others and acting as a leech in a socialist government where you are living as a dependent without any work initiative for your entire lifetime? Is not that instead the disrespectable manner? But if you want to do everything pro bono without any income for your own work, then try to do so, but that will not keep you fed, so do not slight my assertion that others have the right to be paid for their work.
Quote: --- Original message by: OrangeJuice
Raw uncompiled data art commissions can be a thing
Yes, those are artists providing artwork. Artists being paid for their work happens all the time.
Quote: --- Original message by: Super Flanker
So I'm not reading OP's op. Someone want to explain what sparky is scheming for now?
I am scheming for you to be able to earn money from your work. Whether that is via donations or via sales is the initiation of this discussion.
Quote: --- Original message by: il Duce Primo
I would love to see you sell anything first because I doubt you would. Even if you did sell anything it would probably be immaterial for Microsoft to care. Just sell your stuff if you think you can.
I do not plan illegitimate activities without first rationalizing that they are legitimate. I'm not doing something that I know is illegal without a good reason. As far as I am concerned, calling income donations or calling it sales is a mere label, but I am more interested in the premise specifically that the programmers and artists in this community are doing everything pro bono because they think they cannot earn money from their work, which I think is a false slight against professional standards and is disrespectful to major contributions of art.
From the consumer standpoint, if you are enjoying someone else's work so much, shouldn't you feel obligated to compensate them for that experience? Whether that is in the form of donations or sales, that is "six in one hand or half a dozen in the other hand".
If a map author uses the Halo Editing Kit to make tags or maps, they should not be selling their work, but donations are irrelevant to the discussion on the basis that they are entirely unconstrained as far as legal permission -- donations are like gifts/presents in that they are freely permissible in a limited quantity. But if you want to make a living off of your work, I am arguing that by using a non-Microsoft, non-Bungie, non-Gearbox program to do it, you are not actually doing anything wrong because you are not making money off of another person's intellectual property. I am saying that a separate program that supports HEK tag formats and .map file formats making it compatible with the HEK and Halo is the same as an independently-developed conversion utility, and that selling such a utility is permissible because it is completely different software from any other software. Reverse engineering is only relevant to studying file formats in my argument. Are the file format specifications patented? You cannot copyright sections of code, and patents are only for source code -- the means to arriving at the copyrighted finished result. In this situation, if there were any limitations to the study of software, it would need to specifically be as a patent naming specific behaviors of the software as patented and as undisclosed trade secrets, not simply copyrighting the resulting finished product which only stipulates an agreement for expected use of that finished product.
Most people only say that warranties for repair are voided by intentionally "breaking" something (opening it to see its parts). All that means is that the company is unwilling to repair something that you intentionally disassembled. I think arguments that say that you may not study something are founded upon deceit and are irrational, because any student of computer science can disprove that assertion with their comprehension of how computers work.
Anti-virus software modifies how other software works. Firewall software and hardware modify how other software works. Simply having a user permission system in any modern operating system modifies how other software works. When a program like Steam or other "launcher" spawns a program, it modifies how that program works, even though it does it in ways that they presume are acceptable to the users. Are you breaking the EULA of Halo by playing Halo in a virtual machine, or by typing into cmd.exe "start halo.exe", or by otherwise spawning the halo process through another program? These all change the behavior of the game, yet reverse engineering does not even change the behavior of the game but is intended to study how the game works in a more understandable way than your anti-virus software studies how it works.
So in sum, I think that:
- Reverse engineering is normal use of a program, and is for people who do not trust blind execution of software. It is the same as knowing how the Windows operating system works. Either you are told by someone else, or you learn how it works for yourself, or both.
- Legal documents denying that you have the right to understand how software works is dangerous, insecure, deceitful, pointless, unenforceable, and irrational to the end user who is simply trying to understand and use the software they purchased.
- Reverse engineering for study, education, and documentation is permissible and is common practice in every civilized field of study including engineering, music, and the various sciences. Reverse engineering by this definition of purpose is akin to thinking, and that natural right cannot be rationally legislated away. Using the knowledge of how a program works in order to make another program is the matter of concern: if another work is substantially equivalent (in programming, this means the source code because as a very simple example anyone can make a text editor and sell it as different software even though it might have substantially similar executable code but different source code), then it is considered a substantial part of the copyrighted or patented work.
- In the same way, making a different program and having its resulting files be compatible with Halo, whether by "chance" or by intent or by hard work and study of analyzing how these and other computer software programs work or by existing knowledge of how this and other software works simply through knowledge of computer programming and computer science, is legal and permissible in the same way that any other computer programmer makes a computer program with their own source code.
Doesn't this all make sense? Am I missing something, like the specific pattern of 64 bytes of header data being patented? Who in their right mind has the authority to tell you that you cannot study something and then implement your understanding in such a way that you design a new and different product? And if you design a different product, who in their right mind has the authority to tell you that you cannot profit from your work? I call all such things hypocrisy because this is how people do everything that they do.
Quote: --- Original message by: Dennis
Quote: --- Original message by: sparky
I would like to discuss potential for legally earning money for work in this game.
You can't. You do not own the intellectual property rights to the game or anything in it. You would have to license the rights to the game from Microsoft to "legally" make any money from it.
Yes, I am agreeing with this point: we cannot legally sell the results of our use of existing tools provided to us for the purpose of generating those results. Information can be sold; this is how schools work. Information can also be freely given; this is how life works. Microsoft has unfortunately indicated that it does not want to allow other companies to do anything with Halo. Gearbox and Bungie have apparently discontinued their interest in Halo Custom Edition and the Halo Editing Kit. Roger Wolfson appears to be the only person helping the community with his exclusive access to anything related to the game engine. Most or all of the research that enthusiasts like myself have done have been provided to others for free. But this work takes a lot of time and work (decades of it) and is all pro bono. It cannot be logistically maintained, as you know, without some kind of income to support it. That is the premise of my discussion: people should be allowed to sell their own work, regardless of their approach to education that resulted in their knowledge of how to do that work. If they educated themselves more, they will probably know better how to do the work they set out to do. But the work that they do -- as being a unique and deliberate "creation", whether it is artwork or software, even if it shares some similarities with other people's works -- should be recognized as a separate work, and they should therefore have all the rights to it because it is their own work. Remember my argument about text editors all basically doing the same thing and yet being individually coded and so therefore independently sold as different software products each with their own intellectual property rights? Isn't that the same thing as me making my own program that can read and write from and to file formats that Halo uses?
Edited by sparky on Apr 23, 2017 at 04:37 AM
It occurred to me from the start, as I was considering whether to discuss this matter publicly, that I could attempt to contact Microsoft in order to determine what they felt I could do. However, it is frankly no business or right of theirs to determine what I can or cannot do with my own work. Why should one business presume to assert that another business cannot do something with their own work (presuming the work is safe)? That is a lack of double standard and is completely inappropriate for most professional situations; or what should a user of software do if they do not like what the software is doing on their own computer? At most, in both scenarios, they should discuss a matter reasonably and if it is not resolved properly, discontinue association with that company. And by consumer protection standards, share the information publicly so that others can benefit from their knowledge.
In the same way, it is commonly understood that the HEK is not doing things properly, so others here and I have taken it upon ourselves to painstakingly study what the HEK does and to painstakingly publish that information for free (all of this is reverse engineering) and to painstakingly provide alternatives that do similar things but better through free modifications of existing software (like Kornman did with Guerilla) and through additions to existing software (like Ghost did with Bluestreak for 3DS Max) and through completely independent programs (lots of examples are available in that regard).
So if we are doing all this independent work pro bono, why can't we do it for some financial compensation? Aren't there plenty of examples where software developers do not own the file specs that they develop their software to support? Or is it expected of every software company that they give away all rights to their software to every other copyright holder whose file specification they support? Does it even matter if file specs are substantial or necessary aspects of a program? All of this is merely the interpretation of existing data on a user's hard drive. The same program that can rename a file of a certain data structure could also modify it like antivirus software; and all of that is interpreting the copyrighted or patented data on the hard drive, by this standard illegally! Or do operating system developers like Apple and Microsoft legally own every bit of data on a user's hard drive that they and others are altering by means of the operating system software and other software? Ownership does not coincide with compatibility.
But now as a matter of credibility, would you trust software with a license agreement that says that you have no right to run anti-virus and anti-malware and firewall software simultaneously as that software? Of course not! And yet these are effectively jailing and limiting and altering the behavior of that software -- literally breaking the end user license agreement which says you cannot modify the software or alter its behavior -- and yet no one is suing the user for using a firewall or anti-virus or anti-malware software! Why the lack of double standard for someone who is trying to understand how a program works and make other programs that are compatible with it? If I am doing something as routine as piping a connection over SSH, I am altering the behavior of a program and breaking its EULA. And should I be thrown into prison or sued by Apple or Microsoft for inadvertently looking at a tag file in one of the many free and commercially available hex editors for purchase online through Apple's and Microsoft's online app stores? Or maybe I should counter sue because they sold me illegal software! So if looking at the data on my hard drive is not illegal, then neither is reverse engineering illegal, because that is all reverse engineering is: looking at what data is. Or do car manufacturers suddenly not allow you to open the hood of the car to check on how the engine is working and to make sure everything is in place, especially when you first buy the car so you can see it is not a lemon? That is reverse engineering: looking at what data is. The entire argument against reverse engineering is moot; rather, the argument which is valid is against breaking copyright through creating a work that is substantially duplicate without also being purposefully substantially different. So I claim that it only makes sense to let people improve things that have already been done, which is the entire basis of the US patent system: to promote innovation and creativity by looking at what has already been done and doing something continuously better -- and this is the nature of competition that these EULAs are written to avoid! And yet the software of these EULAs use shared libraries and work that others have provided for sale or for free! How can they claim to deny others what they themselves have done: understood and used and improved upon existing copyrighted work?
Professors state what has been done. Lawyers themselves state what has been done. People speak of their own experiences and of the experiences of others who shared with them descriptions of their experiences. All of this is effectively collaboration and a publication of information -- and that can only happen if the person is a liar or if they have instead studied the subject matter themselves through analysis and reverse engineering (that is, looking at and thinking about) whatever the subject matter may be.
Finally, there is an exception in the Digital Millennium Copyright Act which allows for reverse engineering to achieve interoperability with an independently created computer program. Our reverse engineering work in this game is merely for interoperability and to improve upon the software which, like its titles Guerilla, Sapien and Tool imply, are too primitive or unrefined for public use. If it entered into a court proceeding, I think that programs that are independently designed to work with the HEK and Halo, or replace the HEK, would be fine, as long as the work is not hijacking any existing commercial services like Blizzard had with exclusively hosting its networked games. When the Gamespy service was discontinued, I do not think anyone sued the community or Roger Wolfson for updating the game and dedicated server so that it could still run without Gamespy. In the same way, if the software provided is broken and discontinued (and I think it really is, as is obvious by this Halo Maps Forum), then it is appropriate to fix it ourselves or make a replacement program to use and maintain instead.
Edited by sparky on Apr 23, 2017 at 06:58 AM
Quote: --- Original message by: DeadHamster
You would have no copyrights to a program if it's only use is modding another's game [...]
So what is the purpose of win32.dll? Also, as an example, I think Steam is copyrighted by Valve. Steam's only use is in modifying the presentation and organization of other games. It's not a game by itself, it's a separately copyrighted program. So there is some precedent already that you can have as examples indicating that your own individual work is your own copyright. Feel free to argue a more specific definition and purpose for Steam.
The entire premise of the Internet and computer science and history and art has shown that improvements are made first by analyzing what has already been done and working to do something better. Mozart, Beethoven, Scarlatti, Bach and every composer who has studied music has first rewritten scores that other composers wrote in their studies of how to compose music.
Do not presume that new laws can or should disband common practices that have founded civilization that they are designed to support.
Thanks for pointing out the improbability of sales for work in this community; that needed to be said. I don't think it's worth debating though because it is at heart speculation about what might happen with sales. If I were to sell something, it would be for cheaper than the game costs. I have in the past planned to support donations, and I just want to form a premise that if donations occur for user work using a non-Microsoft/Bungie/Gearbox program to design that work, then Microsoft, Bungie, or Gearbox have no grounds to claiming any of that money for themselves. I think that's where this discussion is headed as far as what I am planning.
Edited by sparky on Apr 22, 2017 at 01:17 PM
My premise for reverse engineering being valid I think you can understand as being for the purpose of study and improvement, which for our purposes means making a program that is compatible with other people's work.
No one needs permission to make their own program compatible with another program that someone else copyrighted.
.map file format is not licensed. HEK tag formats are not licensed. The actual data structures of the copyrighted software are not patented. I'm guessing about the lack of patent. But even if they were patented, that should not matter if I am involved in reverse engineering -- either using my own reverse engineering work or someone else's. Patents themselves are provided in order to publish specifications. In other words, the entire patent and trademark system is to make something public knowledge with attribution.
And I already explained regarding why copyright notice should be given -- just like guerilla.exe contains copyright notices for the libpng, JPEG and other third-party software it uses (licensed or not), so attribution credit should be maintained along the chain of use in whatever is designed as a normal standard practice -- when feasible.
Edited by sparky on Apr 22, 2017 at 12:55 PM
Edited by sparky on Apr 22, 2017 at 01:00 PM
You did not disprove my argument.
Regarding what profession I will study, that's my business and not yours, and because you do not know me personally, I merely take your recommendation as sarcasm rather than a professional recommendation.
Regarding legal profession, you cite Dennis and that is fine for this forum because Dennis owns this forum. But Dennis is not a lawyer either. If you want to defer to a lawyer, hire a lawyer and present their professional analysis for public review. But I will tell you that lawyers only say what others have done, which is backwards and akin to the assertion that all advancement is history, a blatantly untrue implied assertion. If you want to improve, you will not do so by repeating history. So even a lawyer's professional analysis I would seek to criticize with logic and reason.
I think in general you are not understanding my rationales, so perhaps there are some ways I can clarify. Let me know if you have any questions about what I said. But I'd like to not argue logic with someone who does not understand what I am saying in the first place. Does this make reasonable sense to you?
Part of my argument is that it is illegal to demand unenforceable rules and also illegal to demand rules inappropriate for normal use of something. I will also say that to lack willingness to discuss reason and logic of a rule is to acquiesce to dictatorship and its full, mindless force.
Edited by sparky on Apr 22, 2017 at 12:49 PM
EDIT: It applies to religion also. It is written that God said, "I do nothing without reason."
Edited by sparky on Apr 22, 2017 at 01:37 PM
If you didn't read what I said, how can you presume to tell me what I said?
Are you illiterate or simply ignorant? You are not even trying to spell complete English words.
If you want to be understood in a language, use the language properly.
Now if you do not want to say anything worthwhile, then for the sake of this forum being worth people's time, go find a link to the Halo EULA and the Halo Editing Kit EULA and edit your post with it. Otherwise, come up with your own ideas in finding something useful to contribute to this discussion, because you literally wasted your time with a pointless insult that was just thrown back in your face. What do you honestly think you are accomplishing by this? If you are illiterate or otherwise do not want to write in a public forum of words, go away.
Edited by sparky on Apr 22, 2017 at 12:30 PM
I would like to discuss potential for legally earning money for work in this game. I will be discussing this over a period of time with different communities.
I categorize work in this game as:
- modding (modders designing mods)
- programming (programmers designing programs)
- asset development (artists designing assets)
By "this game", I mean Halo with the stock Blam! Scripting Engine.
Microsoft wants to retain property of all the commercially-available and default game content. Microsoft has in practice and written terms neglected responsibility for maintenance of the property, which by all reasonable standards constitutes effective disownership of their desired property. Yet out of respect for their wishes, or at least to not offend them, this discussion should presume to respect that Microsoft maintains and owns Halo with the Blam! Scripting Engine which is inclusive of all the commercially-available and default game content, hereafter referred to as the "stock assets".
The stock assets include everything that comes on the Halo PC CD purchased from Microsoft, and everything else that is purchased from that company specifically that was reworked by Gearbox Software: all the map files dated from 2003, the halo.exe game, and similarly everything belonging to Gearbox Software specifically designed by Gearbox Software -- each company owning the direct results of its own labor.
For the sake of argument, the Halo Editing Kit, including Guerilla, Sapien, and tool.exe, will be considered property of Bungie Software and therefore also property of Gearbox Software and/or Microsoft -- in other words, it is considered part of the Halo Custom Edition bundle of ownership.
Artists design their own content or modify existing content to make it available for use with the game. The owner of the game only owns the stock assets, not the third-party work designed by artists. Even the Halo game uses work copyrighted by others not affiliated with Bungie or Microsoft. Therefore, additional work provided to the game by unaffiliated artists should also have their own copyright notices, just as Halo includes copyright notices for the work of others that it includes. Here are some examples from guerilla.exe:
Quote: deflate 1.1.3 Copyright 1995-1998 Jean-loup Gailly
Quote: inflate 1.1.3 Copyright 1995-1998 Mark Adler
Quote: [JPEG] Copyright (C) 1998, Thomas G. Lane
Quote: libpng version 1.0.5 - October 15, 1999
Copyright (c) 1995, 1996 Guy Eric Schalnat, Group 42, Inc.
Copyright (c) 1996, 1997 Andreas Dilger
Copyright (c) 1998, 1999 Glenn Randers-Pehrson
Quote: Guerilla Version 1.0
Copyright (C) 1998
Bungie Software Products, Inc.
The Halo Editing Kit and Halo Custom Edition were provided for free with the expectation that they would be available exclusively to those who have purchased Halo PC and could supply the included CD Key during installation of Halo Custom Edition. The Halo Editing Kit specifically does not require a CD Key, so for practical and reasonable purposes is provided entirely for free by Gearbox Software through Bungie Software who developed it. Property of the Halo Editing Kit remains with Bungie Software and Gearbox Software. There is no implicit or explicit association of monetary gain from using or having access to the Halo Editing Kit, so for purposes of argument, it will be agreed "freely you have received, so freely give" so that all results of the Halo Editing Kit should be freely given without sale.
This means that artists designing their own work maintain ownership of their work when it is converted to be used in the game using the Halo Editing Kit. If you use the Halo Editing Kit to convert assets into tags or maps, you should include copyright information of those assets.
Now let us discuss what should be done if the artist, modder, or programmer uses a program that was not included with the Halo Editing Kit. The copyright of each work belongs to its owner and maintainer: Halo with Microsoft, HEK with Bungie and Gearbox, and so on.
For the sake of argument, reverse engineering is legal and normal use for everyone; for the sake of argument, it is acceptable to study programs in the various computer science fields just as it is acceptable to study the human body in the various medical fields. And for the sake of argument, all such information will be presumed freely accessible as a result of the work of individuals willing and able to studying that for the benefit of public knowledge, hereafter referred to as "public knowledge".
For the sake of argument, all available computer materials, whether made available for free or for sale, and whether given in conjunction with a legal agreement or without a specified legal agreement (such as commonly with open source software), is considered "public knowledge" because of the reasonable legality of reverse engineering.
For the sake of argument, the legality of reverse engineering is based upon intent to study and improve, not based upon an intent to duplicate in whole. Rebranding is irrelevant to this argument, because substantial changes means lack of duplication in whole.
If I design my own application program that is based upon my research and/or contributions to this public knowledge, I am validly designing a different application program that is my own property. As a sculptor studies a structure and carves their own art from their perception of it, to their own ability, so a computer programmer takes what information they can research and applies it to their own application programs. Just as a sculptor has the right to sell his sculpture, so does a computer programmer have the right to sell his application program.
Therefore, I can reverse engineer the Halo Editing Kit on account of reverse engineering as public knowledge, I can make a computer application program to replace it, and I can offer that computer application program for sale. I can alternatively make a computer program that can be used to design assets to be used with the Halo Editing Kit, and I can sell that program -- making it a more suitable alternative to any other program that lacks integration with the Blam! Scripting Engine, such as 3DS Max, Blender, Photoshop, GIMP, or Audacity.
Modders who would otherwise be making .map files through the Halo Editing Kit only for free could use my commercially-available or freely-available computer application program to make and sell .map files because they would not be bound by the Halo Editing Kit license. Just as if Autodesk required all developed content to be free and Blender did not have that requirement, and therefore anyone using 3DS Max had to give away their .obj work whereas Blender users could sell their .obj work, so anyone using a program is bound by the terms of its use.
The argument could be made that the .map file specification is included in the copyrighted work belonging to Microsoft. However, I will note that the Halo Editing Kit generates those .map files and the knowledge of generating a file format is included in computer science and is public knowledge. Price is irrelevant -- the HEK is free and Halo PC is commercial and yet both use the .map file format -- and intellectual property is irrelevant -- the HEK is provided for free by Bungie and Gearbox Software and Halo PC is provided for sale by Microsoft -- and actual obtaining of the information is irrelevant on account of reverse engineering being a fundamental computer science activity, as fundamental as anatomical studies are to biology. As I said already, the only stipulation to reverse engineering for the purpose of study and improvement is that the availability of the information from those studies becomes public knowledge for the advancement of the field of computer programming and computer science; and that is effectively superior to the availability of private instruction among graduate studies. That does not mean that all computer application programs are free, or that demand to provide studies is valid. It is the responsibility of the student to determine whether they should publish their research.
But the argument stands that if you do the work, you deserve credit for your work, and that includes artwork, modding work, programming work, reverse engineering work, publication work, and any other contributions to the computer-related field of study. It is hypocritical to assert Microsoft is the only contributor among the various Halo communities of unemployed contributors.
I think that by replacing the HEK, we can sell our own applications and the results of our own applications. The fact that they are compatible with Halo just makes them superior to any other program that is incompatible with Halo. Without using the IP of the HEK or of Halo in art or through using the HEK or Halo to generate works compatible with them, there is no legal limitation hindering use of our own work.
matt is bad.
Code too clever
Every string in guerilla.exe
I do not guarantee a lack of non-text data. I also added my own comments in brackets .
Apparently, Guerilla was made by Chucky.
Edited by sparky on Apr 20, 2017 at 06:07 PM
Who is Chucky?
Alan is referring to the current Halo development team as listed by Matt Soell. Namely:
Jason Jones - Project Lead
Konrad Shirinian - Network code
Chucky Gough - Physics
Matt Segur - Programmer
Paul Russell - Art
Rob McLees - Art
Shikai Wang - Art
Marcus Lehto - Art
Jamie Griesemer - Design
Stefan Sinclair - Networking (Bungie.net III)
Thanks to all the people who wrote in pointing out that the Charles Gough in the Wanted DOA pic is in fact Halo's Physics Programmer Extraordinare Charles Gough, better know as Chucky. The picture was taken shortly before he absconded to Mexico with the Halo beta.
Edited by sparky on Apr 20, 2017 at 06:14 PM
I wonder if he faced any guerilla warfare in Mexico...
Edited by sparky on Apr 20, 2017 at 06:16 PM
Quote: --- Original message by: SOI_7 Quote: --- Original message by: sparky
Do you think that it is worth doing a patch of Kornman's Guerilla, or doing a somewhat-inclusive patch of Guerilla to fix some of these interface problems? Or is that not worth it? It would seem to be a rather simple step for now.
Most notably, this text clipping issue should be fixed: http://i.imgur.com/SnWGaVs.png Edited by sparky on Apr 20, 2017 at 11:09 AM
You can actually fix that running Guerrilla in Windows Vista compatibility mode
I tried without success.
Do you think that it is worth doing a patch of Kornman's Guerilla, or doing a somewhat-inclusive patch of Guerilla to fix some of these interface problems? Or is that not worth it? It would seem to be a rather simple step for now.
Most notably, this text clipping issue should be fixed:
Edited by sparky on Apr 20, 2017 at 11:09 AM
Are you going to revert this commit back to 16384? It does not seem right to be limited by Guerilla's interface.
Thanks for the information; I will review it with my retribution.h.
When I do my own tests, I'll let you know if any results differ from yours. Hopefully, there are no substantial inconsistencies among the other HEK programs. It's too bad that Guerilla's multi-line text field is limited to interpreting 8192 bytes and crashes if data is otherwise, even though more data is supported by the game engine. At least this is not the case with string_list tag text blocks but only with scenario tags' comment blocks and (with Kornman's Guerilla) also with script text blocks.
Here is what I have now with string types. I was thinking of doing what most other Halo editors have done, including Kornman with Prometheus, and specifying metadata groups like from/to. Kornman actually released the source code to Prometheus, but I've been waiting for him to say something on the Halo Maps Forum about this. Megasean shared the link on my Halo Discord server; I've been glancing through that to see what others have done in their approaches to reverse engineering. It seems like most of the text is generated from the HEK or game executables or files. I noticed several minor textual mistakes. Anyway...
public enum InfDTStringUsage
tagclass, // 4 (fixed, no terminator) (4 characters)
profilename, // 24 (fixed, 11 unicode + 1 (interpreted?) unicode terminator) (1-11 characters)
label, // 32 (fixed with interpreted terminator) (0-31 characters)
tpns, // 255 (variable with terminator) (0-254 characters)
stringlist, // 4096 (variable ASCII or UTF-8 with terminator) (0-4095 characters)
unicodestringlist, // 32768 (variable 2-byte UTF-8 with terminator) (0-16383 characters)
comment, // 16384 (variable with terminator) (0-16383 characters;Guerilla and kornman's Guerilla multi-line text field will crash if over 8191 characters)
scriptsource // 262144 (variable with terminator) (0-262143 characters;kornman's Guerilla multi-line text field will crash if over 8191 characters)
Since these are new programs that are intended to replace Guerilla, it seems best for us to warn users about things that will prevent compatibility with Guerilla but to only provide limits according to Blam in general.
Edited by sparky on Apr 20, 2017 at 09:10 AM
Edited by sparky on Apr 20, 2017 at 10:56 AM
Edited by sparky on Apr 20, 2017 at 11:02 AM
Thanks for the review, DeadHamster.
Publicity for a functional resource is easy. One step at a time!
Does the game engine recognize Unix line endings, or are CRLF Windows line endings necessary? I know that Guerilla does not interpret Unix line endings but interprets Windows line endings. If you could convert the line endings with a program such as Notepad++, you would use fewer bytes; this would be helpful if you want it to remain legible and if the Lua syntax that Blam uses does not need CRLF line endings.
Masters1337: Please test the file size of an .hsc script file or let me know how to do it: do I just add the script file to a "scripts" folder in the scenario's tags subdirectory and issue the "compile scripts" command from within Sapien? I'll be testing the limit as 1 << 18, which is 0x00040000, which is 262144 as maximum bytes, and I will be using Lua comments to fill the data.
EDIT: Kirby's scripting tutorial PDF says to use the corresponding subdirectory of data. I'll test this and report back, thanks for the help.
Edited by sparky on Apr 18, 2017 at 11:17 AM
256 KB is the maximum .hsc script file size. (262144 bytes). If it is 262145 bytes, you get the error in Sapien, "Maximum source file size exceeded."
Edited by sparky on Apr 19, 2017 at 08:46 AM
There are some caveats.
You can have several scripts (sapien and "a hobo" will compile up to 6 .hsc script files in a directory -- it will silently ignore additional script files after the first 6 files), and the total sum bytes of all scripts compiled must be at most 256KB (262144 bytes). The scenario will be able to be opened by Guerilla. Kornman's version of Guerilla will not open a scenario unless every script file used has at most 8192 bytes, because Kornman's version of Guerilla displays the script text in a text box similar to the comment text box and those text boxes only support showing at most 8192 bytes -- so Kornman's version of Guerilla will crash if you try to open a scenario tag that has a script file over 8192 bytes. But it will open fine in the default Guerilla.
Edited by sparky on Apr 19, 2017 at 05:16 PM
Line endings don't matter to the syntax (although I haven't tested for situations where it might) and Kornman's version of Guerilla does not show line endings properly unless they are Windows CRLF format. Changing to Unix line endings will save some bytes.
Edited by sparky on Apr 19, 2017 at 05:20 PM
OK so further testing reveals that MosesOfEgypt was correct: the maximum size of the Comments data in .scenario is 16384 bytes.
Otherwise, you get:
tag data 'editor_comment_data_definition' too large.
When you compile using tool.exe.
However, an additional limitation is imposed by Guerilla: Guerilla will crash if you try to open any (probably not exclusively .scenario) tag where it needs to load more than 8192 bytes in a text block. Since Kornman's version of Guerilla shows the script information in a text block, that means the same limitation is given for script data with that program.
- Maximum Comment Data: 0x4000 (1<<14) (16384) bytes (16KB)
- Maximum Script Data for Sum of All Scripts (that Sapien or a hobo will compile): 0x40000 (1<<18) (262144) bytes (256KB)
- Maximum Script Files that Sapien or a hobo will recognize per directory: 6
- Maximum Multi-line Text Box Data that Guerilla and Kornman's Guerilla will permit without crashing: 0x2000 (1<<13) (8192) bytes (8KB)
So you can see that the interface imposes artificial limitations on the size of data. Another reason why there needs to be an improvement to the HEK.
Edited by sparky on Apr 19, 2017 at 06:43 PM
Also note that Kornman's version of Guerilla "enables" 512 total entries in the Scripts block and 8 total entries in the Source Files block of a scenario tag.
Edited by sparky on Apr 19, 2017 at 06:52 PM
So, in Guerilla's interface...
- Normal text fields (for 1 byte, 2 byte, and 4 byte numbers) are limited to 30,000 characters. Guerilla prevents additional text from being entered.
- String text fields (for labels) are limited to 31 characters. Guerilla prevents additional text from being entered. These always store 32 bytes.
- String text fields (for tag pathname strings) are limited to 254 characters. Guerilla crashes if additional text is entered. These store up to 255 bytes (presumably, especially given some directory path length limitations in Windows XP).
- String multi-line text fields (for comments in .scenario and -- in Korman's version -- for script text in .scenario) are limited to 8191 characters. Guerilla crashes if additional text is entered. These typically store many more bytes, as previously described.
- String multi-line text fields (for .string_list tags) are limited to 30000 characters. Guerilla prevents additional text from being entered. These store up to 4096 bytes; Tool will not compile extra bytes and Guerilla will not save additional bytes.
"tool strings" will compile "ascii/multibyte" strings (ASCII or UTF-8) in this format:
hi this is string A
hi this is string B
As long as each string entry is at most 8191 characters, which in this case translates to 8192 bytes. Actually, it will say that it compiled properly, but in reality, it will not compile the strings into the tag unless they are each at most 4095 characters, which in this case translates to 4096 bytes.
"tool unicode-strings" will compile "16-bit unicode" strings (UTF-16/UCS-2 with Little Endian or Big Endian Byte Order Markings -- they result in the same little endian byte patterns, essentially as 1-byte characters including the string terminator with 1-byte terminators) in the same format as mentioned -- and will actually compile the string entries into the tag if they are each at most 16383 characters, which in this case translates to 32768 bytes, even though Tool will say that it successfully compiled a string that is at most 32767 characters which would have translated into 65536 bytes. Modifying the data to be more than 32768 bytes results in it being unable to be opened in Guerilla, with the error that the string data is too large.
In sum, .string_list entries can be up to 4096 bytes (4095 characters) and .unicode_string_list entries can be up to 32768 bytes (16383 characters). Tool falsely claims to compile entries twice those lengths: .string_list entries up to 8192 bytes (8191 characters) and .unicode_string_list entries up to 65536 bytes (32767 characters).
...which means that the Reclaimer source code is accurate regarding these limits.
Edited by sparky on Apr 19, 2017 at 09:12 PM
Since you know how to do it, could you please put some dummy data into an .hsc and compile it into a scenario?
Did you upload it anywhere else also?
A scenario can have each comment block be up to 8192 bytes including the invisible null terminator byte. How many bytes can an .hsc file be? Does it have the same 8 KB file size limit?
I'm adding some string usage types to facilitate data checking and I have these so far:
I am wondering if anyone can help by simply telling me if an .hsc file has a maximum size in bytes that it can be in order to be compiled into a scenario. The reason I am asking is because Blam scripting is the one area of Halo with which I have the very least experience. I have never compiled a working script into a scenario and never used a script that was compiled into a map file.
The Halo Maps Archive is rather definitive so far. I don't see it being replaced with something else definitive; but anyway, if there is something in the works -- I have my own plans that might be realized this year -- I would be happy to host files as a download location or mirror, for bandwidth or disk space or network speed reasons. There are web services by Amazon and Google that provide probably plenty of disk space. I mean, Dennis could use those free hosting providers as download mirrors. I do not know what people are willing and able to do, if anything, to get the ball rolling again on public contribution submissions.
If anyone wanted to batch download the files from my mirror and add them to a replacement for the Halo Maps Archive, that would be fine with me and they would want to also know that there are several duplicates in the Halo Maps Archive:
File name (normalized in my archives directory), file size (bytes), Halo Maps file page
halo_3_weapons_and_hud.zip 20885424 http://hce.halomaps.org/index.cfm?fid=5777
halo_3_weapons_and_hud_tags.zip 20885424 http://hce.halomaps.org/index.cfm?fid=6131
halo_ce_devmode_for_v1_09.zip 7605 http://hce.halomaps.org/index.cfm?fid=5073
devmode.zip 7605 http://hce.halomaps.org/index.cfm?fid=4995
harvest_police_warthog_tags_v2.zip 3544467 http://hce.halomaps.org/index.cfm?fid=6644
police_warthog_tags_v2.zip 3544467 http://hce.halomaps.org/index.cfm?fid=6649
recon_armor_textured.zip 398057 http://hce.halomaps.org/index.cfm?fid=5009
recon_armor_textured_model.zip 398057 http://hce.halomaps.org/index.cfm?fid=5921
Also, I use Let's Encrypt for https protocol. Halo Maps does not (yet) use https.
I have 6557 (all) of the downloads from the Halo Maps Archive accessible here:https://halo.galaxyverge.com/files/archives/
I downloaded them one-by-one and then used zip and zipinfo to remove the Halo Maps and Gearbox read me files. I also converted the filenames into lowercase to correct alphabetizing. I also converted any non-zip files such as .rar to .zip and converted spaces to underscores. You may use this as a download mirror to help with bandwidth use; I have no substantial limitation of bandwidth on the server, and people are free to download the entire directory in bulk using a tool such as wget if they like.
To use this URL as a mirror, simply convert the file name (without extension) to lowercase, replace spaces with underscores, append the .zip extension, and append this converted file name to the archives directory URL.
Your collection is listed as 6556 files, and I have yet to successfully find the stray extra file that is included.
The archives directory contains archives of other web sites that have Halo files. I also do a download mirror for HaloMD for Mac, which nil developed and maintained. nil gave me a Ruby script to run as cron to mirror the HaloMD files like builds, patches, plug-ins, and map mods. This archives directory has 466.1 GB of files and 146.9 GB of Halo Maps archive files.
My dedicated server is pretty fast and is based in Salt Lake City, Utah. I hope what I did is useful to you and UXB Internet.
So Dennis, we look forward to hearing and seeing what plans you have for continuing Halo Maps. Thank you for your work so far.
Edited by sparky on Apr 17, 2017 at 09:10 PM
Will there be an update this Spring?
Today is Easter. Happy Easter! It is a day to celebrate the victory of Life over Death.
I maintain a mirror of the 6556 Halo Maps Archive files for those who are interested in downloading in bulk (about 145 GB) or without wait time prefaces. This is on a dedicated server and should have nice download speed.