LBA Modding API

[Battler]

I'm thinking of making a .DLL file with a LBA Modding API that then all the various tools can call. The language is going to be C or C++. That way, the current... well.. chaos can be fixed a bit.
Basically, right now we have every single developer writing their own code for loading HQR, (de)compressing, loading a picture, a palette, a model, text, etc. This leads to every program doing things slightly differently as well has having different bugs which can then accumulate and increase the possibility of problems the end user would have while using a mod. Not to mention it has often led to arguments over whose implementation of what is better.

The requirements would be as follows:
1. It should be callable with all its features from any programming language, be it classic Visual Basic, Delphi, C, C++, etc.;
2. Every single piece of code included should go through some discussion among the current developers of the LBA modding tools (that would be me, xesf, and Zink - I think the rest have pretty much disappeared, but correct me if I'm wrong );
3. Optimization is a must - the code must be reasonably fast and shouldn't use too many resources.

That would be it for this post. We need to have a good discussion about this so that we do it right.

[Neko]

Good idea, why waste time doing things over and over?

[Darkflame]

+1

certainly seems a good idea

[Battler]

Awesome then. Let's wait for xesf and Zink to appear, so the work can start.

Also, we need to settle the terminology as well. There is some in use but it does not quite match the terminology used by Adeline themselves in the file with the short descriptions of the LBA .HQR files included with Japanese LBA 1 (that was evidently given by Adeline to whatever company in Japan localized the game, and the company screwed up and included some of it on the CD).
For example, what we call a Brick Layout Library, they seem to call a Block Library or something like that.

[xesf]

Link started this idea few years ago. He started by doing this https://github.com/xesf/lbapi
It's indeed a nice idea, so I guess you can use that as a starting point.

[Quetch]

Wow!!! I'm all up for it!!! Ironically, I was thinking about exactly the same thing for the past 2 months now, I've even contacted yaz0r to see if he still has any useful piece of code, but unfortunately I didn't hear back from him, his webpage must be abandoned by now

[xesf]

He moved on of this subject and I don't think he had something like this anyway.
The person that may have this as closer as possible is Zink. His apps have everything an LBA API will need. It's just on a different language.

[Quetch]

I know he moved on, but yaz0r was one of the best modtool creators out there, just look at all of the other games he's been able to breech. Also, his work on LBA was mainly related to the models with their bone animations and file format, so that would've been a HUGE help

[Battler]

- Quetch: Obviously he created the best modding tools - he has the source code of both LBA 1 and LBA 2. If we all had those source codes, all our modding tools would be much better than they are.

Also, from what I remember, Link's wasn't really an API, despite its name. It was more an ABI (Application Binary Interface) as he intended the code to be included directly while my idea is to create a DLL that other applications can call, which would IMHO be better because, if done right, we would have one single DLL that modding tools wold then dynamically link to.
This way, something that works in one program, would work directly in all others as well, without worrying that someone's slightly different compiler could introduce bugs (eg. from my PCem work, I learned that GCC likes misaligning the stack unless you use -mstackrealign, and the stack misalignment can cause crashes where none should occur). That, and it would make the API usable without problems, regardless of what programming language you're using, be it VB 6, VB.Net, C#, C++, C, Delphi, etc., something that was a problem with Link's ABI attempt, and that I did point out from what I remember.

An idea for the HQR loading functions:
int HQROpen(char *FileName, int FileType, FILE **fPointer);
int HQRAddFile(char *FileName, int CompressionType, FILE **fPointer);
int HQRAddPointer(int DestinationIndex, FILE **fPointer);
int HQRAddNull(FILE **fPointer);
int HQRSetFile(char *FileName, int CompressionType, int Index, FILE **fPointer);
int HQRSetPointer(int DestinationIndex, int Index, FILE **fPointer);
int HQRSetNull(int Index, FILE **fPointer);
int HQRSetCompressionType(int CompressionType, int Index, FILE **fPointer);
int HQRExtractItem(char *FileName, int Decompress, int Index, FILE **fPointer);
int HQRDeleteItem(int Index, FILE **fPointer);
int HQRSave(FILE **fPointer);
int HQRSaveAs(char *DestinationFileName, FILE **fPointer);
int HQRClose(FILE **fPointer);

They're all int because they should IMHO return the status (whether the operation finished successfully, whether there was an error, etc. - we should standardize the error codes too, so then all a modding tool developer has to do is come here, look at our table of error codes, find the function he's using, and the error code, and see what it means and can therefore make their user interface react correctly).
They require a pointer to a file pointer so the API function knows which HQR file to work on, though there could be other ways too.
The FileType at HQROpen is to distinguish between regular HQR's (type 0) and VOX'es (type 1) which IIRC have stuff like hidden entries and so on. Possibly also type 2 for HQS's (Zink's High Quality Scenarios), depending on whether they differ at all from regular HQR's.

[ChaosFish]

Have you tried asking for the original source code? Now that Fred & Co. are the copyright holders... maybe it's not that far fetched?

[Neko]

I doubt that because the games are commercially available again, so it is not freeware.

But definitely ask!
Was yaz0r in the original adeline team?

[ChaosFish]

Does it matter if the games are commercially available nowadays? If someone wants to get the games without paying it only takes a Google search and a few clicks, even for young kids and inexperienced computer users. The difference between the devs and a big publisher is that the devs would more likely be willing to use common sense, plus they want to see their creation live on and they think beyond just money.

Moreover, they can release the source code for the engine without including the game resources, which would make this whole issue irrelevant.

[Battler]

Originally Posted by ChaosFish

Moreover, they can release the source code for the engine without including the game resources, which would make this whole issue irrelevant.

As a matter of fact, that's precisely what ID Software did with Wolfenstein 3D (and Spear of Destiny) and Doom - they released the source code for both games, but not the game resources. Adeline software could do the same.

Also, I do have the original compression code for both type 1 (LZSS) and type 2 (LZMIT), that I got from yaz0r and Fred, but I do not have permission to distribute it. I am allowed however to make binaries out of it and distribute the resulting binaries (hence why I made a .DLL out of it).

Edit: Though the LZMIT code wouldn't be part of either game anyway, except for decompression which is in LBA 2 (I think I got the Adeline decompression too, but I forgot how I got that), as it was only used by their HQR tool (called make_hqr or something like that). The LZSS code however is present in the LBA 2 source code, as LBA 2 uses it to compress saved games.

[xesf]

Guys, yaz0r is just brilliant.

The code he reversed for twin-e he actually didn't use the original source code. He got the source after but he kept doing the reverse but most of it was already playable. We can still see in the current code that is different from the original.

He help me a lot in the beginning to understand LBA internals and also code related stuffs I wasn't aware at the time.
On the LBA Animator Studio tool I did a big part of it was done thanks to him, otherwise we wouldn't had the tool.

Anyway, I've tried a couple of times to get the source code with no luck.
There was a time Fred was nearly there to provide pieces of the code, not the entire source, but he just gave up. Don't know if this is related to the Didier SARL company.
I was mainly asking for few pieces of code missing in the TwinEngine like the Holomap piece which will make the game more playable.

I still believe this could happen some day and a lot more interest on the game will appear. I can say I would love to work on the TwinEngine again if I got the source or missing pieces of it.

[Neko]

I'm kinda lost what kind of cool modding stuff we can do and what we cannot do....

let's say I'm new to this forum, what cool programs should I go check out?

[Polaris]

Originally Posted by Darkflame

+1

I am eager to see what will come out of this ! (and sad, as always, that my programming skills are very limited, hence I can not help)

[Battler]

- Jesse: We can modify pretty much everything of LBA 1, and some things of LBA 2 though very lacky - we lack anything for modifying LBA 2 models, animations, exterior islands, etc.
You should first start with a program that can work with HQR files to familiarize yourself with them.

[Zink]

I do like the idea of a DLL (or several DLLs) with modding routines, but even having the routines written, there is still much work needed to make them universal enough for a DLL. I don't have time for that these days, sorry .
Maybe some time when I'm too tired to do anything else... but don't count on it . Also I would have to convert my code to C, and that's even more work... actually I don't see why stick to C/C++. Delphi can output a DLL that, if done right, will be compatible with all other languages.

"Brick Layout Library" was actually a name made by me. I needed to expand the BLL abbreviation, and that was the best I could come up with then .

I remember I have discussed with Frederick about the LBA2's source code, and he indeed was willing to share some fragments (not everything), but he needed to talk with his team about this, and then they decided to give up on this idea because of the planned commercial release of LBA2 (or so I remember). But I don't think asking again would harm. Maybe they will change their minds. They could release the source with a license that forbids sharing the resources, so that anyone wanting to play whatever new engine version the community releases would still need the HQRs from the original game.

It's really good to have an independent source code for LBA1, but I don't think it can be used for anything public legally anyway. I couldn't find the LBA1's license anywhere on the CD or in the installation directory (maybe it was in printed form in the box, which I don't have), but every license I know these days contains a clause that forbids disassembling, decompiling and reverse engineering whatsoever. Does anyone happen to have the LBA1's original license to check that?

[Battler]

- Zink: But maintaining a Delphi DLL is difficult... Delphi itself is paid, and from what I know, the free alternative is not 100% compatible with Delphi programs (I think I couldn't even get your programs to load right, and then I gave up). A combination of C and C++ is much easier, especially if written properly so it can be compiled with Visual C++, OpenWatcom, and GCC alike.
Also, I can write the code myself, that's not a problem. I have plenty of time. We just need to agree on everything first so that I then have a framework I can fill in that we have all agreed on.

Originally Posted by Zink

I remember I have discussed with Frederick about the LBA2's source code, and he indeed was willing to share some fragments (not everything), but he needed to talk with his team about this, and then they decided to give up on this idea because of the planned commercial release of LBA2 (or so I remember). But I don't think asking again would harm. Maybe they will change their minds. They could release the source with a license that forbids sharing the resources, so that anyone wanting to play whatever new engine version the community releases would still need the HQRs from the original game.

That's precisely what they should do and is also what ID Software did with Wolfenstein 3D and Doom - they released the source codes but forbade distributing the resources, so you still need to own the game if you want to play it (or make your own resources, for which plenty of tools exist for both games). And both Wolfensten 3D and Doom are still actively sold from what I know. Apogee/Jam also did the same for Blake Stone which they also still actively sell. So I really don't see why Adeline/Ludoïd/whatever they are call now can't do the same for both LBA games, and IMHO, they should.
In the worst case, I hope they could at least give me permission to distribute the compression and decompression sources (I doubt they would impact their game sales a lot ), that would at least make me able to stop wasting time trying to modify Haruhiko Okumura's implementation of LZSS to behave like the type 2 (LZMIT) code (I did succeed partially, it produces output that is compatible, just bigger as matches on the edge of the buffer are ignored and I have no way of making his code accept those matches). From what I remember, back then they couldn't allow it because Paul de Senneville owned half the copyright to the LBA games, but from what I understand, the copyright issues have since been cleared so maybe someone needs to talk to Fred in my name about this stuff. If we could get at least the compression and decompression sources publicly distributable, it would be great already.

It's really good to have an independent source code for LBA1, but I don't think it can be used for anything public legally anyway. I couldn't find the LBA1's license anywhere on the CD or in the installation directory (maybe it was in printed form in the box, which I don't have), but every license I know these days contains a clause that forbids disassembling, decompiling and reverse engineering whatsoever. Does anyone happen to have the LBA1's original license to check that?

Actually, reverse engineering is explicitly protected under EU law, though illegal in the US. So while an engine made by reverse engineering LBA 1 (or LBA 2 for that matter) would be legal in the EU, it would probably be illegal in the US.

[Zink]

Originally Posted by Battler

- Zink: But maintaining a Delphi DLL is difficult... Delphi itself is paid, and from what I know, the free alternative is not 100% compatible with Delphi programs (I think I couldn't even get your programs to load right, and then I gave up). A combination of C and C++ is much easier, especially if written properly so it can be compiled with Visual C++, OpenWatcom, and GCC alike.
Also, I can write the code myself, that's not a problem. I have plenty of time. We just need to agree on everything first so that I then have a framework I can fill in that we have all agreed on.

Alright. As for me, you have permission for all my LBA programs' source code to make DLL versions of the routines you will find worth it. All of it has been written by me (unless explicitly stated in the source), so there should be no legal problems. If there is a program without the source release (most likely because I forgot), it's enough to ask .

Sorry, but I don't have time to discuss the implementation details, so I will leave the function headers and the whole construction to your discretion .

[xesf]

Same as Zink, I'd love to be able to help but would be good enough just to follow.

Have a look on the residual twin-e project. it has a lot of code there for both LBA1 and LBA2 all C++