LBA HyperScript with Prolog

[JLC]

A lot of people enjoy to make new game stories.
Anyhow, despite the remarkable efforts of tools developpers,
making a custom LBA game remains time consumming, and also
the script writting may be esoteric for those not involved in computer science.
So this thread will be to discuss the possibility to speed up this development and make it easier.

You will find source code and exe of tools we will speak about in the following links.

HyperScript / Prolog sources / Scan Memory Tool
https://drive.google.com/file/d/1MnJ...usp=share_link
FastLBA & debugger setting
https://drive.google.com/file/d/12kN...usp=share_link


Now a list of problems encountered while making my custom games:
1) each time you modify the script, you wait for videos to restart LBA
(fixed with the FastLBA recompiled version)
2) internal index references becomes fast tricky, especially
if you play with scene rows while rebuilding the .hqr
3) FLAG_GAME settings becomes fast difficult to reference
4) increased nested loops becomes fast unreadable
and so on ....

So the idea would be to make a kind of meta script over life & track scripts.
This one should know the context and generate standard script language from a pseudo language closed to natural language.
So the tool chain would be:
Scene context <-> HyperScript --(compile/link)--> standard script language
---(compile/link)--> binary script code --update scene.hqr
--> fast reload game on the scene --> test

This means:
- making and analysing a new language so making a compiler/linker
- Have a modern tool to patch and control LBA under DosBox
- Have functions to update .hqr script section in C /C++ or C#

You will find first models of these tools in the links above with detailed README.txt explainations.
Into the next post I will explain how we can make in a few lines of code a compiler with Prolog language, starting from a language grammar.

[JLC]

Here is a grammar of the life script:





/*---- LBA DCG GRAMMAR Backus-Naur Form ------

LIFESCRIPT ::= COMPS END | END

COMPS ::= COMP INSTRUCTIONS END_COMPORTMENT | COMPS

COMP ::= COMPORTMENT main | COMPORTMENT NUMBER

INSTRUCTIONS ::= INSTRUCTION | INSTRUCTIONS

INSTRUCTION ::= COMMAND
INSTRUCTION ::= COMMAND ARGS
INSTRUCTION ::= CONTROL

CONTROL ::= CTR CONDITION INSTRUCTIONS ENDCTR

CONDITION ::= VARIABLE OPERATOR NUMBER
| VARIABLE ACTOR OPERATOR NUMBER

ENDCTR ::= ENDIF | ELSE CONTROL ENDIF

CTR ::= IF | ONEIF | OR_IF | SWIF

ARGS ::= NUMBER | ARGS
ACTOR ::= NUMBER

NUMBER = {1,2 ...255}

VARIABLE = {ACTION,....ZONE}

OPERATOR = {'==','!=','>','<','>=','<='}

COMMAND = { COMPORTMENT, SET_COMPORTMENT, END_COMPORTMENT ..... }

--------------------------------------*/
Formal grammars DCG are naturally included into prolog
So the Prolog code has few differences:


% DCG section

lscript --> comps,["END"],!;["END"].

comps --> ["COMPORTMENT","main"], instructions,["END_COMPORTMENT"],!.
comps --> ["COMPORTMENT","main"], instructions,comps.

instructions --> instruction.
instructions --> instruction,instructions.


instruction --> command,!.
instruction --> control,!.

control --> ctr, condition, instructions, endctr.

condition --> variable, operator, number.


ctr --> ["IF"],!.
ctr --> errctr.
endctr --> ["ENDIF"].

number --> [N],{number(N)},!.
number --> errnum.

command --> ["SET_COMPORTMENT"],number,!.


variable --> ["ZONE"],!.
variable --> errvar.

operator --> ["=="].


errctr --> [C], {incerr,string_concat(C,": unknown control\n",S), write(S)}.
errvar --> [C], {incerr,string_concat(C,": unknown variable\n",S), write(S)}.
errnum --> [N], {incerr,string_concat(N,": incorrect number format\n",S), write(S)}.

-------------------------------
You see, I just type it globally "AS IT IS"


now we execute like this
> P =["COMPORTMENT","main",
"IFa","ZONEb","==",3,"SET_COMPORTMENT","1","ENDIF",
"END_COMPORTMENT",
"END"
],

phrase(lscript,P).



and it replies

IFa: unknown control
ZONEb: unknown variable
1: incorrect number format
3 errors
true.


fixing the 3 errors
P =["COMPORTMENT","main",
"IF","ZONE","==",3,"SET_COMPORTMENT",1,"ENDIF",
"END_COMPORTMENT",
"END"
]


execute again and you get
no error
true.


error management are set in the terminals of the tree
If it compiles 0 errors, you just put write(OPcode) instead.
this is the link phase.
then the binary code is immediatly written.
{} into the DCG section are pure Prolog code insertions.


In the next post we will see the meta grammar.

[bloodhound]

Looks like a great idea! I haven't worked with scripted scenes before. Is this example how it's currently scripted or is that your version?

COMPORTMENT main
IF ZONE == 3
SET_COMPORTMENT 1
ENDIF
END_COMPORTMENT
END

I'm wondering if making it too similar to normal English would make it hard for people to get the syntax right? So if I look at this example:

twinsen goto zone 1.
if twinsen into zone1 and action then bonus1.
bonus1 if add 3 lifepoints.
if twinsen closeto grobo1
then twinsen takes magicball and twinsen throw magicball to grobo1.
if twinsen over lift1 and action then lift1 go down to sewer1.

I could imagine that it would be easier for people to get the syntax correct if it's more terse and consistent, for example using YAML:

steps:
  - actor: twinsen
    triggers:
      - type: enter
        params:
          - zone1
      - type: action
    result:
      "$ref": /results/bonus1
  - actor: twinsen
    triggers:
      - type: proximity
        object: grobo1
    result:
      - action: take
        params:
          - magicball
      - action: throw
        params:
          - magicball
          - grobo1
  - actor: twinsen
    triggers:
      - type: trigger_object
        params:
          - lift1
      - type: action
    result:
      - action: move
        actor: lift1
        params:
          - sewer1
results:
  bonus1:
    target: lifepoints
    expr: "_ + 3"

What do you think? Of course it could be that your natural syntax is actually very simple. Good documentation, maybe with some syntax diagrams would help a lot there.

I think RenPy also has a good language for a similar purpose by the way.

[JLC]

Your notation could be parsed in Prolog although it is more confortable with natural language. Also for young people or people not scientist in computer structures, natural language is easier. But the grammar must be strong and without ambiguity.
As you can see in the meta file mcplife.pl into the doc, I found some inconsistencies in the grid numbers, index numbers and so on.

Hazardous environment around the script may lead to a failure at execution.
So now the work is to define a language, to inspect different script in the game and to see if most of them complies with the grammar.
Anyhow part of tested scripts would be inserted at link time.
For example you have here a little game to fix the problem of the lift (going to the sewers)
https://drive.google.com/file/d/1M_O...usp=share_link
This means to insert code into different scripts, keeping coherence for the whole game. Feasibility of this must be evaluated before to go on.

[bloodhound]

Very nice! For me the biggest use case would be to write a small in-game cut scene, like "Zoe walks towards Twinsen. Zoe says 'Where have you been all the time?'. Animation: Zoe wags her finger.", etc.
For a real mod with a story and scripted interaction I'm too bad of a procrastinator (as people might have noticed by that I didn't manage to come up with a pirate drawing in time for the contest).

[JLC]

The first thing to do is to review architecture.
Incoherence between hqr ranks, internal indexes (numbers, not symbolic names actually), access to hqr files with tools in different languages,

global variable not referenced so everywhere into the code, all this lead to a "gaz factory".


1) The first thing would be to put all this into a database.
This would give immediate reference, acting as a dictionnary and great modifications facilities with SQL select for update.
Also the coherence of the hqr file making would be assured.
And custom codes should be made and used as libraries from everywhere if the database is online



2) using this database, think to rewrite the LBA life engine (DolLife function into Perso.c). This allow to add some AI for example if the engine rewritten in Prolog.
Anyhow rewriting engine means also to rewrite acces to graphic, music and so on.
In a first time, just a tool as described above don't set the bar too high as it may lead to an unfinished tool.

[JLC]

While testing custom mods issues (LBA remake & LBArchitect), problem comes mainly from index reference errors.
Some indexes are not associated with a symbolic name, just a number.
So using a database would fix this and also allow to check cross references, for example knowing fast where you set and modify
global variables, also can extract context parameters for scripts / metascripts
So the organisation of the tool would be like this:






For instance, SCENE_HQR will be the first to insert.
The main windows: Prolog XPCE graphic library (editor and debugger built with XPCE)
It's enough for script testing and portable on Win32,64 Linux, Mac.
Compress / Decompress should be rewritten in C dll close to the .asm expand file.
Also C dll manage C calls in form of Prolog predicates in a few lines.
So from Prolog the features implemented woud be:
Load /Save scripts
Ask database
compile / rebuild .hqr
launch the game for test

The last thing is to know if a custom game engine rewritten in Prolog would be usefull.
This implies rewritting all the game especially graphics display & sounds.
OpenGL Prolog library exist, but it's too heavy and for a first time I don't do it.

[bloodhound]

Maybe an in-memory hash table would make the architecture simpler? For HQR compression and decompression I feel like there are already 10 different implementations, but feel free to write one in Prolog as well

I agree that rewriting the entire game engine is too much effort, and I don't see much benefit either, especially since there is already a rewriting of most of the game engine in TypeScript.

How comes you know so much Prolog? Did you get to use that at work and did you already finish other side projects with it?

[JLC]

I exchanged some PM with Adrien.He made a great work for remake version. He told me, if I want to help, there was many things to fix for the LBA1 version. As you can see, custom games made from LBArchitect and remake version, all of them are using the Script language defined into Zinc LBArchitect tool. Remake version use a custom game engine, LBArchitect the original game engine. Checking and testing it appears that 95% of instructions possibly generate a correct behaviour at run time, but 5% differs and makes the game jamming or not running as we hope. So the interest of a database would be to split hqr files (scene mainly) into relational tables model to isolate some critical piece of code known to generate issue. From script extracted since original binaries, re-generate binaries and compare them to original using a binary editor. See what differs (instruction, arguments, context) and fix the problem. This is a machine work made into a batch. SQL is an universal language and database tables match with prolog predicates.
For compress and decompress I need a C tool written closed to original assembly code, short and using binaries operator | & >> <<, and capable to de-compress and re-compress identically to the origine. Sorry I don't find.
For the last question: yes, of course.

[bloodhound]

Yes, staying in close contact with Adrien is a great idea, as he's friendly and knows a lot.

If you want to create a tool that should also appeal to people who don't have a computer science background, then it would be nice, if the installation process didn't necessitate installing a database and a C compiler apart from the Prolog compiler. A tool that I find great is the LBA packager, because it doesn't require any installation. Probably the LBA packager can also be used for manually modifying the HQR files with the new scripts for people who prefer that option. For compression and decompression, I'm only aware of tools written in Pascal, C#, JavaScript and Python.

[JLC]

No need of C compiler. I am writting a lbatools.dll
It may be called from Prolog, C,C#,javasript ...
calling arguments: file (scene.hqr), scene/grid index number, actor number ...

It returns track and life scripts
Then you can modify and reinsert, making a new .hqr
compress and expand are written ASM inline
expand already works.
Some time required to finish and test all this.

Database is for development and at term may be remote, so nothing difficult to install.

[bloodhound]

Super! Just out of curiosity, why did you write compress and expand in assembly instead of Prolog or C#?

[Polaris]

I need to learn programming...

[JLC]

Prolog is 3rd generation language excellent for language analyse, writting compilers but compress is low level, better made with language level 1 or 2 like ASM or C.
The goal is a dll, independant of language, but I will not use OO language like C++ Pascal or C# and this for a good reason: using them means to rewrite part of the code, because original code use direct struture memory mapping into hqr files. Object implementation use another mapping.
So the original code can run under C/C++ visual studio, but you will have to make some modifications
1) solution is Win32 application using MASM for debug setting to trace into ASM inline
2) 2022 version has a bug but not 2017 so using 2017
3) take care of memory aligment
for example this struct is required for compression and directly mapped:


typedef struct {
ULONG SizeFile;
ULONG CompressedSizeFile;
WORD CompressMethod; // 0 stored 1 LZS
} T_HEADER;


declare:
T_HEADER header;
Now in visual studio sizeof(header) = 12 , but 10 into DOSBox under debugger:




so I had to use sizeof(header)-2 in my code or it will crash.
sizeof is compiler dependant and original code used sizeof to write offsets and binary settings.
This is why I talk about database. Database records would contain script, image, sounds as bytes.
Relations are maintained by the relational model. So then no problem of offsets, indexes, insertions and so on,and good portability.




Now, the compress is just written like this:

void Expand(UBYTE* src, UBYTE* dest, ULONG count)
{
__asm {
push esi
push edi
push ebp

/* original ASM inserted here */
termine:
pop ebp
pop edi
pop esi
}
}

Like this it works quite well and done in C in a few line.
Just keep/restore context as the ASM use these 3 registers.


Now I will just make a tool over LBArchitect or remake, not rewrite the game.

[bloodhound]

@Polaris: I spent a lot of time learning programming and I still only understand half of what JLC does. He learned things a few decades before me and then I guess was in academia or a deep-tech company, so the technologies are very different.

By the way, to make the header size the same you can also specify the alignment (see GCC docs about this). What debugger do you use in DosBox? I wonder if that's similar to what the programming environment looked like for the LBA team.

[JLC]

You have the complete tutorial for modifying, debugging compiling LBA sources there:
https://drive.google.com/file/d/12kN...usp=share_link
I compiled with open watcom 2.0, but used watcom 10 debugger into DosBox.

For LBA hypertext I compiled with Visual Studio but have to do it again with MINGW GCC to make a dll callable by prolog.
I agree, aligment specification may be set.
I finally choose the configuration:
Main window will be in Prolog using its object oriented layer linked to XPCE (C graphic win32 API). This is not 'up to date' but quite enough to make the main window and fast to write.
For the database, tables will be Prolog terms:
for example:
:- module("lbabase",[game/3,.....]).

game(GameID, GameName, NbScenes).
scene(SceneID,GameID,GridID,ListActors).
actor(ActorID,GameID,SceneID,TrackScriptID,LifeScripID).
lifescript(LifeScriptID,FileRef).
trackscript(TrackScripID,FileRef).


Prolog index the first arguments, and engine backtrack simulating the select instruction so it is like a relational database in memory.
From the main window we just load it from a file by :-use_module(lbabase,[ExportedTerm1,...]).
and saved on disk by save_module
So it avoids to use MySQL in a first time, but directly mappable to it at term.
hqr handling and LBA game loading will be on a dll, C code interfaced with prolog predicate.


At the beginning, grids with blank scripts (empty scenes) are required.
The meta language is written, translated into script language, and this last translated into LBA binary commands. These are inserted into a scene.hqr and a button load the game immediatly for a test.

[Lupin]

This is an interesting approach.

A few years ago, I was also interested in making a higher level language on top of the existing LBA scripting language(s), my approach was leaning more towards a visual scripting language (thus the blockly-based scripting in the remake editor, see attached screenshot for reference). I was intending to add some higher level constructs for making it easier for non programmers.

Some advantages of the visual scripting approach are that it leaves less space for syntax errors, and it makes exploring the possibilities easier, just by tinkering with the visual blocks, without the need for an external documentation.

I never ended up building any of those higher level constructs, though, and nowdays I have less time/motivation for LBA-related projects.

What do you want out of your HyperScript exactly? You mentioned syntax (wanting to have something closer to natural language), but what about semantics? How do you see things working? Would it be more of a declarative approach, or do you intend to remain within the procedural framework of the original scripting language?

[JLC]

Just to explain, here is the first model of the user interface.
I made it with a prolog object oriented layer linked to an old win32 API,
so be tolerant, I have not all the feature of 2023 but it's enough to do what we need.
Also it may be written fast in a few lines, it is closed to smalltalk-80
with get send and message between graphical objects

(

hs :-
new(F, frame('LBA HyperScript')),
send(F,geometry,'1200x600+100-600'),

send(F,append,new(HyperScript,view)),
send(F,append,new(Script,view)),
send(F,append,new(Console,view)),
send(F,append,new(D,dialog)),

send(Script,right,HyperScript),
send(Console,below,HyperScript),
send(D,above,Script),
send(HyperScript,label,'Hyper Script'),
send(Script,label,'Script'),
send(Console,label,'Console'),
send(Console,size,size(80,10)),
.........
)


and you get this:






So we create a project in file menu, in games we link to existing grid with a blank scene so no script
actor is selected also.
Then we enter the 'Hyper' script.
Compilation displays the code into Script view, errors or comments are displayed into console view.
these last 2 ones are read only.
there is a prolog database in memory to assign symbolic names to numbers (grid index numbers and so on) and to manage code generation.
Once compiled 0 errors you can generate binaries and a dll linked to prolog rebuild scene.hqr.
then cliquing on the Twinsen icon in the icons bar you can load and test directly on dosBox.
Once the code is good you can copy / paste to Remake or LBArchitect.
Tools utility has a cross reference feature to find for example all occurences of a global variable into your whole game.


The meta langage will insert on keyword typical behaviours for example the code for using a grid lift down to sewers.
These template scripts will be stored into the database.
It is ambitious project, and to be tested, but it's worth it if it works...

[JLC]

Some more explainations about semantic.
As you know, language parsing has 3 levels:
1) lexical analysis: verify the leaves of syntaxic tree (the words of dictionary) are correct
2) syntaxic analysis: verify the sentence complies with the grammar
3) semantic analysis: verify the sentence has a sense into the whole context

(1) is quite easy to do for LBA script
(2) easy for script language but for meta script, pay attention to ambiguity:
for example the following simple grammar analyse the following sentence:
'the man writes a program'
(NG nominal group, VG verbal group)

SENTENCE ::= NG VG
NG ::= ARTICLE NAME
VG ::= VERB COMPLEMENT
COMPLEMENT ::= NG END
END ::= []
ARTICLE = {the,a}
NAME = {man,program}
VERB = {writes}

executing this, we bind the P variable to the sentence (uppercase are variables),
and phrase prebuilt predicate is for executing a DCG grammar in Prolog.
The top level of the interpreter do only one thing: evaluate to true or false.
So here true means the sentence complies with the grammar.
?- P =[the,man,writes,a,program],phrase(sentence,P).
true.

Now I let P unbound, add display instruction on the console, and force the result to fail
with the predicate fail (this one always returns false).
This force the inference engine to backtrack and evaluate all the possibilities.
You get this:

?- phrase(sentence,P), atomic_list_concat(P,' ',S), writeln(S), fail.
the man writes the man
the man writes the program
the man writes a man
the man writes a program
the program writes the man
the program writes the program
the program writes a man
the program writes a program
a man writes the man
a man writes the program
a man writes a man
a man writes a program
a program writes the man
a program writes the program
a program writes a man
a program writes a program
false.

As you can see, all these sentences complies with the grammar,
but a part of them has no sense.
Semantically, this grammar is ambiguous.

To take rid of this we must use a restricted and not ambiguous grammar.
Also Prolog has intelligent editor feature that evalutes on the fly as you are typing a sentence
if this one complies with the language ore not.
Script language is actually to be compared with an assembly (first generation) language,
as retro generated from binaries.
Hyper Script would at least make it 3rd generation, using prebuilt pieces of code.
So natural language keywords would be comparable to functions linked to these libraries of code at link time.