Xref: news2.mv.net comp.os.msdos.djgpp:3646
From: nicolas AT dsys DOT ceng DOT cea DOT fr (Eric NICOLAS)
Newsgroups: comp.os.msdos.djgpp
Subject: Dynamic loading (was DXE,DLL,ELF)
Date: 9 May 1996 14:57:11 GMT
Organization: Commissariat a l'energie atomique, Grenoble (France)
Lines: 141
Message-ID: <4mt147$rqu@news.cea.fr>
Reply-To: nicolas AT dsys DOT ceng DOT cea DOT fr
NNTP-Posting-Host: hudson.ceng.cea.fr
To: djgpp AT delorie DOT com
DJ-Gateway: from newsgroup comp.os.msdos.djgpp

Hi djgpp gurus !

I saw these days several posts about DXE, DLL, and ELF. All
those problems came from the demand for dynamic loading under
djgpp.

Here is a "resume" of what is wanted and what is available :

WANTED:	Dynamically loaded code.
WHY:	1. Less disk use
	2. Easilly bug-fix simply replacing dynamicly linked code
PROBLEM:1. More memory used when using DLL (all the functions in the
	   DLL are in memory, even if they are never used by program)
	2. Problem when using I/O, Malloc and Global vars
EXIST:	DXE by Charles Sandman
	DLL prototype (where ??)


Thinking of all that, I have a proposal to make. I don't know
if it is really realistic. So I want guru's opinion, and if
possible help.

Here is a quite precise description of functions/algorithm/files
that could be used.

--------------
The basic idea would be to have no linker at all. There would
only be a generic loader.exe which would load all the .o needed
by the program (begining with the one which contains main()).

Here are some ideas for implementation :

1. There would be a first global file : the MODULE LIST (ML).
   It would be the list of all ".o" object files availables
   on the system (computer(+network)). It would be a text
   file like :
   	C:/DJGPP/MODULES/printf.o
   	C:/DJGPP/MODULES/scanf.o
   	C:/DJGPP/MODULES/puts.o
   	...
   It would also be possible to put wildcards in this list :
   	C:/DJGPP/MODULES/*.o
   This file would be written "by hand" after each software
   installation (DJGPP, Third party libraries...). In most
   cases, a new software will simply add a line like :
   	DRIVE:/.../SOFTWARE/MODULES/*.o
   
2. Second global file : the REFERENCE LIST (RL). Il would be
   a list of all references (symbols) exported by modules. This
   would be a text file like :
   	_printf	C:/DJGPP/MODULES/prinf.o
   	_puts	C:/DJGPP/MODULES/puts.o
   For each symbol, there would be first the symbol name and then
   the object file where it is implemented.
   This file should be of course automatically built from the
   MODULE LIST, scanning all declared object files.

3. In memory, there would be a third list : the AVAILABLE
   REFERENCE LIST (ARL). It would contain all references (symbols)
   that exist in memory. For each reference, the list would keep
   following information :
   	- Name of the reference (ex: _printf)
   	- Address of the code in memory (32bit pointer)
   	- Use counter (lock) which recalls how many module use
   	  this reference. When lock==0, no module use this block,
   	  and it can be discarded from memory (This is usefull only
   	  if several programs can be loaded/discarded from memory at
   	  one time).
   	- Name of the module that implement this reference (or 
   	  offset in AML (see later)).

4. In memory, there would be a last list : the AVAILABLE MODULE
   LIST (AML). It would contain the list of all modules that exists
   in memory. For each module, it would keep following information :
   	- Name of the file (path)
   	- Address in memory of the block (returned from a malloc())
   	- Use counter (Sum of all use counters for references exported
   	  by this module)
   	- Time of non-use (if multi-app).

Now here is the algorithm for loading a .o file into memory :

PROCEDURE LoadModule
	Open file. Scan for length.
	Create a block in memory for it, with a malloc()
	Load the block from file into memory
	Scan all external references.
	FOR each external reference DO
		IF (Reference IN ARL) THEN
			LABEL SolveReference:
			Increment the use counter for this reference both in ARL and AML.
			Replace all access to this reference in this module
			by the effective address found in the ARL
		ELSE
			Search in the AL for the module that export this reference -> NewModule
			IF (No module export this reference) THEN
				FATAL ERROR: External reference not resolved
				EXIT
			ENDIF
			LoadModule(NewNodule) (recursive call)
			GOTO SolveReference
		ENDIF
	ENDDO

The code for loader.exe would be :
	Parse command line. The first argument is the object file .o with
	the main() function.
	LoadModule(argv[1])

Of course, the loader would setup the code for this mechanism to work. Basic
IO functions such as malloc(), free()... would be put in the ARL so all
function would use the same code for those critical funcs.

Extensions would be to manage .a archives instead of many .o files of course.

Please note that with this system, only needed functions are loaded in memory.
EXE's size is reduced to the minimum, as duplicate code is only present once
on the disks. Bug fix is very easy, just replace old ".o" files in the
"/MODULES" directories.

The BIG difficulty is to read ".o" files seeking for information about
exported/imported symbols, and places in the code where references to 
those symbols have to be replaced by effective address.

-----------------

This is a very long message, and quite complex. Sorry if my english
is not that good :-) I hope some people will read it and try to understand
what I meant. I am really waiting for opinion.

Please post answers to my mailbox too, My news provider does not diffuse
all messages.

-- 
Eric Nicolas <nicolas AT dsys DOT ceng DOT cea DOT fr>

Take a look to the SWORD home page :
	france:	http://bunny.ensea.fr/Pages_Perso/Cedric_Joulain/sword.web/home.html
	us:	http://www.iquest.net/~cworley/sword.web/home.html