-------------------------------------------------------------------------------- Dynamically Loadable Module Support library for DJGPP. Copyright (C) 2000 by Andrew Zabolotny -------------------------------------------------------------------------------- Hello there, fellow programmer! I believe if you're reading this I don't have to explain why dynamically - loadable module support is a must for any modern operating system. Last but not least, having dynamic libraries support on any operating system is useful if you want to use LGPL libraries, and don't want to disclose your Ultimate Know-How contained in other source files linked against those libraries. Specifically, section 6 of LGPL places several restrictions on `work that uses the library' which could be worked around IF the LGPLed library is put into a dynamically-loadable module. This library was designed specifically for the Crystal Space project (see http://crystal.sourceforge.net), but it is very general and useful in other environments as well. License ------- This library is ABSOLUTELY free. That is, no LGPL, no Mozilla, no credits, no anything (although crediting me is not prohibited :-). Do whatever you want with it. As a consequence, you get ABSOLUTELY no warranties. That is, you get nothing for nothing. (The intention of making this library free is to set a single standard for dynamically-loadable modules under DJGPP. I hate all thouse thousands of non-free similar standards in the software industry, and the only reason why new and new ones appear is because someone is trying to avoid paying several megabucks for the joy of using someone else' standard). Credits ------- This library is in (a very small) part based on work by Charles Sandmann and DJ Delorie. DJGPP C library contains a very simple-and-dirty function called dxeload() and a very simple-and-dirty tool called dxegen which become the `seed' from which this library has grown. How it works ------------ First of all, let's clarify the terms. The original DJGPP dxe package used the `dxe' extension for dynamic modules (I believe it stands for `dynamic executable'). I took this extension (why not?) and developed the version 2 of the DXE format. Thus in the following `DXE' is used as a synonym for `dynamically loadable executable module'. Also I call them sometimes `dynamic modules' or simply `modules'. There are several ways to use DXE modules. The library allows you either to load/unload modules at runtime or to link with them (or rather with so-called `import libraries') at link time (e.g. statically link with dynamic libraries). The library allows to build either dynamic modules that don't have unresolved symbols as well as dynamic modules that DO have unresolved symbols (and allows you to resolve them at runtime). The first way to use DXE modules is implemented through the dlopen() API. This API closely mimics the one found on most contemporan Unices, thus you may even try to compile some Unix apps that uses dynamic libraries without a single change. Statically linking against dynamic libraries allows for a very easy way to introduce dynamic modules into any program. `Static linking' means that during linking stage of your program you link against certain library, called a `import library' in the following, which provides all the functions present in certain DXE module. The symbols points to some very small wrappers, and when you call any of those functions, the library is automatically loaded and the call is redirected further. Also you may load/unload the library manually by calling some special functions called load_MODNAME and unload_MODNAME functions (where MODNAME stands for the name of your dynamic module). The dxe2gen tool can build these import libraries, so basically you just take a existing DXE module and generate the corresponding import library for this module (see the `Import libraries' section). Then you link against it... and voila! it works. Static linkage against dynamic libraries has a drawback: you CANNOT have exported data inside the library (at least you can't use it). This happens because the symbols are resolved to the wrappers and not to the actual symbols inside the module. That is, if you have a variable called `i' in a dynamic module, and you assign `i = 1' from your program which is statically linked against that module, you will actually write over the wrapper and not into the actual location where the contents of `i' are. If you really need to set/get values of some variables inside a dynamic module, write a couple of set_i and get_i *functions*. Building and installing ----------------------- The package comes in binary-less form :-) thus you will need to build the library and the dxe2gen tool manually. Its very easy though. I suppose you already have DJGPP set up correctly (otherwise why you are reading this? :-) thus you just should type: make and you will get the libdl.a library as well as the dxe2gen tool (which is used to build dynamic libraries). Install dxe2gen.exe under X:/djgpp/bin (where X is the drive where your DJGPP installation is) and libdl.a under X:/djgpp/lib. Also you should copy the contents of include/ directory into the X:/djgpp/include directory. Also you will need to copy the src/dxe2.ld file into your X:/djgpp/lib directory. This is a linker script file that tells the GNU linker how a dynamic module should be built. For all you lazy people I have made a special target in makefile, so basically you don't need to do all the above mentioned operations, but rather execute a single command: make install This command will build everything and install everything in the respective places. The above detailed description was put here rather to let you know what will happen when you type `make install' :-) Also the package contains a number of samples/testcases which you may build to see the dynamic modules in action. To compile them just type: make test You will end up with a bunch of executables and dxe modules. Find yourself what they do :-) (the sources are in test/ directory). The API ------- The dynamic module loader API is contained within the file dlfcn.h. There is also a private header file dxe2.h which contains all the non-standard API functions and structures, but you don't need to include it directly since dlfcn.h already includes it. Thus, in all programs you just need to #include . Here are the basic functions of the Unix-like API: dxe_h dlopen (const char *filename, int mode); This function tries to load a DXE. The `mode' field is a combination of RTLD_XXX flags, of which just one works (others are ignored but present in include file for Unix compatibility). The flag is RTLD_GLOBAL which means that all symbols in this module are made public and subsequently loaded modules with unresolved symbols will `see' them and will try to find the unresolved references through them. The function returns a handle. The handle is basically an `void *'. If the function cannot find a file called `filename' all the components of a environment variable called `LD_LIBRARY_PATH' is scanned for that file. For example, you could set it to something like this (I did it at the start of my djgpp.env file, right after DJDIR variable): LD_LIBRARY_PATH=%DJDIR%/dxe;%DJDIR%/lib;c:/usr/dxe;d:/something The components of the path are prepended to `filename' and a file with that name is looked for. int dlclose (dxe_h handle); Well, guess what... this function CLOSES a dynamic module. Whoa! The module is unloaded, all pointers that point somewhere inside module are invalid after that. void *dlsym (dxe_h handle, const char *name); Return the address of a symbol within a DXE. Both functions and data symbols are supported. The returned address is just a (void *) thus you usually will want to typecast it to whatever prototype you have. By specifying RTLD_DEFAULT in the place of `handle' you have access to the global symbol scope. char *dlerror (); This function returns a simple text string (no linefeed at the end) which describes somehow the reason why last dlopen() has failed. -------------- There are several extensions to the dynamic loader interface, that contain non-standard functions providing access to some features not found in other flavours of dynamic module loading facilities. int dlregsym (dxe_symbol_table *symtab); Register a table of symbols to be exported into the loaded modules. You can register any number of such tables. When a module with unresolved external symbols is loaded, all these tables are searched for the respective symbol. If no one is found, the last-resort handler is called. If even the last-resort handler cannot return an address, an error condition is raised. The effect of dlregsym() is cumulative. That is, you can call it multiple times to register several export symbol tables, and all of them will be taken into account when loading a new module. int dlunregsym (dxe_symbol_table *symtab); This is the inverse of dlregsym(), provided for sake of completeness. I never had the actual need to unregister a table of symbols that have been already registered. void dlsetres (void *(*errh) (const char *)); Set the last-resort handler called when a unresolved symbol cannot be found in all the symbol tables that the dynamic loader have at his disposition. The handler should return NULL to rise a error condition, otherwise it should return a valid address. For example, as a last resort, the errh routine could return the address of a dummy function -- this allows to load modules which you don't know in advance which unresolved symbols it contains. Of course, the functions that use this last-resort dummy function will be, most likely, unuseable but at least you may query the address of some table inside the module, for example, and process it somehow. extern void (*dlerrstatmod) (const char *module); This is a *pointer* to a function (e.g. replaceable) containing a pointer to a function that is being called when static linking fails because of missing module. Note that due to delayed nature of static linkage, the error can pop up very late! If you want to check it at startup, call the "load_MODULENAME" function explicitly. The function should never return. extern void (*dlerrstatsym) (const char *module, const char *symbol); This is a pointer to a function that is being called when during static linking the dynamic loader finds that some symbol is missing from dynamic module. The function should never return. -------------- There are also a number of useful macros, which you may use instead of explicitely filling structures etc. Here are them: DXE_EXPORT_TABLE(name) DXE_EXPORT_TABLE_AUTO(name) DXE_EXPORT(symbol) DXE_EXPORT_END These macros allows you to define a table of symbols that are going to be exported into subsequently loaded modules. For a example see the `Building a DXE' section. Also if you use DXE_EXPORT_TABLE_AUTO instead of DXE_EXPORT_TABLE the table will be automatically registered with the dynamic loader during program startup (thus you don't need to call dlregsym(name) manually). DECLARE_STATIC_DXE(name) This macro declares the two functions that are present if you are statically linking against a dynamic library (see the `How it works' section for details on static linking). Note that `name' should be in capitals! After declaring the module with the above macro, you can call the load_NAME and unload_NAME functions to dynamically load and unload the statically linked dynamic library. Building a DXE -------------- To build a DXE module you will need the dxe2gen tool. If you don't have it, make it by typing "make" in the directory where you see the `Makefile'. You will end with a dxe2gen.exe tool and with the libdl.a library (which provides the API described above). Now suppose you have some library which you want to turn into a DXE module. To do this inside the makefile for your library add a rule like this: mylib.dxe: one.o two.o three.o four.o dxe2gen -o $@ $^ That is, the `dxe2gen -o mylib.dxe one.o two.o three.o four.o' will build the mylib.dxe dynamically-loadable module. All public (e.g. non-static) symbols will be exported, and you may query the address for any of them with the dlsym() API function. Now suppose you use some functions from the C library, like strcpy(), strlen() and so on. In this case dxe2gen will list all those symbols and will tell you that they are `unresolved', because these functions are not present in any of the object files you have specified on the command line. From now on you have several ways to go: -*- You can link the DXE module with the C library by specifying "-lc" on the dxe2gen command line. This effectively will add all the unresolved symbols to your DXE module, taking them from the C library. Drawbacks: * If your program and/or other modules use same functions, this will duplicate the code through all of them. * Many functions won't work this way because they refer to other symbols in other libraries such as "-lgcc", the later usually refers some symbols from crt0.o and you can't link with crt0.o (well, you can but you cannot launch crt0.o's initialization routine because it will screw many things up). For example, you can't link against -lc to get the printf() routine. Most file functions and memory allocation functions are a no-no as well. -*- Other way is to leave the symbols unresolved and resolve them at runtime. To tell dxe2gen to not barf at unresolved symbols, tell him `-U'. This will build a DXE module with unresolved symbols. Before loading such a module you should provide somehow the missing symbols to the DXE loader so that it can fix up all the references to those missing functions. If it won't succeed, the loader will fail. There are several ways to provide symbol references to loader. The first one is to provide a explicit pointer to every function you are going to export into dynamically-loadable modules. You can do it with a couple of handy macros: #include DXE_EXPORT_TABLE (exported_symbols) DXE_EXPORT (printf) DXE_EXPORT (strcpy) DXE_EXPORT (strcat) ... DXE_EXPORT_END Now you should pass this export table to the dynamic linker: dlregsym (exported_symbols); Allright, now the loader knows these symbols so if any loaded module has references to them, it knows how to resolve them. You may call dlregsym as much as you want, the symbols are accumulating in a internal table of the dynamic loader. Also you may unregister symbols with dlunregsym() function (although I don't foresee why someone may want it :-) Another way to resolve symbols is to make all exported symbols in one module global, thus they get added to the global symbol table and when a new module with unresolved symbols is loaded, these shared libraries are searched for the corresponding exported symbol. This allows you to use symbols from one module as regular `extern's in another module. A simple example: ---------- module A void something () { ... } ---------- module B extern void something () void something_else () { something (); } When you link module B, you use the -U switch to suppress the warning about symbol `something' being unresolved. Now from your program you first load the module A: dlopen ("moduleA.dxe", RTLD_GLOBAL); then load the module B: dlopen ("moduleB.dxe", 0); Done, the references are resolved. Note that during first dlopen() call we specify the RTLD_GLOBAL flag so that all exported symbols becomes visible to other modules. ---------------- Also sometimes you may want to build a DXE module from a ready library. For example, I have build DXE modules of libjpeg.a, libpng.a and libz.a without recompiling the libraries. For this you should specify a special option to the linker: --whole-archive (this is GNU ld option, it is just passed to ld by dxe2gen): dxe2gen -o z.dxe -I libz_i.a --whole-archive -U libz.a \ -D "Zlib compression library" dxe2gen -o png.dxe -I libpng_i.a --whole-archive -U libpng.a \ -D "Portable Network Graphics (PNG) Reference Library" dxe2gen -o jpeg.dxe -I libjpeg_i.a --whole-archive -U libjpeg.a \ -D "The Independent JPEG Group's JPEG software" I recommend calling all import libraries "libsomething_i.a", to avoid confusion. Of course, if you don't have long filenames it is not always possible. Well, in this case it's up to you how to solve the problem :-) you may call it for example libsom_i.a. Don't forget that all programs linked against libsomething_i.a should be linked against libdl.a as well. Import libraries ---------------- Import libraries are a special kind of libraries (.a) which contains a number of very small wrappers (just long jumps to the actual functions inside the dynamic module (before you cry that jumps are ineffective I should tell you that windoze uses a similar mechanism, and nobody cries windoze is ineffective, heh-heh)) and two functions: load_MODNAME and unload_MODNAME. Initially all jumps are directed to load_MODNAME, thus when you call any function from a dynamic module it initially arrives to load_MODNAME function. Then the module is loaded, all jumps are directed to the respective locations inside the dynamic module, and the last call is restarted. As you may understand (and I have mentioned above), this approach does not work with variables. This is one of the drawbacks of this method. To build a import library you should use the -I switch on the dxe2gen command line. The import library can be built from a ready DXE module as well as `on-the-fly' during DXE generation process. For example, you can do: dxe2gen -I libmy.a -o my.dxe one.o two.o three.o as well as dxe2gen -o my.dxe one.o two.o three.o dxe2gen -I libmy.a my.dxe Both above sequences are equivalent. Contacts -------- The preffered way to contact me is by e-mail. My address is: Andrew Zabolotny, I also run an ICQ client from time to time, you may find me through the search facility of icq (don't remember my icq #).