Date: Tue, 13 Jul 1993 18:34:40 -0600 From: jweiss AT silver DOT sdsmt DOT edu (John M. Weiss) To: djgpp AT sun DOT soe DOT clarkson DOT edu Subject: memory problems A few days back I posted a message asking for solutions to the problem of determining how much electronic RAM was available on the heap. While it did not generate as much mail as the getcwd question :-), there were several interesting responses, and requests to post the best answers. So here goes (and I apologize in advance for the length): ------------------------------------------------------------------------ 1. Jeremy Mathers wrote: I don't think there is any RAM free function - since DJGPP explicitly makes no logical distinction between real and virtual memory (some would say that this is a strength of the package). However, this question comes up frequently, and one solution is to put your swapfile on a small disk - say, a small RAM disk. Now, your program will fail very soon after it starts using virtual mem. My response: ----------- This is an ingenious solution, but suffers from the obvious disadvantage of requiring the user to reconfigure his/her system. ------------------------------------------------------------------------ 2. Pieter Kunst (P.J.) wrote: I had more or less the same requirement for my memory hungry programs. What I did is rather simple: I added my own wrapper functions around 'malloc()' and 'free()', and added an extra function which calculates the amount of memory currently allocated. It works fine. The total amount of available physical memory is displayed when you use 'debug32' to run the program. Another reason for this approach is that you can add extra checks in 'free()' to prevent "releasing" memory which was not obtained by a previous call to 'malloc()', and at exit of your program you can check for "memory leaks" (blocks of memory which are not freed by the program). "Memory leaks" in a program can be a serious source for unexpected swapping. My response: ----------- If the "extra function" computes only *allocated* memory, then it does not really address my problem. I want to compute *available* memory, which is almost the inverse. ------------------------------------------------------------------------ 3. Croquette Vincent wrote: I have had the same pb for the same motivations, apparently there is no (simple) way to determine how much free RAM, you have but my experience with 2D FFT is that provide you have basically enough RAM in the machine to fit your 2D array + 1~2 Mb you will swapp only once (or not) and the FFT is going to be fast, we are currently dealing with array 64*32768. However, you have to be carefuill when allocating a 2D array with DJGPP: the allocation procedure work by chunk of power of 2 and if you allocate a 512 element line, it needs 4 extra bytes for the record and thus reserve a 1024 chunk ! Thus allocating a 512*512 array line by line will in fact freeze 2 times the expected size, with big array this turns out to be dramatic. One way to overcome this pb is to allocate the big chunk 512*512 in one operation and then map an array of pointers to begining of lines. If my explanantion is difficult to understand, I can repeat in french or I can send an example if needed. My response: ----------- Please do *not* repeat in French! I am allocating memory exactly as you describe (see next message). I am curious how you determined that the malloc overhead is 4 bytes. ------------------------------------------------------------------------ 4. George R. Welch wrote: I have fought with this problem a lot, and I believe it is boils down to malloc using some buddy-system-like algorythm that only allocates in powers of 2. It is an *extreme* nuicance, and can be really *devastating* if you do something like: malloc(16 megs + small number), because your system will have to come up with 32 megs to satisfy the malloc. Fortunately, sbrk() does not suffer this problem, so I wrote my own malloc that just calls sbrk() and returns an appropriate pointer. The disadvantage is that I never dealt with free(), because I don't need it. My response: ----------- The observation that malloc(n) often seems to allocate far more than n bytes is well worth noting. Be assured that I am indeed allocating the dynamic 2-D array in one big chunk (well, really one small chunk followed by one big chunk). The problem is that the array is first accessed row-wise and then column-wise, and the latter causes an excessive amount of paging to disk (*far* more than just one swap) when it does not all fit in RAM. I will attempt to upload my dynamic 2-D array code to omnigate.clarkson.edu in /pub/msdos/djgpp/pub/alloc2d.zip later today. I believe it may be generally useful. I have also experimented with sbrk(), which is faster and has less overhead than malloc() (ref: C Users Journal, Feb 91, p.97). Most texts recommend against using this low-level system call, for good reason: it establishes a linear (LIFO) heap rather than a random heap. BTW, sbrk(0) returns a pointer to the current break address, which is a convenient way to determine just how much memory has been allocated: #include #include main() { char *c; unsigned long m1, m2; m1 = (unsigned long)sbrk(0); c = malloc(1024); m2 = (unsigned long)sbrk(0); printf("%d bytes were just allocated\n", m2 - m1); } Turbo C outputs "1032 bytes were just allocated" - the 8 extra bytes are obviously malloc bookkeeping overhead. However, with DJGPP the output is "4096 bytes were just allocated"! I don't think it's simply a power of 2, since 4096 sounds suspiciously like a page size - have to take a look at the code for malloc sometime.... ------------------------------------------------------------------------ 5. jmiller AT chess DOT eisc DOT utoledo DOT edu wrote: Your problem is very familiar! I have had students write programs for the same application and things get very slow when page faulting begins. One crude solution might be to do a system call to mem or memory if you are using 4dos and redirect the output to a file which your program can open and read. The best solution would be to modify go32 so that that calls could be made to the XMS driver, EMS driver or the bios for extended memory (depending on how go32 is getting memory) to implement a C function that would return the amount of free memory. Things are likely to be much simpler when go32 version 2 becomes available since interrupts will be much better supported. This would allow gcc to compile code that would perform the calls to the various drivers directly and would be relatively trivial. My response: ----------- More ingenious solutions. I also prefer the latter, which avoids making a system call to a possibly nonexistent program (MEM.EXE only became available with DOS 5.0). In fact, I have written a C-callable library of EMS/XMS routines for Turbo C, and tried porting it to GCC. Unfortunately, I used Turbo-specific features (such as pseudo register variables), and I know very little about the GNU assembler. Would anyone care to help translate my routines to GCC/GAS? My translation produces the error: Unsupported DOS request at eip=...... Perhaps the necessary interrupts are not even supported (sigh). ------------------------------------------------------------------------ Again, I apologize for the length of this message. Thanks to all who tried to help, or at least commiserate! - JW Dr. John M. Weiss, Associate Professor Department of Mathematics and Computer Science South Dakota School of Mines and Technology 501 East St. Joseph Street Rapid City, SD 57701-3995 605-394-6145 jweiss AT silver DOT sdsmt DOT edu