Mail Archives: djgpp/1999/05/19/07:34:31
We use a C program I have written years ago to create word indices for big
volumes of text (>600 Mbytes) excluding some common words we do not want users
to search for.
Originally the program was compiled with DJGPP 2.01 and had a list size of 256
items for excluded words. As the total data size (compressed text+indices)
exceeded the capacity of a CD I changed the number of excludable words to 1024
and recompiled the program with DJGPP version 2.02. When run the program
consumes all available memory then crashes (yes I check every allocation for
success...).
As a last resort I dug up the 'old' 2.01 DJGPP package and recompiled the
program with it. It now runs without problems again.
As the resulting file is also very big (> 200 Mbytes) we must process
separately for individual letters. All processing is done in DOS mode in
memory on a 350Mhz PII system with 256 Mbytes of main memory and a free disk
space of about 1 Gbyte and CWSDPMI version r3.
No optimizations were used as sometimes I found them erroneous.
The resulting file of one pass containing the index for words starting with
letter 'a' was about 60 Mbytes. The hangup occurs in 'realloc()'. It is
usually a page fault.
I seem to remember for something about a change in malloc()'s codes.Can it be
the cause? I am allocating memory of file size+1 for the excluded words. Then
each word consumes a block whose size is 8 + (word length+1) and for each
occurances 5 bytes more. These latter bytes are stored in reallocated buffers,
which means a lot of reallocations.
So I am curious: what could cause this behaviour?
--
________________________________________________________________
Question #7: How Can I Make Linux More Like Windows?
'Hmmm. Rebuild the kernel to use every memory-hogging feature you can find.
Reboot every
couple of days whether you need to or not. And every 18 months or so, send a
check for $99
to Bill Gates. That should do the trick.' (CNET - 10 questions about Linux
[4.28.1999])
- Raw text -