On 9/20/2005 11:14 AM, Philip Warner wrote:

> Jan Wieck wrote:
> 
>> What you could to to keep it simple is to go with a free() approach.
>> free() buffers that are over a certain size after they are applied.
>> And have the fetch thread wait if the buffered amount exceeds your
>> limit. In addition, you probably want to make the initial fetch size a
>> config parameter and also make the actual number of fetched rows
>> depending on the buffers fill level, so to speak. The larger the
>> buffer is, the fewer rows to fetch in order to avoid "fetching 100 50M
>> rows at once" by surprise.
> 
> Thanks for the replies; to summarize the plan (please say yay or nay!):
> 
> - modify remote_worker.c to make initial fetch size a config parameter
> - modify remote_worker.c (and related) to alloc/free large blocks (say >
> 1MB, or a user-settable value)
> - add a config parameter 'fetch buffer limit'

yes, yes, yes

> - modify remote_worker.c to do fetches <= 'initial fetch size' based on
> currently used memory and 'fetch buffer limit'. Minimum 1.
> - modify remote_worker.c to pause after completing one complete fetch
> cycle to pause if exceeding 'fetch buffer limit', and automagically wake
> up again....hmmm.
> 
> (should we just skip the last one?)

I would say to modify remote_worker.c so that it treats "too much memory 
in use" as if there would be no more buffer lines available. The latter 
can happen already, and remote_worker will then wait on the condition 
variable for the local apply thread to return empty buffer lines to the 
pool. That event (returning lines to pool) is what triggers the 
condition variable and thereby wakes up remote_worker. Since this event 
is not guaranteed to be in connection with any free(), you'll have to 
recheck and eventually wait again.


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck at Yahoo.com #