Christopher Browne wrote:

>
>> log_origin | log_xid | count
>>------------+---------+-------
>>          3 | 4795427 | 45228
>>          3 | 4795451 |     6
>>          3 | 4795452 |     6
>>          3 | 4795481 |     6
>>          3 | 4795501 |     6
>>          3 | 4795505 |     6
>>....etc for 8000 more rows.
>>
>>The large single xid has about 1.5GB of data and was caused by doing an
>>"insert...select..." statement.
>>
>>Can slon be hacked to avoid buffering so much in memory?
>>    
>>
>
>I don't see any easy "slon hack" here; I'm not so sure that the
>problem is actually within slon; it may be that the big transaction is
>causing the backend to fall over, in which case trying to "fix" slon
>would be a futile exercise...
>  
>
It's the slon worker process that dies by exhausting its memory. It
seems to want to read the entire xid's worth of log entries into memory
while applying them in background, and since reading is a lot faster
than writing that translates to > 1.5GB memory for the one process. I've
logged a bug on this, since it seems reasonable to vbe able to ask slon
to either not get too far ahead of itself or set a limit on the total
buffer size it uses.

>The approach that comes to mind is more of a "data hack"; to chop out
>that XID and apply it via "hacked" means.
>  
>
Thanks for this.