"Jason L. Buberel" <jason at buberel.org> writes:
> This configuration is running latest 1.2.11 slon on 8.2.4. After running a pg_bulkload command and starting slony back up, something that slony tried to do triggered
> the postmaster to restart suddenly. The listener for node 2 of a 3-node cluster starting up following the use of pg_bulkload to import 600k records:
> Sep 14 09:26:40 srv2 postgres-8.2[13372]: [3-2] STATEMENT:  SELECT pg_bulkload($1)
> Sep 14 09:26:57 srv2 sshd(pam_unix)[13389]: session opened for user jason by (uid=0)
> Sep 14 09:27:07 srv2 sshd(pam_unix)[13389]: session closed for user jason
> Sep 14 09:27:18 srv2 postgres-8.2[13490]: [2-1] NOTICE:  BULK LOAD START
> Sep 14 09:40:37 srv2 postgres-8.2[13490]: [3-1] NOTICE:  BULK LOAD END  (632832 records)
> Sep 14 09:40:37 srv2 postgres-8.2[13490]: [4-1] LOG:  duration: 799245.160 ms  execute <unnamed>: SELECT pg_bulkload($1)
> Sep 14 09:40:37 srv2 postgres-8.2[13490]: [4-2] DETAIL:  parameters: $1 = '/pgdata/8.2/sample_csv.ctl'
> then:
> Sep 14 10:48:18 srv2 slon[13841]: [27-1] 2007-09-14 10:48:18 CDT CONFIG main: configuration complete - starting threads
> Sep 14 10:48:18 srv2 postgres-8.2[13852]: [2-1] NOTICE:  Slony-I: cleanup stale sl_nodelock entry for pid=23070
> Sep 14 10:48:18 srv2 postgres-8.2[13852]: [3-1] NOTICE:  Slony-I: cleanup stale sl_nodelock entry for pid=1288
> Sep 14 10:48:18 srv2 postgres-8.2[13852]: [4-1] NOTICE:  Slony-I: cleanup stale sl_nodelock entry for pid=1292
> Sep 14 10:48:18 srv2 slon[13841]: [29-1] 2007-09-14 10:48:18 CDT CONFIG enableNode: no_id=1
> Sep 14 10:48:18 srv2 slon[13841]: [30-1] 2007-09-14 10:48:18 CDT CONFIG enableNode: no_id=3
> followed by:
> Sep 14 10:59:26 srv2 slon[13841]: [45-1] 2007-09-14 10:59:26 CDT WARN   cleanupThread: "select "_ar_primary".logswitch_weekly(); " - server closed the connection
> Sep 14 10:59:26 srv2 slon[13841]: [45-2]  unexpectedly
> Sep 14 10:59:26 srv2 slon[13841]: [45-3]        This probably means the server terminated abnormally
> Sep 14 10:59:26 srv2 slon[13841]: [45-4]        before or while processing the request.
> Sep 14 10:59:26 srv2 slon[13841]: [46-1] 2007-09-14 10:59:26 CDT FATAL  cleanupThread: "delete from "_ar_primary".sl_log_1 where log_origin = '2' and log_xid <
> Sep 14 10:59:26 srv2 slon[13841]: [46-2]  '151706861'; delete from "_ar_primary".sl_log_2 where log_origin = '2' and log_xid < '151706861'; delete from
> Sep 14 10:59:26 srv2 slon[13841]: [46-3]  "_ar_primary".sl_seqlog where seql_origin = '2' and seql_ev_seqno < '1686420'; select "_ar_primary".logswitch_finish(); " -
> Sep 14 10:59:26 srv2 postgres-8.2[25487]: [2-1] LOG:  server process (PID 13861) was terminated by signal 11
> Sep 14 10:59:26 srv2 postgres-8.2[25487]: [3-1] LOG:  terminating any other active server processes
> Sep 14 10:59:26 srv2 postgres-8.2[14136]: [5-1] WARNING:  terminating connection because of crash of another server process
> The postmaster is now running again, but I am afraid to restart slony again. Anything I can do to verify the integrity of the slon configuration that will not crash
> the postmaster?

logswitch_weekly() doesn't invoke any "wacky C stuff" that I'd expect
to be at all sensitive that way.  Indeed, it doesn't do much.

- It checks what the date is, to see if it's time to switch logs yet.

- It saves that time.

- It fiddles with the value of the sequence sl_log_status.

- And... nothing else.

This doesn't smell at all like a plausible cause for a sig11.

And the log doesn't tell us that it caused the problem - rather it
tells us that the problem terminated it.  The cleanup thread work was
being done on process 13841, terminated due to what was up with 13861.

I think you need to look, if you can, at what backend process 13861
was doing.  That, I believe, is the thing that's responsible for the
problem.
-- 
select 'cbbrowne' || '@' || 'linuxfinances.info';
http://www3.sympatico.ca/cbbrowne/finances.html
"I would rather spend 10 hours reading someone else's source code than
10  minutes listening  to Musak  waiting for  technical  support which
isn't." -- Dr. Greg Wettstein, Roger Maris Cancer Center