Curious Mystery Of Killed Process

[Thomas Dineen]

Gentle People:

I have a question about "The Curious Mystery Of Killed Process At 4:56 AM"?
For some while now I have been developing a optimization application, which
depending on the machine may run for several days.

    The development process has played out over a month or more of 
testing, finding many bugs
including several Segmentation Fault Crashes, which are usually found 
and fixed
with the help of GDB.

    Now I have a new one: After running for several days the application 
terminates,
and prints "Killed" in the shell window. Now know this did not likely 
occur via the
keyboard because of physical security. Can a process kill itself? Is 
this simply caused
by a bug which must be chased down? Could this be caused by a shortage 
of resources?
Memory? Threads? Is there a limit on the number of threads?
I am running GCC 4.4.7 on CentOS 7.0

Thank You for any thoughts.

Thomas Dineen

[Jon LaBadie]

On Thu, Jul 30, 2020 at 02:52:16PM -0700, Thomas Dineen wrote:
>  Gentle People:
>  
>  I have a question about "The Curious Mystery Of Killed Process At 4:56
>  AM"?  For some while now I have been developing a optimization
>  application, which depending on the machine may run for several days.
>  
>     The development process has played out over a month or more of
>  testing, finding many bugs including several Segmentation Fault
>  Crashes, which are usually found and fixed with the help of GDB.
>  
>     Now I have a new one: After running for several days the
>  application terminates, and prints "Killed" in the shell window. Now
>  know this did not likely occur via the keyboard because of physical
>  security. Can a process kill itself? Is this simply caused by a bug
>  which must be chased down? Could this be caused by a shortage of
>  resources?  Memory? Threads? Is there a limit on the number of
>  threads? 
I was thinking of user limits.  Check the "ulimit" command
for a list of limits imposed on user processes.  For example,
my current open files limit is 1024.  Are you opening files
and not closing unneeded ones?

A limit new to me is "signals pending".  Might you be ignoring
signals to avoid being killed?  Maybe something is generating
signals to your process and they queued until the limit was
exceeded.

jon
-- 
Jon H. LaBadie                  jonfu(a)jgcomp.com

[Samuel Sieb]

On 7/30/20 2:52 PM, Thomas Dineen wrote:
>      Now I have a new one: After running for several days the
> application 
>  terminates,
>  and prints "Killed" in the shell window. Now know this did not likely 
>  occur via the
>  keyboard because of physical security. Can a process kill itself? Is 
>  this simply caused
>  by a bug which must be chased down? Could this be caused by a shortage 
>  of resources?
>  Memory? Threads? Is there a limit on the number of threads?
>  I am running GCC 4.4.7 on CentOS 7.0 
This is the Fedora list, CentOS questions should be asked on their list. 
  CentOS 7.0 is also really old.

"Killed" is generally a hard kill from something outside the process. 
Check the journal and audit logs to see if there are any messages about 
this.  Is it always at that time?

[George N. White III]

On Thu, 30 Jul 2020 at 19:49, Thomas Dineen <tdineen(a)ix.netcom.com&gt; wrote:

>  Gentle People:
>
>  I have a question about "The Curious Mystery Of Killed Process At 4:56 AM"?
>  For some while now I have been developing a optimization application, which
>  depending on the machine may run for several days.
>
>      The development process has played out over a month or more of
>  testing, finding many bugs
>  including several Segmentation Fault Crashes, which are usually found
>  and fixed
>  with the help of GDB.
>
>      Now I have a new one: After running for several days the application
>  terminates,
>  and prints "Killed" in the shell window. Now know this did not likely
>  occur via the
>  keyboard because of physical security. Can a process kill itself? Is
>  this simply caused
>  by a bug which must be chased down? Could this be caused by a shortage
>  of resources?
>  Memory? Threads? Is there a limit on the number of threads?
>  I am running GCC 4.4.7 on CentOS 7.0
>  
There should be more details in dmesg and log files.   OOM Killer is a
prime suspect.


-- 
George N. White III