Restart TG apps for high mem-usage

[Toshio Kuratomi]

Yaakov Nemoy wrote:
> On Nov 25, 2007 4:00 PM, Toshio Kuratomi <a.badger at gmail.com> wrote:
>> Here's a short script to test our TG apps run via supervisor for
>> excessive memory usage and restart them if necessary.  We could run this
>> via cron in alternate hours on each app server.  Does this seem like a
>> good or bad idea to people?
>>
>> -Tosjio
> 
> +1, but does it make sure all transactions are finished?  I know smolt
> does not have good transaction protection.  If a transaction fails
> halfway through, we might have a mess.
> 
Not if the app doesn't.  From a brief test, TG apps do not do this.

The script is asking supervisor to shutdown the application.  supervisor 
sends a TERM to the TG app (we can configure it to send something other 
than TERM if we want but I don't see any documentation that leads me to 
believe it will be different with a HUP or QUIT).  At that point it 
looks like a TG app will immediately shutdown and rollback any current 
transactions.

smolt is on shaky ground if it's not using transactions correctly... At 
the beginning of the month when smolt was getting hit hard we did pretty 
much this same thing except manually instead of via a script when we 
noticed that smolt was giving timeouts and taking up 1G+ of RAM.  I 
think the current smolt code is using SQLAlchemy, correct?  It's pretty 
easy to use transactions so that you don't leave the db in an 
inconsistent state with that configuration.  Using the session's 
implicit transaction flushed just before the return should do the safe 
thing.  You can look through the code later and find additional places 
where you can safely flush the transaction if there's a need.

-Toshio