Which of the changes introduced in F21 might affect bash history?

[Alexander Ploumistos]

Hello,

I have upgraded 3 systems to a nonproduct F21 and I have installed F21
to a couple of others using the x86_64 Live Workstation image. They
all suffer from intermittent losses of bash history. When the machines
are booted or rebooted, more than half the times, there are commands
missing from .bash_history files. It's not only that the previous
sessions have not been appended to the files, entire portions of the
files are wiped on occasion, e.g. history files that were more than
2000 lines long are now reduced to just a couple of commands.

There was a bug filed against systemd during F21 testing (#1170765),
where I followed up, but I have no idea if systemd really is to blame
or if there is something else that could be touching history files. I
have read the release notes as well as the system administrator's
guide, but couldn't find anything relevant. Does anyone have any idea
what might be off here? Which system components are involved in
terminal history management?

Regards

[Florian Weimer]

On 02/16/2015 09:34 AM, Alexander Ploumistos wrote:
>  I have upgraded 3 systems to a nonproduct F21 and I have installed
> F21
>  to a couple of others using the x86_64 Live Workstation image. They
>  all suffer from intermittent losses of bash history. When the machines
>  are booted or rebooted, more than half the times, there are commands
>  missing from .bash_history files. It's not only that the previous
>  sessions have not been appended to the files, entire portions of the
>  files are wiped on occasion, e.g. history files that were more than
>  2000 lines long are now reduced to just a couple of commands. 
I see something similar, but without rebooting, just by opening and
closing shells.  A lot of the history is suddenly missing, even in
running shells (probably because it is re-read from disk, and something
goes wrong with the merging).

-- 
Florian Weimer / Red Hat Product Security

[drago01]

On Mon, Feb 16, 2015 at 9:34 AM, Alexander Ploumistos
<alex.ploumistos(a)gmail.com&gt; wrote:
>  Hello,
>
>  I have upgraded 3 systems to a nonproduct F21 and I have installed F21
>  to a couple of others using the x86_64 Live Workstation image. They
>  all suffer from intermittent losses of bash history. When the machines
>  are booted or rebooted, more than half the times, there are commands
>  missing from .bash_history files. It's not only that the previous
>  sessions have not been appended to the files, entire portions of the
>  files are wiped on occasion, e.g. history files that were more than
>  2000 lines long are now reduced to just a couple of commands.
>
>  There was a bug filed against systemd during F21 testing (#1170765),
>  where I followed up, but I have no idea if systemd really is to blame
>  or if there is something else that could be touching history files. I
>  have read the release notes as well as the system administrator's
>  guide, but couldn't find anything relevant. Does anyone have any idea
>  what might be off here? Which system components are involved in
>  terminal history management? 
Bash ... the last closes session "wins" so if you have multiple shells
open one might overwrite the history of the other one.

[Reindl Harald]

Am 16.02.2015 um 10:44 schrieb drago01:
>  On Mon, Feb 16, 2015 at 9:34 AM, Alexander Ploumistos wrote:
> > I have upgraded 3 systems to a nonproduct F21 and I have installed F21
> > to a couple of others using the x86_64 Live Workstation image. They
> > all suffer from intermittent losses of bash history. When the machines
> > are booted or rebooted, more than half the times, there are commands
> > missing from .bash_history files. It's not only that the previous
> > sessions have not been appended to the files, entire portions of the
> > files are wiped on occasion, e.g. history files that were more than
> > 2000 lines long are now reduced to just a couple of commands.
> >
> > There was a bug filed against systemd during F21 testing (#1170765),
> > where I followed up, but I have no idea if systemd really is to blame
> > or if there is something else that could be touching history files. I
> > have read the release notes as well as the system administrator's
> > guide, but couldn't find anything relevant. Does anyone have any idea
> > what might be off here? Which system components are involved in
> > terminal history management?
>
>  Bash ... the last closes session "wins" so if you have multiple shells
>  open one might overwrite the history of the other one 
but that is not new and not the same people describe in this thread

[Alexander Ploumistos]

On Mon, Feb 16, 2015 at 11:44 AM, drago01 <drago01(a)gmail.com&gt; wrote:
>  Bash ... the last closes session "wins" so if you have
> multiple shells
>  open one might overwrite the history of the other one. 
Up to F20, no matter how many terminals or tabs I had open, everything
found its way to bash history. This problem happens with a single
terminal being open, with many terminals and tabs, with remote ssh
sessions and with gnu screen. It really makes no difference.

[John Florian]

>  -----Original Message-----
>  From: devel-bounces(a)lists.fedoraproject.org [mailto:devel-
>  bounces(a)lists.fedoraproject.org] On Behalf Of drago01
>  Sent: Monday, February 16, 2015 04:45
>  To: Development discussions related to Fedora
>  Subject: Re: Which of the changes introduced in F21 might affect bash
>  history?
>  
>  On Mon, Feb 16, 2015 at 9:34 AM, Alexander Ploumistos
>  <alex.ploumistos(a)gmail.com&gt; wrote:
>  > Hello,
>  >
>  > I have upgraded 3 systems to a nonproduct F21 and I have installed F21
>  > to a couple of others using the x86_64 Live Workstation image. They
>  > all suffer from intermittent losses of bash history. When the machines
>  > are booted or rebooted, more than half the times, there are commands
>  > missing from .bash_history files. It's not only that the previous
>  > sessions have not been appended to the files, entire portions of the
>  > files are wiped on occasion, e.g. history files that were more than
>  > 2000 lines long are now reduced to just a couple of commands.
>  >
>  > There was a bug filed against systemd during F21 testing (#1170765),
>  > where I followed up, but I have no idea if systemd really is to blame
>  > or if there is something else that could be touching history files. I
>  > have read the release notes as well as the system administrator's
>  > guide, but couldn't find anything relevant. Does anyone have any idea
>  > what might be off here? Which system components are involved in
>  > terminal history management?
>  
>  Bash ... the last closes session "wins" so if you have multiple shells
>  open one might overwrite the history of the other one.
>  

True, unless you have "shopt -s histappend".  I have that and like others
started seeing unexplained losses starting w/ F21. I have so many fires going right now
that I wasn't sure if I was just imagining this or not.  I'm sort of glad to hear
I'm not alone.


On another almost related note, I've never had any luck convincing bash to honor my
desire for a very large HISTSIZE despite unsetting HISTFILESIZE but I've not looked
into that issue for several years.  I'd love to dig up commands I used several years
ago even if I have to grep an auxiliary history file; I don't mind dedicated several
MB to this.  I mostly made the problem go away by starting a personal wiki but that
requires more effort.

--
John Florian

[Kevin Fenzi]

Perhaps this post from AdamW would be related: 

https://www.happyassassin.net/2015/01/16/bash-history-with-multiple-sessi...

kevin

[John Florian]

>  -----Original Message-----
>  From: devel-bounces(a)lists.fedoraproject.org [mailto:devel-
>  bounces(a)lists.fedoraproject.org] On Behalf Of Kevin Fenzi
>  Sent: Monday, February 16, 2015 09:25
>  To: devel(a)lists.fedoraproject.org
>  Subject: Re: Which of the changes introduced in F21 might affect bash
>  history?
>  
>  Perhaps this post from AdamW would be related:
>  
>  https://www.happyassassin.net/2015/01/16/bash-history-with-multiple-
>  sessions/
>  
>  kevin 
Thanks for that Kevin!  When I last attempted defeating bash's history trimming I had
tried to beat it at its own game by migrating a healthy chunk of ~/.bash_history into my
own auxiliary file during login so that whatever I did in that session would be
(hopefully) less than what had been archived therefore making it unnecessary for bash to
do anything.  It sort of worked and why it sort of failed was due to a bug I just
couldn't see or due to a bad choice in that chunk size I never learned -- it just
never bothered me quite enough to sort it all out.

I really like this prompt_command() idea; that's very clever.  Thank you (and AdamW)
for sharing that!


--
John Florian

[Alexander Ploumistos]

I can't believe I had missed this, but one difference I've found is
that F21 has moved to bash-4.3, while F20 is using 4.2. Then I came
across this post by Chet Ramey, the maintainer of bash, and I got
confused:
http://lists.gnu.org/archive/html/bug-bash/2013-07/msg00096.html

[Tomasz Torcz]

On Mon, Feb 16, 2015 at 10:34:48AM +0200, Alexander Ploumistos wrote:
>  There was a bug filed against systemd during F21 testing (#1170765),
>  where I followed up, but I have no idea if systemd really is to blame
>  or if there is something else that could be touching history files. I
>  have read the release notes as well as the system administrator's
>  guide, but couldn't find anything relevant. Does anyone have any idea
>  what might be off here? Which system components are involved in
>  terminal history management? 
  systemd only involvement seem to be killing processes in user
session being closed.  Maybe bash is killed before it flushes
history?

-- 
Tomasz   .. oo o.   oo o. .o   .o o. o. oo o.   ..
Torcz    .. .o .o   .o .o oo   oo .o .. .. oo   oo
o.o.o.   .o .. o.   o. o. o.   o. o. oo .. ..   o.

[Alexander Ploumistos]

On Mon, Feb 16, 2015 at 12:06 PM, Tomasz Torcz <tomek(a)pipebreaker.pl&gt; wrote:
>    systemd only involvement seem to be killing processes in user
>  session being closed.  Maybe bash is killed before it flushes
>  history? 
This would make sense if the problem only involved session history not
getting appended (which does happen), but we are also seeing user and
root .bash_history files randomly losing parts of their contents or
even being emptied.

[Stephen John Smoogen]

On 16 February 2015 at 01:34, Alexander Ploumistos <
alex.ploumistos(a)gmail.com&gt; wrote:

>  Hello,
>
>  I have upgraded 3 systems to a nonproduct F21 and I have installed F21
>  to a couple of others using the x86_64 Live Workstation image. They
>  all suffer from intermittent losses of bash history. When the machines
>  are booted or rebooted, more than half the times, there are commands
>  missing from .bash_history files. It's not only that the previous
>  sessions have not been appended to the files, entire portions of the
>  files are wiped on occasion, e.g. history files that were more than
>  2000 lines long are now reduced to just a couple of commands.
>
>  There was a bug filed against systemd during F21 testing (#1170765),
>  where I followed up, but I have no idea if systemd really is to blame
>  or if there is something else that could be touching history files. I
>  have read the release notes as well as the system administrator's
>  guide, but couldn't find anything relevant. Does anyone have any idea
>  what might be off here? Which system components are involved in
>  terminal history management?
>
>  
Could you see if either of these are set?

#export HISTCONTROL=ignoredups:erasedups
#export HISTTIMEFORMAT="%h %d %H:%M:%S> "

I started seeing this in RHEL-6 bash after I took this from one of my
fedora boxes to get a common bashrc

I commented them out and the problem with bash losing history every now and
then went away. I figured that one of the control items was not ready for
primetime and forgot to follow up furhter.


-- 
Stephen J Smoogen.

[Alexander Ploumistos]

On Mon, Feb 16, 2015 at 7:41 PM, Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  Could you see if either of these are set?
>
>  #export HISTCONTROL=ignoredups:erasedups
>  #export HISTTIMEFORMAT="%h %d %H:%M:%S> " 
Nope, nowhere. I have only a few aliases set and that's it. Fedora
defaults always worked for me.

After reading the changelog for bash-4.3 and picking up the trail from
Adam Williamson's blog post and bug report up to the systemd-devel
list, it seems that indeed systemd is the culprit (or cgroups
notifications in the kernel, depending on who you might ask). Does
this warrant an interim solution (e.g. deploying the script with
prompt_command) until it is fixed? Should we let bash poc (Ondrej
Oprala) know about it?