Upgrade metrics

[Axel Thimm]

Hi,

is it possible to create some relative metrics of release upgrades,
e.g. FC5 -> FC6, FC5 -> F7?

The interesting part is to create a graph over time to answer the
question whether the 13 month EOL is really paying off. And perhaps on
base of that graph one could revise this decision. At the very leats
it would be interesting to notice the upgrade behaviour of Fedora
users.

I have no idea how to do that other than marking the HTTP Agent of yum
on F<N> accordingly, as well as the HTTP Agent in anaconda's yum
(noting fresh installs vs upgrades). Worth to add this for F8?
-- 
Axel.Thimm at ATrpms.net

[Freddie Rosario]

Axel,

I believe that using data from smolt (http://smolt.fedoraproject.org/stats)
and plugging that data into something like a round robin database(
http://oss.oetiker.ch/rrdtool/) might be the solution you are looking for.



On 6/15/07, Axel Thimm <Axel.Thimm(a)atrpms.net&gt; wrote:
>  Hi,
>
>  is it possible to create some relative metrics of release upgrades,
>  e.g. FC5 -> FC6, FC5 -> F7?
>
>  The interesting part is to create a graph over time to answer the
>  question whether the 13 month EOL is really paying off. And perhaps on
>  base of that graph one could revise this decision. At the very leats
>  it would be interesting to notice the upgrade behaviour of Fedora
>  users.
>
>  I have no idea how to do that other than marking the HTTP Agent of yum
>  on F<N> accordingly, as well as the HTTP Agent in anaconda's yum
>  (noting fresh installs vs upgrades). Worth to add this for F8?
>  --
>  Axel.Thimm at ATrpms.net
>
>  _______________________________________________
>  Fedora-infrastructure-list mailing list
>  Fedora-infrastructure-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
>
>
>  

-- 
--Freddie

[Bill Nottingham]

Axel Thimm (Axel.Thimm(a)ATrpms.net) said: 
>  The interesting part is to create a graph over time to answer the
>  question whether the 13 month EOL is really paying off. And perhaps on
>  base of that graph one could revise this decision. At the very leats
>  it would be interesting to notice the upgrade behaviour of Fedora
>  users.
>  
>  I have no idea how to do that other than marking the HTTP Agent of yum
>  on F<N> accordingly, as well as the HTTP Agent in anaconda's yum
>  (noting fresh installs vs upgrades). Worth to add this for F8? 
Well, you could have smolt update the release for a particular hw
UUID, and track things that way.

Bill

[Mike McGrath]

Axel Thimm wrote:
>  Hi,
>
>  is it possible to create some relative metrics of release upgrades,
>  e.g. FC5 -> FC6, FC5 -> F7?
>
>     
We didn't really start taking metrics until FC6.
>  The interesting part is to create a graph over time to answer the
>  question whether the 13 month EOL is really paying off. And perhaps on
>  base of that graph one could revise this decision. At the very leats
>  it would be interesting to notice the upgrade behaviour of Fedora
>  users.
>
>  I have no idea how to do that other than marking the HTTP Agent of yum
>  on F<N> accordingly, as well as the HTTP Agent in anaconda's yum
>  (noting fresh installs vs upgrades). Worth to add this for F8?
>    
>     
So here's a graph of the smolt vs yum ip stuff:

https://admin.fedoraproject.org/cacti/graph.php?action=view&rra_id=al...

The problem with the smolt numbers is that anyone doing kickstarts, 
installing via runlevel 3, yum upgrades or anaconda upgrades will never 
get prompted to enable smolt and there are privacy concerns just 
enabling it by default.  So I think the only way we can do it is start 
to seed people now, get the word out about smolt, and start monitoring 
upgrades from F7 to F8 and so on.

Adding a query string or changing the http agent on yum could also be 
useful.  It did just dawn on me that I've stopped monitoring FC6 yum 
connections and perhaps I should re-enable it.  The scary thing is I 
still regularly get requests regarding Fedora Core 1!!  (we moved it 
from the primary mirror to 
http://archives.fedoraproject.org/fedora/linux/core/1/ and that brought 
some people out of the woodwork)

I'm always open to suggestions for changes to smolt or our yum method.

    -Mike

[Nils Breunese]

Mike McGrath wrote:

>  It did just dawn on me that I've stopped monitoring FC6 yum  
>  connections and perhaps I should re-enable it.  The scary thing is  
>  I still regularly get requests regarding Fedora Core 1!!  (we moved  
>  it from the primary mirror to http://archives.fedoraproject.org/ 
>  fedora/linux/core/1/ and that brought some people out of the woodwork) 
I know lots of server providers are still having customers on FC1-4.  
Some people really don't seem to care about getting patches. I was  
asked to look at a server this week, but they didn't tell me exactly  
what it was running. So the first thing I do is 'ca /etc/redhat- 
release'. Yup, FC1...

Nils Breunese.

[Christopher Blizzard]

On Fri, 2007-06-15 at 08:54 -0500, Mike McGrath wrote:
>  
>  The problem with the smolt numbers is that anyone doing kickstarts, 
>  installing via runlevel 3, yum upgrades or anaconda upgrades will
>  never 
>  get prompted to enable smolt and there are privacy concerns just 
>  enabling it by default.  So I think the only way we can do it is
>  start 
>  to seed people now, get the word out about smolt, and start
>  monitoring 
>  upgrades from F7 to F8 and so on.
>   
What happens in the case that the network stuff isn't up when smolt runs
during startup?  Did it end up being able to wait for the network and
queue it for later?  I'm a little surprised that the number of units
that we have is as small as it is, and how few of them are laptops.  Or
maybe it's just that our support for laptops is so bad that no one is
bothering anymore. :)

Would be nice to move smolt out of just the firstboot and do some user
surveys as well, but I guess we haven't speced that out and haven't had
time to do the work to make that happen.

--Chris

[Mike McGrath]

Christopher Blizzard wrote:
>  On Fri, 2007-06-15 at 08:54 -0500, Mike McGrath wrote:
>    
> > The problem with the smolt numbers is that anyone doing kickstarts, 
> > installing via runlevel 3, yum upgrades or anaconda upgrades will
> > never 
> > get prompted to enable smolt and there are privacy concerns just 
> > enabling it by default.  So I think the only way we can do it is
> > start 
> > to seed people now, get the word out about smolt, and start
> > monitoring 
> > upgrades from F7 to F8 and so on.
> >
> >     
>
>  What happens in the case that the network stuff isn't up when smolt runs
>  during startup?  Did it end up being able to wait for the network and
>  queue it for later?  I'm a little surprised that the number of units
>  that we have is as small as it is, and how few of them are laptops.  Or
>  maybe it's just that our support for laptops is so bad that no one is
>  bothering anymore. :)
>     
It keeps trying as best as it can until network comes up.  If all that 
fails (say a reboot is required before network comes up) the monthly 
smolt checkin will start up and we'll get them a month after they 
install.  The monthly checkin will also let us do queries on 'active' 
users say people that have submitted in the last 6 months or 12 months.

>  Would be nice to move smolt out of just the firstboot and do some
> user
>  surveys as well, but I guess we haven't speced that out and haven't had
>  time to do the work to make that happen.
>     
Well we do have smoltGui available and we're working on adding a rating 
system for the entire profile as well as individual devices.  The 
question is how to get people to run it.

    -Mike

    -Mike

[Paulo Santos]

Wouldnt a small popup like the advise of new updates be usefull for smoltGui
?

Something like:
- "New survey is available"
- you click in it, do the survey and send it with the profile, it would also
be nice to have a checkbox to disable the announcements.

Paulo

On 6/15/07, Mike McGrath <mmcgrath(a)redhat.com&gt; wrote:
>  Christopher Blizzard wrote:
>  > On Fri, 2007-06-15 at 08:54 -0500, Mike McGrath wrote:
>  >
>  >> The problem with the smolt numbers is that anyone doing kickstarts,
>  >> installing via runlevel 3, yum upgrades or anaconda upgrades will
>  >> never
>  >> get prompted to enable smolt and there are privacy concerns just
>  >> enabling it by default.  So I think the only way we can do it is
>  >> start
>  >> to seed people now, get the word out about smolt, and start
>  >> monitoring
>  >> upgrades from F7 to F8 and so on.
>  >>
>  >>
>  >
>  > What happens in the case that the network stuff isn't up when smolt runs
>  > during startup?  Did it end up being able to wait for the network and
>  > queue it for later?  I'm a little surprised that the number of units
>  > that we have is as small as it is, and how few of them are laptops.  Or
>  > maybe it's just that our support for laptops is so bad that no one is
>  > bothering anymore. :)
>  >
>  It keeps trying as best as it can until network comes up.  If all that
>  fails (say a reboot is required before network comes up) the monthly
>  smolt checkin will start up and we'll get them a month after they
>  install.  The monthly checkin will also let us do queries on 'active'
>  users say people that have submitted in the last 6 months or 12 months.
>
>  > Would be nice to move smolt out of just the firstboot and do some user
>  > surveys as well, but I guess we haven't speced that out and haven't had
>  > time to do the work to make that happen.
>  >
>
>  Well we do have smoltGui available and we're working on adding a rating
>  system for the entire profile as well as individual devices.  The
>  question is how to get people to run it.
>
>      -Mike
>
>      -Mike
>
>  _______________________________________________
>  Fedora-infrastructure-list mailing list
>  Fedora-infrastructure-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
>

[Christopher Blizzard]

On Fri, 2007-06-15 at 18:29 +0200, Paulo Santos wrote:
>  
>  Something like:
>  - "New survey is available"
>  - you click in it, do the survey and send it with the profile, it
>  would also be nice to have a checkbox to disable the announcements. 
>   
That's a good idea.  I always worried about how to handle surveys on the
client side because that's where the data is.  But if you connect it to
the smolt data on the server then you can measure based off that.

The thing about surveys, though, is that they are connected to the
current revisions of some of the software on the machine and not
connected to the hardware profile.  More the combination, and they
change over time.  What we want to be able to do is measure the
experience with that combination and then measure our improvements over
time.

I get the impression that there's a one to one mapping from the hardware
profile to UUID and that you can change the entire profile and the old
data that was in there was lost.  (Strangely enough when I looked up my
UUID for this machine it was clearly someone else's machine.  I ran
smolt and stepped on it.  I was a little surprised that we had a
collision there.)

--Chris

[Christopher Blizzard]

On Fri, 2007-06-15 at 10:51 -0500, Mike McGrath wrote:
>  
>  It keeps trying as best as it can until network comes up.  If all
>  that 
>  fails (say a reboot is required before network comes up) the monthly 
>  smolt checkin will start up and we'll get them a month after they 
>  install.  The monthly checkin will also let us do queries on 'active' 
>  users say people that have submitted in the last 6 months or 12
>  months. 
Does it do things like reload dns when the network config changes?
Because, tee hee, glibc does not handle this kind of thing for you.

>  
>  Well we do have smoltGui available and we're working on adding a
>  rating 
>  system for the entire profile as well as individual devices.  The 
>  question is how to get people to run it.
>   
Who is working on that?  You know, so I can stick my little nose into
it.

--Chris

[Mike McGrath]

Christopher Blizzard wrote:
>  On Fri, 2007-06-15 at 10:51 -0500, Mike McGrath wrote:
>    
> > It keeps trying as best as it can until network comes up.  If all
> > that 
> > fails (say a reboot is required before network comes up) the monthly 
> > smolt checkin will start up and we'll get them a month after they 
> > install.  The monthly checkin will also let us do queries on 'active' 
> > users say people that have submitted in the last 6 months or 12
> > months.
> >     
>
>  Does it do things like reload dns when the network config changes?
>  Because, tee hee, glibc does not handle this kind of thing for you.
>
>    
> > Well we do have smoltGui available and we're working on adding a
> > rating 
> > system for the entire profile as well as individual devices.  The 
> > question is how to get people to run it.
> >
> >     
>
>  Who is working on that?  You know, so I can stick my little nose into
>  it.
>     
No one right now but one of the interns has expressed interest.  He's 
already helped me get the client running on Debian and Ubuntu.  The 
ratings system was the next thing.  Jcollie had a prototype for the 
client IIRC as well.  If you are interested or no someone who's 
interested let me know.  Its just been a matter of not having enough 
time for me to get it in.

    -Mike

[Axel Thimm]

On Fri, Jun 15, 2007 at 08:54:53AM -0500, Mike McGrath wrote:
>  Axel Thimm wrote:
>  >Hi,
>  >
>  >is it possible to create some relative metrics of release upgrades,
>  >e.g. FC5 -> FC6, FC5 -> F7?
>  >
>  >  
>  We didn't really start taking metrics until FC6. 
Yes, I know, I was just giving examples of where it would had been
useful in today's context. Testing 13 month life span would probable
be better done in the F8 release time, e.g. checking how many FC6
users really upgrade to F8 skipping one release (which is what the 13
month span was about).

If we need FC6 to send out more info in some way (smolt, yum http
agent etc) we should implant that early in both FC6 and F7, so we know
in autumn what people are really doing (always in relative numbers,
e.g. FC6->F8 upgrades in comparison to F7->F8 upgrades and F8 install
from scratch)

>  >The interesting part is to create a graph over time to answer
> the
>  >question whether the 13 month EOL is really paying off. And perhaps on
>  >base of that graph one could revise this decision. At the very leats
>  >it would be interesting to notice the upgrade behaviour of Fedora
>  >users.
>  >
>  >I have no idea how to do that other than marking the HTTP Agent of yum
>  >on F<N> accordingly, as well as the HTTP Agent in anaconda's yum
>  >(noting fresh installs vs upgrades). Worth to add this for F8?
>  >  
>  >  
>  So here's a graph of the smolt vs yum ip stuff:
>  
>
> https://admin.fedoraproject.org/cacti/graph.php?action=view&rra_id=al...
Nice graph! This gives us an estimate of absolute numbers, but not yet
upgrades vs bare metal installs. It nicely shows that smolt and
yum data are coherent.

>  The problem with the smolt numbers is that anyone doing kickstarts, 
>  installing via runlevel 3, yum upgrades or anaconda upgrades will never 
>  get prompted to enable smolt and there are privacy concerns just 
>  enabling it by default. 
That doesn't matter, we won't get everyone and there will be an
unknown dark factor. But the numbers we get can be divided to remove
that unknown constant, e.g. create relative numbers. For example:

"In the first 4 weeks after the release of F8 and until EOL of FC6 we
saw that:

   70% did a bare metal install
   20% upgraded from F7
    9% upgraded from FC6
    1% did something else (like upgrading from RHL9)"

These are the numbers we would need to justify a 13 month EOL time
including all efforts that go with it, or whether it wasn't worth it
because say only 1% would make really use for FC6->F8 upgrades and we
better invest that man time into something else.

also if these numbers can be seen over time we can spot when the user
do the bare metal installs and the upgrades. I would guess that after
a release one would have a large bare-metal-install peak and some time
after that the upgrades would slowly grow as well.

>  The scary thing is I still regularly get requests regarding Fedora
>  Core 1!!  (we moved it from the primary mirror to
>  http://archives.fedoraproject.org/fedora/linux/core/1/ and that
>  brought some people out of the woodwork) 
Wow, that's really scary. FC1 should of course live forever in
archives.fp.o for historical reason, but actually using it is really
questionable :)
-- 
Axel.Thimm at ATrpms.net

[Mike McGrath]

Axel Thimm wrote:
> > So here's a graph of the smolt vs yum ip stuff:
> >
> >
> https://admin.fedoraproject.org/cacti/graph.php?action=view&rra_id=al...
> >     
>
>  Nice graph! This gives us an estimate of absolute numbers, but not yet
>  upgrades vs bare metal installs. It nicely shows that smolt and
>  yum data are coherent.
>
>     
How do you get this info again?  I swear there was a place in lshal that 
used to say install source (network, cd) and install method (kickstart, 
upgrade) though I can't seem to find it now and I'm wondering if I made 
that up.

    -Mike

[Ask Bjørn Hansen]

On Jun 15, 2007, at 8:25, Axel Thimm wrote:

>  Wow, that's really scary. FC1 should of course live forever in
>  archives.fp.o for historical reason, but actually using it is really
>  questionable :) 
So you wouldn't approve of my RHL7 boxes?  :-)

Those particular boxes are so old (both software and hardware) that  
we never install anything new, just slowly migrate stuff away.   Even  
on our RHEL3 boxes we tend not to install anything new (the software  
is too old and annoying), but all the old stuff is Just Fine when  
things are already running.   (Safer on RHEL3 than RHL7 obviously).


  - ask

-- 
http://develooper.com/ - http://askask.com/