Shipping package metadata in a package

[Richard Hughes]

Hi all,

I'm after some advice / ideas. When you install Fedora 20/21 and then
launch gnome-software it has to go and download some metadata before
it can show anything. This is a pretty bad first experience,
considering subsequent runs of gnome-software just open straight away.
GNOME Software operates on the logic that *any* version of the
metadata is better than nothing, and only refreshes once a week when
the user is idle.

There are two ways to fix the jarring UX. We could either ship the
fedora package metadata pre-prepared in PackageKit, maybe using
something like %ghost so the new metadata is ignored. The other way is
to start the metadata downloading in the initial-start wizard thing,
although in practice the download takes a couple of minutes to
complete, and the wizard typically takes much less than that before
the user has configured everything. The downside to shipping prepared
metadata is that the package size is larger, and that the metadata
would be *very* out of date after a year or so.

I'm open to ideas, thanks.

Richard.

[Mathieu Bridon]

Hi,

On Fri, 2014-01-24 at 10:01 +0000, Richard Hughes wrote:
>  There are two ways to fix the jarring UX. We could either ship the
>  fedora package metadata pre-prepared in PackageKit, maybe using
>  something like %ghost so the new metadata is ignored. The other way is
>  to start the metadata downloading in the initial-start wizard thing,
>  although in practice the download takes a couple of minutes to
>  complete, and the wizard typically takes much less than that before
>  the user has configured everything. The downside to shipping prepared
>  metadata is that the package size is larger, and that the metadata
>  would be *very* out of date after a year or so. 
Rather than during the initial setup thing, why not start it at first
boot, much earlier?

Another option would be to have it done as the last step of the
installer.


-- 
Mathieu

[Richard Hughes]

On 24 January 2014 10:03, Mathieu Bridon <bochecha(a)fedoraproject.org&gt; wrote:
>  Rather than during the initial setup thing, why not start it at
> first
>  boot, much earlier? 
We need to wait until the user has setup a network connection.

>  Another option would be to have it done as the last step of the
>  installer. 
Won't that download the metadata onto the live image, not the fresh install?

Richard.

[Adam Williamson]

Richard Hughes <hughsient(a)gmail.com&gt; wrote:
> On 24 January 2014 10:03, Mathieu Bridon
> <bochecha(a)fedoraproject.org&gt;
> wrote:
> > Rather than during the initial setup thing, why not start it at first
> > boot, much earlier?
>
> We need to wait until the user has setup a network connection.
>
> > Another option would be to have it done as the last step of the
> > installer.
>
> Won't that download the metadata onto the live image, not the fresh
> install?
>
> Richard.
> -- 
> devel mailing list
> devel(a)lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/devel
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct 
Not necessarily, the live installer is of course capable of deploying stuff into the
installed system.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin DOT net
http://www.happyassassin.net

[Vít Ondruch]

Dne 24.1.2014 11:03, Mathieu Bridon napsal(a):
>  Hi,
>
>  On Fri, 2014-01-24 at 10:01 +0000, Richard Hughes wrote:
> > There are two ways to fix the jarring UX. We could either ship the
> > fedora package metadata pre-prepared in PackageKit, maybe using
> > something like %ghost so the new metadata is ignored. The other way is
> > to start the metadata downloading in the initial-start wizard thing,
> > although in practice the download takes a couple of minutes to
> > complete, and the wizard typically takes much less than that before
> > the user has configured everything. The downside to shipping prepared
> > metadata is that the package size is larger, and that the metadata
> > would be *very* out of date after a year or so.
>  Rather than during the initial setup thing, why not start it at first
>  boot, much earlier? 
Not sure I like the idea. I install the system, eager to start using it,
but in background will run some service downloading some data, which
will make my system sluggish. That would be user experience far from
ideal ...


Vít

>  Another option would be to have it done as the last step of the
>  installer.
>
>

[Richard Hughes]

On 24 January 2014 12:20, Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
>  Not sure I like the idea. I install the system, eager to start using
> it,
>  but in background will run some service downloading some data, which
>  will make my system sluggish. That would be user experience far from
>  ideal ... 
In this instance, when run with background attribues, PackageKit uses
idle bandwidth and with a lower priority than if the transaction was a
foreground task.

Richard.

[Vít Ondruch]

Dne 24.1.2014 15:05, Richard Hughes napsal(a):
>  On 24 January 2014 12:20, Vít Ondruch <vondruch(a)redhat.com&gt;
> wrote:
> > Not sure I like the idea. I install the system, eager to start using it,
> > but in background will run some service downloading some data, which
> > will make my system sluggish. That would be user experience far from
> > ideal ...
>  In this instance, when run with background attribues, PackageKit uses
>  idle bandwidth and with a lower priority than if the transaction was a
>  foreground task.
>
>  Richard. 

Well, if there is more of such tasks .... I wish there is system which
would allow to queue such "unwanted" tasks, and execute them later after
start one by one.



Vít

[Miloslav Trmač]

On Fri, Jan 24, 2014 at 3:05 PM, Richard Hughes <hughsient(a)gmail.com&gt; wrote:

>  In this instance, when run with background attribues, PackageKit
> uses
>  idle bandwidth and with a lower priority than if the transaction was a
>  foreground task.
>  
Huh, we have a concept of "idle bandwidth"?  Who/how manages this?
    Mirek

[Tomasz Torcz]

On Fri, Jan 24, 2014 at 05:50:02PM +0100, Miloslav Trmač wrote:
>  On Fri, Jan 24, 2014 at 3:05 PM, Richard Hughes
> <hughsient(a)gmail.com&gt; wrote:
>  
>  > In this instance, when run with background attribues, PackageKit uses
>  > idle bandwidth and with a lower priority than if the transaction was a
>  > foreground task.
>  >
>  
>  Huh, we have a concept of "idle bandwidth"?  Who/how manages this? 
  TCP/IP manages.  There is "Low Priority" congestion algorithm implemented
by Linux.

-- 
Tomasz   .. oo o.   oo o. .o   .o o. o. oo o.   ..
Torcz    .. .o .o   .o .o oo   oo .o .. .. oo   oo
o.o.o.   .o .. o.   o. o. o.   o. o. oo .. ..   o.

[Simo Sorce]

On Fri, 2014-01-24 at 10:01 +0000, Richard Hughes wrote:
>  Hi all,
>  
>  I'm after some advice / ideas. When you install Fedora 20/21 and then
>  launch gnome-software it has to go and download some metadata before
>  it can show anything. This is a pretty bad first experience,
>  considering subsequent runs of gnome-software just open straight away.
>  GNOME Software operates on the logic that *any* version of the
>  metadata is better than nothing, and only refreshes once a week when
>  the user is idle.
>  
>  There are two ways to fix the jarring UX. We could either ship the
>  fedora package metadata pre-prepared in PackageKit, maybe using
>  something like %ghost so the new metadata is ignored. The other way is
>  to start the metadata downloading in the initial-start wizard thing,
>  although in practice the download takes a couple of minutes to
>  complete, and the wizard typically takes much less than that before
>  the user has configured everything. The downside to shipping prepared
>  metadata is that the package size is larger, and that the metadata
>  would be *very* out of date after a year or so.
>  
>  I'm open to ideas, thanks. 
Does it need all the data upfront ?

Maybe you can have a small place with initial metadata that contains a
subset of the packages and is quick to download so you can show
something then in the background start downloading the real full
metadata and update the UI as data comes in ?

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Richard Hughes]

On 24 January 2014 14:10, Simo Sorce <simo(a)redhat.com&gt; wrote:
>  Does it need all the data upfront ? 
That's a good question. Not much at all just to show the overview page.

>  Maybe you can have a small place with initial metadata that contains
> a
>  subset of the packages and is quick to download so you can show
>  something then in the background start downloading the real full
>  metadata and update the UI as data comes in ? 
I'll have a play with this and see how complicated it is. All the UI
elements can update when the data is changed, so it certainly should
be possible. Thanks!

Richard.

[Zbigniew Jędrzejewski-Szmek]

On Fri, Jan 24, 2014 at 10:01:09AM +0000, Richard Hughes wrote:
>  The downside to shipping prepared
>  metadata is that the package size is larger, and that the metadata
>  would be *very* out of date after a year or so. 
I don't see the problem with
stale metadata of this kind. It's not like
a description or screenshots of a GUI application are completely outdated
after a year. There are exceptions, but most of the time, Fedora wouldn't
even update the package in a certain release if the underlying toolkit
changes or significant new functionality is added. So for packages which
were in the initial release of given Fedora version, slightly stale metadata
shouldn't be an issue. For packages which were added during the release,
the metadata might be missing. But then they would be downloaded during
the update... Of course this is all true in the "stable limit", where
packages have their appdata. This might take a release or two to reach
this point. If you're trying to solve the problem for the *next* release,
that might be different. But probably improving the metadata files and
adding screenshots so everything is there is a better use of time in
the long run than adding a way to download updates faster.

BTW. Reminded by this thread, I've added appdata files for the calibre
package. In the appdata file I have a screenshot:
  <screenshot type="default" width="1200"
height="675">http://in.waw.pl/~zbyszek/calibre/calibre-main-window.png</screenshot>
but gnome-software doesn't display it, despite displaying the textual
content of the appdata file just fine. appdata-validate also doesn't complain.
AFAICT, there's nothing wrong with this picture, the download works, etc. 
Any pointers?

Zbyszek

[Richard Hughes]

On 24 January 2014 15:39, Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; wrote:
>  BTW. Reminded by this thread, I've added appdata files for the
> calibre
>  package. In the appdata file I have a screenshot:
>    <screenshot type="default" width="1200"
> height="675">http://in.waw.pl/~zbyszek/calibre/calibre-main-window.png</screenshot>
>  but gnome-software doesn't display it, despite displaying the textual
>  content of the appdata file just fine. appdata-validate also doesn't complain.
>  AFAICT, there's nothing wrong with this picture, the download works, etc.
>  Any pointers? 
GNOME in Fedora 20 only displays the description, the UI for
downloading screenshots and thumbnails only came in 3.11 -- if you use
rawhide it should work perfectly. I opened an upstream bug before I
knew you did this; perhaps you could submit the file upstream?
https://bugs.launchpad.net/calibre/+bug/1271974 I really appreciate
the help, thanks.

Richard

[Zbigniew Jędrzejewski-Szmek]

On Fri, Jan 24, 2014 at 03:44:07PM +0000, Richard Hughes wrote:
>  On 24 January 2014 15:39, Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl&gt; wrote:
>  > BTW. Reminded by this thread, I've added appdata files for the calibre
>  > package. In the appdata file I have a screenshot:
>  >   <screenshot type="default" width="1200"
> height="675">http://in.waw.pl/~zbyszek/calibre/calibre-main-window.png</screenshot>
>  > but gnome-software doesn't display it, despite displaying the textual
>  > content of the appdata file just fine. appdata-validate also doesn't complain.
>  > AFAICT, there's nothing wrong with this picture, the download works, etc.
>  > Any pointers?
>  
>  GNOME in Fedora 20 only displays the description, the UI for
>  downloading screenshots and thumbnails only came in 3.11 -- if you use
>  rawhide it should work perfectly. I opened an upstream bug before I
>  knew you did this; perhaps you could submit the file upstream? 
Done.

>  https://bugs.launchpad.net/calibre/+bug/1271974 I really appreciate
>  the help, thanks. 
I was running the rawhide version... Before I had
gnome-software-3.11.1-1.fc21.x86_64,
and now I upgraded to 3.11.4-2.fc21.x86_64. Unfortunately 3.11.4-2.fc21.x86_64
crashes all the time for me. At least I think it crashed when I first started it,
because the window disappeared when I was looking for the screenshots. But now,
it doesn't actually *crash*, the window just disappears after a few seconds. In the
logs:

Jan 24 10:56:13 bupkis gnome-session[1958]: Window manager warning: Buggy client sent a
_NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x3e00008 (Software)
Jan 24 10:56:13 bupkis gnome-session[1958]: Window manager warning: meta_window_activate
called by a pager with a 0 timestamp; the pager needs to be fixed.
Jan 24 10:56:13 bupkis PackageKit[26841]: get-packages transaction /11744_bddbeadd from
uid 1001 finished with success after 45ms
Jan 24 10:56:14 bupkis gnome-session[1958]: Window manager warning: last_focus_time
(861528554) is greater than comparison timestamp (861528547).  This most likely represents
a buggy client sending inaccurate timestamps in messages such as _NET_ACTIVE_WINDOW. 
Trying to work around...

That is all.

Zbyszek

[Richard Hughes]

On 24 January 2014 16:08, Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; wrote:
>  I was running the rawhide version... Before I had
> gnome-software-3.11.1-1.fc21.x86_64,
>  and now I upgraded to 3.11.4-2.fc21.x86_64. Unfortunately 3.11.4-2.fc21.x86_64
>  crashes all the time for me. 
Yup, sorry about that, there are a couple of threading fixes in git.
I'll do a new release on Monday. jhbuild is working flawlessly for me,
but that's obviously much harder to set up.

Richard

[Dennis Gilmore]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

El Fri, 24 Jan 2014 10:01:09 +0000
Richard Hughes <hughsient(a)gmail.com&gt; escribió:
>  Hi all,
>  
>  I'm after some advice / ideas. When you install Fedora 20/21 and then
>  launch gnome-software it has to go and download some metadata before
>  it can show anything. This is a pretty bad first experience,
>  considering subsequent runs of gnome-software just open straight away.
>  GNOME Software operates on the logic that *any* version of the
>  metadata is better than nothing, and only refreshes once a week when
>  the user is idle.
>  
>  There are two ways to fix the jarring UX. We could either ship the
>  fedora package metadata pre-prepared in PackageKit, maybe using
>  something like %ghost so the new metadata is ignored. The other way is
>  to start the metadata downloading in the initial-start wizard thing,
>  although in practice the download takes a couple of minutes to
>  complete, and the wizard typically takes much less than that before
>  the user has configured everything. The downside to shipping prepared
>  metadata is that the package size is larger, and that the metadata
>  would be *very* out of date after a year or so.
>  
>  I'm open to ideas, thanks.
>  
You have discounted a third way and its one that I think we should
follow even though you have repeatably fought against it.  That is
provide a web app that provides the metadata. linked to an online app
store or not. you can make the data for the front page be small, and
give a basic template or holding page if no network is available. you
could then download the metadata in the background as people work. it
could be used to solve not only this problem but the problem of how to
get the data together and make it available for the users.

Dennis

Dennis
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.22 (GNU/Linux)
>
> iQIcBAEBAgAGBQJS4t+YAAoJEH7ltONmPFDRcFIP/376CQ598HbJAljI5c9v+r13
> h3BKpXHe1+27Rd38OtKeySLmvK17bGbb7F3NupTM2mNqydu9ga9iJyggCZwrZgBp
> h2toj+DtZ8ZSSu72qK8M8VitwQwfTbN2BfHUHmWCqVvaGe+ALM7RA1/3nQNtRmuo
> 0md/l827exq6ZIsb8w64PntjyPs1LCv1GS1LFQJxQ6ebcii7faN31DAvUoIFpB2M
> 8X4LFBVtXd8M+Ie0lqMBsOUSyWk1BtoENIOCqpWv0trbAL99yC+v5uBsMw5eOGNN
> g7lYTuLqrU3Eo8OGDDSraS6FRT+V3SBR+qSR/woEr4gqp5JfQXjhi+ulcDeVi6r/
> l4W2/ky/p77ZqQYKcxI7ATQjvLVjeNo3K9ce9RRgws3WzKwOG11GLvBg4+5/HlFb
> HfClIMl548ViAAj5ccOQZFbyGZ4eRSsnV9m9LainGpiuVpRCCtUqZMq3lVZRd1Wl
> UY1o2Bo3HMVdsWG5DSTDPqFgbCeiHvUDNQ1kFdnhJVTL6HgEcITq7wz9UXLVxVE4
> aTPa20I1tI21bEYVbmy7KQN+s0VIf9hX/+uXREHWKvMEhwy3LUC/HKCfM9AOf5EC
> fpWINoDl1c1Mw3ywquwwqE3MAklo1bG5bvlr7Qfke3y+wIq9CulWybN2FKvW8oAP
> 2SEw30yh7jZVW/5oNEgf
> =ThbZ
> -----END PGP SIGNATURE-----
>

[Bill Nottingham]

Richard Hughes (hughsient(a)gmail.com) said: 
>  I'm after some advice / ideas. When you install Fedora 20/21 and
> then
>  launch gnome-software it has to go and download some metadata before
>  it can show anything. This is a pretty bad first experience,
>  considering subsequent runs of gnome-software just open straight away.
>  GNOME Software operates on the logic that *any* version of the
>  metadata is better than nothing, and only refreshes once a week when
>  the user is idle. 
(I assume that this is different than the normal repository metadata.)

I think starting this at some point during the initial-setup process or
during the gnome-software first startup is fine is fine ; I've seen many
other application stores that have this behavior where the first run is
slower than subsequent runs.  It could even be messaged in the UI - have it
start a tour of gnome-software on first run that shows the user how the
interface works & so on, as a distraction while the data is downloading.

Bill

[drago01]

On Fri, Jan 24, 2014 at 10:51 PM, Bill Nottingham <notting(a)redhat.com&gt; wrote:
>  Richard Hughes (hughsient(a)gmail.com) said:
> > I'm after some advice / ideas. When you install Fedora 20/21 and then
> > launch gnome-software it has to go and download some metadata before
> > it can show anything. This is a pretty bad first experience,
> > considering subsequent runs of gnome-software just open straight away.
> > GNOME Software operates on the logic that *any* version of the
> > metadata is better than nothing, and only refreshes once a week when
> > the user is idle.
>
>  (I assume that this is different than the normal repository metadata.)
>
>  I think starting this at some point during the initial-setup process or
>  during the gnome-software first startup is fine is fine ; I've seen many
>  other application stores that have this behavior where the first run is
>  slower than subsequent runs. 
Does not mean that we shouldn't try to do better.