Introducing Steam + video game testing criteria

[Neal Gompa]

Hey all,

I would like for us to have some testing criteria around gaming and
Steam so that we can ensure we're offering a working gaming experience
in Fedora Linux releases. This is motivated by the issue we had in the
F37 cycle where glibc broke popular multiplayer games[1]. I was
reminded of this when I launched Steam today on F38 and zenity
crashed[2].

I would like to propose the following criterion for Steam itself as a
Beta Blocker bug:
"Steam MUST be able to be installed and have its basic functionality
work with no visible errors. Basic functionality for Steam includes:
logging into a Steam account and installing a Windows/Proton game and
a Linux/SteamOS native game."

For gaming itself, I would like to propose the following criterion as
a Final Blocker bug:
"Steam games identified as Deck Verified by ProtonDB.com (see
https://www.protondb.com/explore?selectedFilters=whitelisted) MUST
launch and let the user play the game. This criterion is not intended
to judge performance, merely accessibility. At least one
Windows/Proton game and one Linux/SteamOS native game MUST be tested
in this manner."

Now, the tricky issue here is how to wordsmith the check for
anti-cheat systems. I don't want to specifically call out just EAC,
but I also don't know of a good mix of games with different
anti-cheats. The important thing is to catch regressions and see if
it's something we can resolve. In the EAC case from F37, it was easy
for us to deal with, but if it's genuinely broken in a way we can't
deal with it on the Fedora side, I don't know what we're supposed to
do, so I'm wary of doing some kind of blocker criterion for that.

I'd also like this to be imposed on both release-blocking desktops:
GNOME and KDE Plasma.

Any ideas welcome and appreciated!

[1]: https://pagure.io/fesco/issue/2873
[2]: https://bugzilla.redhat.com/2177287

--
真実はいつも一つ！/ Always, there's only one truth!

[Adam Williamson]

On Thu, 2023-04-20 at 08:20 -0400, Neal Gompa wrote:
>  Hey all,
>  
>  I would like for us to have some testing criteria around gaming and
>  Steam so that we can ensure we're offering a working gaming experience
>  in Fedora Linux releases. This is motivated by the issue we had in the
>  F37 cycle where glibc broke popular multiplayer games[1]. I was
>  reminded of this when I launched Steam today on F38 and zenity
>  crashed[2].
>  
>  I would like to propose the following criterion for Steam itself as a
>  Beta Blocker bug:
>  "Steam MUST be able to be installed and have its basic functionality
>  work with no visible errors. Basic functionality for Steam includes:
>  logging into a Steam account and installing a Windows/Proton game and
>  a Linux/SteamOS native game."
>  
>  For gaming itself, I would like to propose the following criterion as
>  a Final Blocker bug:
>  "Steam games identified as Deck Verified by ProtonDB.com (see
>  https://www.protondb.com/explore?selectedFilters=whitelisted) MUST
>  launch and let the user play the game. This criterion is not intended
>  to judge performance, merely accessibility. At least one
>  Windows/Proton game and one Linux/SteamOS native game MUST be tested
>  in this manner."
>  
>  Now, the tricky issue here is how to wordsmith the check for
>  anti-cheat systems. I don't want to specifically call out just EAC,
>  but I also don't know of a good mix of games with different
>  anti-cheats. The important thing is to catch regressions and see if
>  it's something we can resolve. In the EAC case from F37, it was easy
>  for us to deal with, but if it's genuinely broken in a way we can't
>  deal with it on the Fedora side, I don't know what we're supposed to
>  do, so I'm wary of doing some kind of blocker criterion for that.
>  
>  I'd also like this to be imposed on both release-blocking desktops:
>  GNOME and KDE Plasma.
>  
>  Any ideas welcome and appreciated! 
I'm against this. We have never blocked the OS on proprietary third-
party applications. I don't think it's a path we want to go down.

I'm in favour of testing common third-party stuff before release and
fixing it if we can, but we have always said we will not block Fedora
on this, and I don't think we should change that.

(I did actually test Steam, but I used the flatpak version, not RPM...)

I'm fixing the zenity bug, BTW.
-- 
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net

[Neal Gompa]

On Thu, Apr 20, 2023 at 2:28 PM Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  On Thu, 2023-04-20 at 08:20 -0400, Neal Gompa wrote:
>  > Hey all,
>  >
>  > I would like for us to have some testing criteria around gaming and
>  > Steam so that we can ensure we're offering a working gaming experience
>  > in Fedora Linux releases. This is motivated by the issue we had in the
>  > F37 cycle where glibc broke popular multiplayer games[1]. I was
>  > reminded of this when I launched Steam today on F38 and zenity
>  > crashed[2].
>  >
>  > I would like to propose the following criterion for Steam itself as a
>  > Beta Blocker bug:
>  > "Steam MUST be able to be installed and have its basic functionality
>  > work with no visible errors. Basic functionality for Steam includes:
>  > logging into a Steam account and installing a Windows/Proton game and
>  > a Linux/SteamOS native game."
>  >
>  > For gaming itself, I would like to propose the following criterion as
>  > a Final Blocker bug:
>  > "Steam games identified as Deck Verified by ProtonDB.com (see
>  > https://www.protondb.com/explore?selectedFilters=whitelisted) MUST
>  > launch and let the user play the game. This criterion is not intended
>  > to judge performance, merely accessibility. At least one
>  > Windows/Proton game and one Linux/SteamOS native game MUST be tested
>  > in this manner."
>  >
>  > Now, the tricky issue here is how to wordsmith the check for
>  > anti-cheat systems. I don't want to specifically call out just EAC,
>  > but I also don't know of a good mix of games with different
>  > anti-cheats. The important thing is to catch regressions and see if
>  > it's something we can resolve. In the EAC case from F37, it was easy
>  > for us to deal with, but if it's genuinely broken in a way we can't
>  > deal with it on the Fedora side, I don't know what we're supposed to
>  > do, so I'm wary of doing some kind of blocker criterion for that.
>  >
>  > I'd also like this to be imposed on both release-blocking desktops:
>  > GNOME and KDE Plasma.
>  >
>  > Any ideas welcome and appreciated!
>
>  I'm against this. We have never blocked the OS on proprietary third-
>  party applications. I don't think it's a path we want to go down.
>
>  I'm in favour of testing common third-party stuff before release and
>  fixing it if we can, but we have always said we will not block Fedora
>  on this, and I don't think we should change that.
>
>  (I did actually test Steam, but I used the flatpak version, not RPM...)
>
>  I'm fixing the zenity bug, BTW. 
Is there a way we can add regular testing for it without making it a
blocker then?


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Adam Williamson]

On Thu, 2023-04-20 at 17:38 -0400, Neal Gompa wrote:
>  On Thu, Apr 20, 2023 at 2:28 PM Adam Williamson
>  <adamwill(a)fedoraproject.org&gt; wrote:
>  > 
>  > On Thu, 2023-04-20 at 08:20 -0400, Neal Gompa wrote:
>  > > Hey all,
>  > > 
>  > > I would like for us to have some testing criteria around gaming and
>  > > Steam so that we can ensure we're offering a working gaming experience
>  > > in Fedora Linux releases. This is motivated by the issue we had in the
>  > > F37 cycle where glibc broke popular multiplayer games[1]. I was
>  > > reminded of this when I launched Steam today on F38 and zenity
>  > > crashed[2].
>  > > 
>  > > I would like to propose the following criterion for Steam itself as a
>  > > Beta Blocker bug:
>  > > "Steam MUST be able to be installed and have its basic functionality
>  > > work with no visible errors. Basic functionality for Steam includes:
>  > > logging into a Steam account and installing a Windows/Proton game and
>  > > a Linux/SteamOS native game."
>  > > 
>  > > For gaming itself, I would like to propose the following criterion as
>  > > a Final Blocker bug:
>  > > "Steam games identified as Deck Verified by ProtonDB.com (see
>  > > https://www.protondb.com/explore?selectedFilters=whitelisted) MUST
>  > > launch and let the user play the game. This criterion is not intended
>  > > to judge performance, merely accessibility. At least one
>  > > Windows/Proton game and one Linux/SteamOS native game MUST be tested
>  > > in this manner."
>  > > 
>  > > Now, the tricky issue here is how to wordsmith the check for
>  > > anti-cheat systems. I don't want to specifically call out just EAC,
>  > > but I also don't know of a good mix of games with different
>  > > anti-cheats. The important thing is to catch regressions and see if
>  > > it's something we can resolve. In the EAC case from F37, it was easy
>  > > for us to deal with, but if it's genuinely broken in a way we can't
>  > > deal with it on the Fedora side, I don't know what we're supposed to
>  > > do, so I'm wary of doing some kind of blocker criterion for that.
>  > > 
>  > > I'd also like this to be imposed on both release-blocking desktops:
>  > > GNOME and KDE Plasma.
>  > > 
>  > > Any ideas welcome and appreciated!
>  > 
>  > I'm against this. We have never blocked the OS on proprietary third-
>  > party applications. I don't think it's a path we want to go down.
>  > 
>  > I'm in favour of testing common third-party stuff before release and
>  > fixing it if we can, but we have always said we will not block Fedora
>  > on this, and I don't think we should change that.
>  > 
>  > (I did actually test Steam, but I used the flatpak version, not RPM...)
>  > 
>  > I'm fixing the zenity bug, BTW.
>  
>  Is there a way we can add regular testing for it without making it a
>  blocker then? 
Sure, we have lots of these. It's what the "Optional" milestone in the
test matrices means: any test whose milestone is "Optional"
is...optional.

You can write up a test case (or more than one) for this and propose
adding it to the desktop matrix as an Optional row (or, well, probably
several rows), I think that would be fine.
-- 
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net

[Kamil Paral]

On Thu, Apr 20, 2023 at 11:39 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:

>  Is there a way we can add regular testing for it without making it a
>  blocker then?
>  
For popular third-party software, I think this testing occurs quite
naturally - people just use it. E.g. Steam, Chrome, VSCode, etc. I've been
running Steam on F38 since Beta (and it helped me to discover an issue in
mutter, which was later accepted as a blocker - but not because of Steam,
but because of general issues). But if you want to have an explicit test
case, Adam described how to do it. We could also have a test day for
popular third-party software, if it makes sense.

Note that we had proprietary software-related blockers in the past. We
accepted the EAC glibc issue as a blocker in one of the recent releases. We
also accepted the RPM signatures change as a blocker for F38. But in these
cases, it was always accepted as an exceptional blocker confirmed by FESCo.
I guess it makes sense to use this exceptional process for these cases
(blocking on issues with third-party software, which can affect our user
base badly). We could talk about some general rule related to popular
third-party software, but that would again need to go through FESCo. At the
moment, we only block on our own code in our own workflows, with a few very
specific exceptions (boot usb creation, dual boot configuration, cloud
images deployment).

[Adam Williamson]

On Tue, 2023-04-25 at 13:49 +0200, Kamil Paral wrote:
>  On Thu, Apr 20, 2023 at 11:39 PM Neal Gompa
> <ngompa13(a)gmail.com&gt; wrote:
>  
>  > Is there a way we can add regular testing for it without making it a
>  > blocker then?
>  > 
>  
>  For popular third-party software, I think this testing occurs quite
>  naturally - people just use it. E.g. Steam, Chrome, VSCode, etc. I've been
>  running Steam on F38 since Beta (and it helped me to discover an issue in
>  mutter, which was later accepted as a blocker - but not because of Steam,
>  but because of general issues). But if you want to have an explicit test
>  case, Adam described how to do it. We could also have a test day for
>  popular third-party software, if it makes sense. 
To be fair, we did miss
https://bugzilla.redhat.com/show_bug.cgi?id=2177287 (I missed it
because I run Steam via Flatpak, not RPM). So we do have scope to
improve there. I do think testing commonly used third-party stuff is a
good idea, to be clear, and I'm all in favor of adding optional test
cases for it.
-- 
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net

[Neal Gompa]

On Tue, Apr 25, 2023 at 12:18 PM Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  On Tue, 2023-04-25 at 13:49 +0200, Kamil Paral wrote:
>  > On Thu, Apr 20, 2023 at 11:39 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  >
>  > > Is there a way we can add regular testing for it without making it a
>  > > blocker then?
>  > >
>  >
>  > For popular third-party software, I think this testing occurs quite
>  > naturally - people just use it. E.g. Steam, Chrome, VSCode, etc. I've been
>  > running Steam on F38 since Beta (and it helped me to discover an issue in
>  > mutter, which was later accepted as a blocker - but not because of Steam,
>  > but because of general issues). But if you want to have an explicit test
>  > case, Adam described how to do it. We could also have a test day for
>  > popular third-party software, if it makes sense.
>
>  To be fair, we did miss
>  https://bugzilla.redhat.com/show_bug.cgi?id=2177287 (I missed it
>  because I run Steam via Flatpak, not RPM). So we do have scope to
>  improve there. I do think testing commonly used third-party stuff is a
>  good idea, to be clear, and I'm all in favor of adding optional test
>  cases for it. 
And I'm fine with us adding some optional test cases for this. Would
running a test day each cycle also be possible?



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Adam Williamson]

On Tue, 2023-04-25 at 12:20 -0400, Neal Gompa wrote:
>  On Tue, Apr 25, 2023 at 12:18 PM Adam Williamson
>  <adamwill(a)fedoraproject.org&gt; wrote:
>  > 
>  > On Tue, 2023-04-25 at 13:49 +0200, Kamil Paral wrote:
>  > > On Thu, Apr 20, 2023 at 11:39 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  > > 
>  > > > Is there a way we can add regular testing for it without making it a
>  > > > blocker then?
>  > > > 
>  > > 
>  > > For popular third-party software, I think this testing occurs quite
>  > > naturally - people just use it. E.g. Steam, Chrome, VSCode, etc. I've been
>  > > running Steam on F38 since Beta (and it helped me to discover an issue in
>  > > mutter, which was later accepted as a blocker - but not because of Steam,
>  > > but because of general issues). But if you want to have an explicit test
>  > > case, Adam described how to do it. We could also have a test day for
>  > > popular third-party software, if it makes sense.
>  > 
>  > To be fair, we did miss
>  > https://bugzilla.redhat.com/show_bug.cgi?id=2177287 (I missed it
>  > because I run Steam via Flatpak, not RPM). So we do have scope to
>  > improve there. I do think testing commonly used third-party stuff is a
>  > good idea, to be clear, and I'm all in favor of adding optional test
>  > cases for it.
>  
>  And I'm fine with us adding some optional test cases for this. Would
>  running a test day each cycle also be possible? 
I don't see any reason why not! Sumantro puts out a call for test days
each cycle, or you could just go ahead and file a ticket for one at
https://pagure.io/fedora-qa/issues now (give it the 'test days' tag and
the Fedora 39 milestone). Sumantro is in charge of that process, and
there's a lot of flexibility in terms of who is mainly responsible for
running each individual test day; you can certainly take an active role
in setting it up and running it if you'd like to.
-- 
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net

[Peter Robinson]

On Thu, Apr 20, 2023 at 1:20 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  Hey all,
>
>  I would like for us to have some testing criteria around gaming and
>  Steam so that we can ensure we're offering a working gaming experience
>  in Fedora Linux releases. This is motivated by the issue we had in the
>  F37 cycle where glibc broke popular multiplayer games[1]. I was
>  reminded of this when I launched Steam today on F38 and zenity
>  crashed[2].
>
>  I would like to propose the following criterion for Steam itself as a
>  Beta Blocker bug:
>  "Steam MUST be able to be installed and have its basic functionality
>  work with no visible errors. Basic functionality for Steam includes:
>  logging into a Steam account and installing a Windows/Proton game and
>  a Linux/SteamOS native game."
>
>  For gaming itself, I would like to propose the following criterion as
>  a Final Blocker bug:
>  "Steam games identified as Deck Verified by ProtonDB.com (see
>  https://www.protondb.com/explore?selectedFilters=whitelisted) MUST
>  launch and let the user play the game. This criterion is not intended
>  to judge performance, merely accessibility. At least one
>  Windows/Proton game and one Linux/SteamOS native game MUST be tested
>  in this manner."
>
>  Now, the tricky issue here is how to wordsmith the check for
>  anti-cheat systems. I don't want to specifically call out just EAC,
>  but I also don't know of a good mix of games with different
>  anti-cheats. The important thing is to catch regressions and see if
>  it's something we can resolve. In the EAC case from F37, it was easy
>  for us to deal with, but if it's genuinely broken in a way we can't
>  deal with it on the Fedora side, I don't know what we're supposed to
>  do, so I'm wary of doing some kind of blocker criterion for that.
>
>  I'd also like this to be imposed on both release-blocking desktops:
>  GNOME and KDE Plasma.
>
>  Any ideas welcome and appreciated! 
So this is a proprietary platform, we don't even block on something
like the binary nvidia drivers, so I'm not sure how we can block on
something that is mostly out of our control.

If there is a change in an upstream kernel that triggers a crash do we
have to await a fix from them or roll back to an older kernel?

Also are there free games that can be played for testing or
reproducing bugs that don't require you to put in a credit card to be
able to use them on Steam (sorry, I've never used it).

Peter

[John P]

Being that Fedora is so strongly for open source sodtware, I personally
think any blockers shouldn't be from or for any 3rd party proprietary
software like Steam (or NVIDIA like mentioned). I would expect any blockers
to be applications or packages that come with Fedora by default. I'd be
happy to test Steam as I have a decent Steam library but I don't think it
should be a blocker by any means. New to QA Fedora, can't wait to get
involved.

On Thu, Apr 20, 2023, 16:15 Peter Robinson <pbrobinson(a)gmail.com&gt; wrote:

>  On Thu, Apr 20, 2023 at 1:20 PM Neal Gompa <ngompa13(a)gmail.com&gt;
> wrote:
>  >
>  > Hey all,
>  >
>  > I would like for us to have some testing criteria around gaming and
>  > Steam so that we can ensure we're offering a working gaming experience
>  > in Fedora Linux releases. This is motivated by the issue we had in the
>  > F37 cycle where glibc broke popular multiplayer games[1]. I was
>  > reminded of this when I launched Steam today on F38 and zenity
>  > crashed[2].
>  >
>  > I would like to propose the following criterion for Steam itself as a
>  > Beta Blocker bug:
>  > "Steam MUST be able to be installed and have its basic functionality
>  > work with no visible errors. Basic functionality for Steam includes:
>  > logging into a Steam account and installing a Windows/Proton game and
>  > a Linux/SteamOS native game."
>  >
>  > For gaming itself, I would like to propose the following criterion as
>  > a Final Blocker bug:
>  > "Steam games identified as Deck Verified by ProtonDB.com (see
>  > https://www.protondb.com/explore?selectedFilters=whitelisted) MUST
>  > launch and let the user play the game. This criterion is not intended
>  > to judge performance, merely accessibility. At least one
>  > Windows/Proton game and one Linux/SteamOS native game MUST be tested
>  > in this manner."
>  >
>  > Now, the tricky issue here is how to wordsmith the check for
>  > anti-cheat systems. I don't want to specifically call out just EAC,
>  > but I also don't know of a good mix of games with different
>  > anti-cheats. The important thing is to catch regressions and see if
>  > it's something we can resolve. In the EAC case from F37, it was easy
>  > for us to deal with, but if it's genuinely broken in a way we can't
>  > deal with it on the Fedora side, I don't know what we're supposed to
>  > do, so I'm wary of doing some kind of blocker criterion for that.
>  >
>  > I'd also like this to be imposed on both release-blocking desktops:
>  > GNOME and KDE Plasma.
>  >
>  > Any ideas welcome and appreciated!
>
>  So this is a proprietary platform, we don't even block on something
>  like the binary nvidia drivers, so I'm not sure how we can block on
>  something that is mostly out of our control.
>
>  If there is a change in an upstream kernel that triggers a crash do we
>  have to await a fix from them or roll back to an older kernel?
>
>  Also are there free games that can be played for testing or
>  reproducing bugs that don't require you to put in a credit card to be
>  able to use them on Steam (sorry, I've never used it).
>
>  Peter
>  _______________________________________________
>  test mailing list -- test(a)lists.fedoraproject.org
>  To unsubscribe send an email to test-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Neal Gompa]

On Thu, Apr 20, 2023 at 5:14 PM Peter Robinson <pbrobinson(a)gmail.com&gt; wrote:
>  On Thu, Apr 20, 2023 at 1:20 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  >
>  > Hey all,
>  >
>  > I would like for us to have some testing criteria around gaming and
>  > Steam so that we can ensure we're offering a working gaming experience
>  > in Fedora Linux releases. This is motivated by the issue we had in the
>  > F37 cycle where glibc broke popular multiplayer games[1]. I was
>  > reminded of this when I launched Steam today on F38 and zenity
>  > crashed[2].
>  >
>  > I would like to propose the following criterion for Steam itself as a
>  > Beta Blocker bug:
>  > "Steam MUST be able to be installed and have its basic functionality
>  > work with no visible errors. Basic functionality for Steam includes:
>  > logging into a Steam account and installing a Windows/Proton game and
>  > a Linux/SteamOS native game."
>  >
>  > For gaming itself, I would like to propose the following criterion as
>  > a Final Blocker bug:
>  > "Steam games identified as Deck Verified by ProtonDB.com (see
>  > https://www.protondb.com/explore?selectedFilters=whitelisted) MUST
>  > launch and let the user play the game. This criterion is not intended
>  > to judge performance, merely accessibility. At least one
>  > Windows/Proton game and one Linux/SteamOS native game MUST be tested
>  > in this manner."
>  >
>  > Now, the tricky issue here is how to wordsmith the check for
>  > anti-cheat systems. I don't want to specifically call out just EAC,
>  > but I also don't know of a good mix of games with different
>  > anti-cheats. The important thing is to catch regressions and see if
>  > it's something we can resolve. In the EAC case from F37, it was easy
>  > for us to deal with, but if it's genuinely broken in a way we can't
>  > deal with it on the Fedora side, I don't know what we're supposed to
>  > do, so I'm wary of doing some kind of blocker criterion for that.
>  >
>  > I'd also like this to be imposed on both release-blocking desktops:
>  > GNOME and KDE Plasma.
>  >
>  > Any ideas welcome and appreciated!
>
>  So this is a proprietary platform, we don't even block on something
>  like the binary nvidia drivers, so I'm not sure how we can block on
>  something that is mostly out of our control.
>
>  If there is a change in an upstream kernel that triggers a crash do we
>  have to await a fix from them or roll back to an older kernel?
>
>  Also are there free games that can be played for testing or
>  reproducing bugs that don't require you to put in a credit card to be
>  able to use them on Steam (sorry, I've never used it). 
Steam has a toggle in the storefront for "free-to-play" games that you
can add to your Steam library without entering a credit card.



-- 
真実はいつも一つ！/ Always, there's only one truth!