GitHub source URLs not working

[Susi Lehtola]

Hi,


I am experiencing problems updating packages employing GitHub source
URLs. For instance,

$ spectool -g python-pyscf.spec
Downloading:
https://github.com/pyscf/pyscf/archive/v2.0.1/pyscf-2.0.1.tar.gz
Download failed:
404 Client Error: Not Found for url:
https://codeload.github.com/pyscf/pyscf/tar.gz/v2.0.1/pyscf-2.0.1
-   0.0 B Elapsed Time: 0:00:00



The URLs used to work. Has anyone else noticed the same problem?
-- 
Susi Lehtola
Fedora Project Contributor
jussilehtola(a)fedoraproject.org

[Sebastian Crane]

Dear Susi,

>  $ spectool -g python-pyscf.spec
>  Downloading:
>  https://github.com/pyscf/pyscf/archive/v2.0.1/pyscf-2.0.1.tar.gz
>  Download failed:
>  ...
>  The URLs used to work. Has anyone else noticed the same problem? 
Maybe GitHub has changed its source archive paths -
https://github.com/pyscf/pyscf/archive/refs/tags/v2.0.1.tar.gz works
for me.

Best wishes,

Sebastian

[Fabio Valentini]

Fri, Nov 26, 2021 at 8:37 PM Susi Lehtola
<jussilehtola(a)fedoraproject.org&gt; wrote:
>  Hi,
>
>
>  I am experiencing problems updating packages employing GitHub source
>  URLs. For instance,
>
>  $ spectool -g python-pyscf.spec
>  Downloading:
>  https://github.com/pyscf/pyscf/archive/v2.0.1/pyscf-2.0.1.tar.gz
>  Download failed:
>  404 Client Error: Not Found for url:
>  https://codeload.github.com/pyscf/pyscf/tar.gz/v2.0.1/pyscf-2.0.1
>  -   0.0 B Elapsed Time: 0:00:00
>
>
>
>  The URLs used to work. Has anyone else noticed the same problem? 
Yes, I've experienced similar problems today.

If this is not a bug but a deliberate change by GitHub, it's going to
be a major PITA to fix all those .spec files for sources hosted on
GitHub ...
So ... does anybody know (or can find out) whether it's the former or
the latter?

Fabio

[Nico Kadel-Garcia]

On Fri, Nov 26, 2021 at 4:39 PM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  Fri, Nov 26, 2021 at 8:37 PM Susi Lehtola
>  <jussilehtola(a)fedoraproject.org&gt; wrote:
>  >
>  > Hi,
>  >
>  >
>  > I am experiencing problems updating packages employing GitHub source
>  > URLs. For instance,
>  >
>  > $ spectool -g python-pyscf.spec
>  > Downloading:
>  > https://github.com/pyscf/pyscf/archive/v2.0.1/pyscf-2.0.1.tar.gz
>  > Download failed:
>  > 404 Client Error: Not Found for url:
>  > https://codeload.github.com/pyscf/pyscf/tar.gz/v2.0.1/pyscf-2.0.1
>  > -   0.0 B Elapsed Time: 0:00:00
>  >
>  >
>  >
>  > The URLs used to work. Has anyone else noticed the same problem?
>
>  Yes, I've experienced similar problems today.
>
>  If this is not a bug but a deliberate change by GitHub, it's going to
>  be a major PITA to fix all those .spec files for sources hosted on
>  GitHub ...
>  So ... does anybody know (or can find out) whether it's the former or
>  the latter?
>
>  Fabio 
It's also Friday right after Thanksgiving, "Black Friday". And most
python modules .spec files, which use pypi.org as their source
repository rathter than github, seem to be workin.

Directly referencing github is sometimes useful, for example for
anyone who bothers to straighten the somewhat crazed renaming of
"ansible" and ansible-core" as "ansible-core" and
"ansible-collections, or who works with ansible colleciton modules.
But how much of github references use 'codeload.github.com' rather
than 'github.com' ?There hae been way, way too many syntaxes for
pulling tarballs for git repos, it's sometimes quite confusing.
Especially for python modules, since the e python repo owner can
delete and replace arbitrary tags at whim.

[Fabio Valentini]

On Fri, Nov 26, 2021 at 11:30 PM Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  It's also Friday right after Thanksgiving, "Black Friday". And most
>  python modules .spec files, which use pypi.org as their source
>  repository rathter than github, seem to be workin. 
(snip)

>  Directly referencing github is sometimes useful, for example for
>  anyone who bothers to straighten the somewhat crazed renaming of
>  "ansible" and ansible-core" as "ansible-core" and
>  "ansible-collections, or who works with ansible colleciton modules.
>  But how much of github references use 'codeload.github.com' rather
>  than 'github.com' ?There hae been way, way too many syntaxes for
>  pulling tarballs for git repos, it's sometimes quite confusing.
>  Especially for python modules, since the e python repo owner can
>  delete and replace arbitrary tags at whim. 
If you look at the actual request URL, you'll see that it is for github.com.
The codeload.github.com URL is then a result of a server-side redirect.

There *have* been many way to load tarballs from GitHub, but this has
pretty much standardized in recent 2-3 years.
And the fact that the URL scheme that is documented in the Packaging
Guidelines (and probably the one used by the forge macros) is broken
does not bode well.

Fabio

[Nico Kadel-Garcia]

On Fri, Nov 26, 2021 at 5:50 PM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  On Fri, Nov 26, 2021 at 11:30 PM Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  >
>  > It's also Friday right after Thanksgiving, "Black Friday". And most
>  > python modules .spec files, which use pypi.org as their source
>  > repository rathter than github, seem to be workin.
>
>  (snip)
>
>  > Directly referencing github is sometimes useful, for example for
>  > anyone who bothers to straighten the somewhat crazed renaming of
>  > "ansible" and ansible-core" as "ansible-core" and
>  > "ansible-collections, or who works with ansible colleciton modules.
>  > But how much of github references use 'codeload.github.com' rather
>  > than 'github.com' ?There hae been way, way too many syntaxes for
>  > pulling tarballs for git repos, it's sometimes quite confusing.
>  > Especially for python modules, since the e python repo owner can
>  > delete and replace arbitrary tags at whim.
>
>  If you look at the actual request URL, you'll see that it is for github.com.
>  The codeload.github.com URL is then a result of a server-side redirect. 
To quote Willie Wonka: "Strike that: reverse it". The
codeload.github.com URL typically generates a successful redirect,
which is not working.

I'd encourage throwing most of thee varied 'get the designated
tarball' schemes to simplify the schemes and make it more reliable.

>  There *have* been many way to load tarballs from GitHub, but this
> has
>  pretty much standardized in recent 2-3 years.
>  And the fact that the URL scheme that is documented in the Packaging
>  Guidelines (and probably the one used by the forge macros) is broken
>  does not bode well. 
*Absolutely* correct! I'd hate to have to change it in the standards.
Do any of us know anyone over at github.com to let them know directly?
I'm not working for a company with such a contract right now, my repos
there are personal right now.

[Todd Zullinger]

Fabio Valentini wrote:
>  Fri, Nov 26, 2021 at 8:37 PM Susi Lehtola
> > I am experiencing problems updating packages employing GitHub source
> > URLs. For instance, 
[...]
> > https://github.com/pyscf/pyscf/archive/v2.0.1/pyscf-2.0.1.tar.gz
> > Download failed:
> > 404 Client Error: Not Found for url:
> > https://codeload.github.com/pyscf/pyscf/tar.gz/v2.0.1/pyscf-2.0.1 
[...]
> > The URLs used to work. Has anyone else noticed the same problem?
>  
>  Yes, I've experienced similar problems today.
>  
>  If this is not a bug but a deliberate change by GitHub, it's going to
>  be a major PITA to fix all those .spec files for sources hosted on
>  GitHub ...
>  So ... does anybody know (or can find out) whether it's the former or
>  the latter? 
https://www.githubstatus.com/ shows most github services are
having issues.  That (hopefully) indicates the source
download issues are part of the outage rather than an
intentional change¹.

https://www.githubstatus.com/incidents/r5qrpp2f5fc0 is the
direct incident URL, I believe.

¹ but it could be an intentional change which has also led
  to -or simply coincided with- an outage.

-- 
Todd

[Sérgio Basto]

On Fri, 2021-11-26 at 14:37 -0500, Susi Lehtola wrote:
>  Hi,
>  
>  
>  I am experiencing problems updating packages employing GitHub source
>  URLs. For instance,
>  
>  $ spectool -g python-pyscf.spec
>  Downloading:
>  https://github.com/pyscf/pyscf/archive/v2.0.1/pyscf-2.0.1.tar.gz
>  Download failed:
>  404 Client Error: Not Found for url:
>  https://codeload.github.com/pyscf/pyscf/tar.gz/v2.0.1/pyscf-2.0.1
>  -   0.0 B Elapsed Time: 0:00:00
>  
>  
>  
>  The URLs used to work. Has anyone else noticed the same problem? 
already reported here (4 hours ago) : 
https://github.com/github/feedback/discussions/8149


>  -- 
>  Susi Lehtola
>  Fedora Project Contributor
>  jussilehtola(a)fedoraproject.org
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure 
-- 
Sérgio M. B.

[Maxwell G]

On Friday, November 26, 2021 6:15:33 PM EST Sérgio Basto wrote:
>  On Fri, 2021-11-26 at 14:37 -0500, Susi Lehtola wrote:
>  > Hi,
>  > 
>  > 
>  > I am experiencing problems updating packages employing GitHub source
>  > URLs. For instance,
>  > 
>  > $ spectool -g python-pyscf.spec
>  > Downloading:
>  > https://github.com/pyscf/pyscf/archive/v2.0.1/pyscf-2.0.1.tar.gz
>  > Download failed:
>  > 404 Client Error: Not Found for url:
>  > https://codeload.github.com/pyscf/pyscf/tar.gz/v2.0.1/pyscf-2.0.1
>  > -   0.0 B Elapsed Time: 0:00:00
>  > 
>  > 
>  > 
>  > The URLs used to work. Has anyone else noticed the same problem?
>  
>  already reported here (4 hours ago) : 
>  https://github.com/github/feedback/discussions/8149 
Hi everyone,

This issue has been fixed. For anyone who wasn't following the Github thread that
Sérgio linked, here is the response from a Github staff member:

>  Hi everyone. 👋🏻
>  
>  A change in the handling of URL schemes was deployed a couple of days ago that caused the
> regression being discussed here. Due to the amount of traffic that the archive endpoints
> see, and the high baseline of 404s on them, this regression did not cause an unusual
> increase of errors that would've caused our alerting to kick in. The change has just
> been rolled back, so the issue is fixed. We will investigate this issue further after the
> weekend and take the appropriate steps to make sure similar regressions don't happen
> in the future.
>  
>  Thank you for reaching out, your patience and understanding. 
Thanks,
Maxwell

-- 
Maxwell G (@gotmax23)
Pronouns: He/Him/His
PGP Key Fingerprint: f57c76e5a238fe0a628e2ecef79e4e25e8c661f8
PGP Keyserver: hkp://keyserver.ubuntu.com
gotmax(a)e.email

[Sebastian Crane]

Dear Susi,

>  The URLs used to work. Has anyone else noticed the same problem?
Good news! GitHub has fixed this issue, and the URLs now work again:

https://github.com/github/feedback/discussions/8149#discussioncomment-171...

Best wishes,

Sebastian