Would it be possible to have recent GHCs in non-modular repositories?

[hekkaidekapus]

Greetings everyone,

Let me start with a probably wrong statement: it seems that the Modularity 
project allows for packaging and installing many major releases for the same 
package. I gather that is a great upside for packagers like Jens, especially 
in face of the way Haskell projects manage dependencies.

People who are not really into Modularity can disable the relevant 
repositories. That is where begin my troubles — because I am not a Modularity 
user. I would like to install a packaged GHC and its its dependencies, and 
only that; for any other Haskell package, I can do the build myself. The issue 
is that Fedora non-modular repositories ship an old GHC, namely version
GHC-8.6.5. The modular repositories, however, are very up-to-date.

Considering the staggering efforts Jens puts into maintaining a large scale of 
Haskell packages, I first thought he downright gave up on packaging non-
modular GHC due to intractable reasons; so, I embarked on a journey of 
building stuff to see where things might break.

But, I have not seen anything break (yet?). If I download the SRPMs Jens makes 
for modular repositories, they build fine as ursine packages, and I guess that 
is not surprising. The apparent downside is using the --allowerasing option 
when dnf-installing them; as I wrote above, that is not an issue for the use 
case I outlined: all non-boot packages can be built on-demand according to the 
local application settings; IMHO avoiding global package databases should be 
an uncontroversial practice.

So, what have I got? Let’s see:

1. Were the RPMs really built inside Fedora?

#Yes, a)
rpm --query --list ghc-compiler-8.10.1-93.fc32.x86_64.rpm
# /usr/bin/ghc
# …
# /usr/lib64/ghc-8.10.1/bin/ghc
# …

#Yes, b)
annocheck --disable-hardened --enable-notes /usr/lib64/ghc-8.10.1/bin/ghc
# annocheck: Version 9.21.
# Notes: Range: 0x431db8 .. 0x5634b5.
#   [O] Version: 3a1
# Notes: Range: 0x431fc0 .. 0x431fc5.
#   [O] Version: 3p906
#   [O] Stack Protection: None
#   [O] Tool: annobin gcc 10.0.1 20200311
#   [O] Tool: running gcc 10.0.1 20200311
#   [O] ABI: c0013
#   [O] PIC: -fpie
#   [O] Short Enums: Not Used
#   [O] FORTIFY: not detected
#   [O] GLIBCXX_ASSERTIONS: Defined
#   [O] Optimization: -O2
#   [O] Control Flow Protection: Full
#   [O] Omit Frame Pointer: Yes
#   [O] Stack Clash Protection: Enabled
#   [O] Stack Realign: Not enabled
# Notes: Range: 0x431fc0 .. 0x5634a5.
#   [O] Version: 3a1b)

2. Do the built binaries behave as expected?

#Yes:
λ cat > hello.hs << EOF
main = putStrLn "Hello, Haskell SIG!\n"
EOF
λ /usr/bin/ghc hello.hs 
[1 of 1] Compiling Main             ( hello.hs, hello.o )
Linking hello ...

So, I have a couple of questions: What are the reasons behind letting  
GHC-8.6.5 linger in non-modular repositories whereas a fresh compiler could 
take its place? Is there anything third parties could do to alleviate the 
burden Jens has certainly to shoulder? Would be there a way to avoid non-
modular users having to build GHC from sources every time there is a new 
release?

It goes without saying, I am very grateful to everything Jens has done and 
does for this SIG.

Regards,
h.

[Jens Petersen]

Hi, not sure why this mail didn't read my inbox...
Tristan kindly pointed it out to me today.

>  Let me start with a probably wrong statement: it seems that the
> Modularity 
>  project allows for packaging and installing many major releases for the same 
>  package. I gather that is a great upside for packagers like Jens, especially 
>  in face of the way Haskell projects manage dependencies. 
TBH I am not entirely convinced modules are ideal for Haskell, but it is there is now.
It does have one advantage that it allows canonical installation of a different version of
a package without any packaging tweaks or hacks though.
Perhaps that is what you mean. :)

>  People who are not really into Modularity can disable the relevant 
>  repositories. That is where begin my troubles — because I am not a Modularity 
>  user. I would like to install a packaged GHC and its its dependencies, and 
>  only that; for any other Haskell package, I can do the build myself. The issue 
>  is that Fedora non-modular repositories ship an old GHC, namely version
>  GHC-8.6.5. The modular repositories, however, are very up-to-date. 
So err, why not use the modules? :)
I am not clear on your objection to using modules.
It seems to me the ghc:8.10 module already does exactly what you want.

What do you need from ghc-8.10 btw?

Personally I have found ghc-8.6.5 is still fine for me and I can use stack if I need a
newer or older ghc.

>  Considering the staggering efforts Jens puts into maintaining a large
> scale of 
>  Haskell packages, I first thought he downright gave up on packaging non-
>  modular GHC due to intractable reasons; so, I embarked on a journey of 
>  building stuff to see where things might break. 
No that is not the case - I am tracking Stackage LTS releases.
Fedora 32 is based on LTS 14 (LTS 15 was a little late to make the last release).
Fedora 33 will hopefully rebase to LTS 15 or 16 with ghc-8.8.3.

>  But, I have not seen anything break (yet?). If I download the SRPMs
> Jens makes 
>  for modular repositories, they build fine as ursine packages, and I guess that 
>  is not surprising. The apparent downside is using the --allowerasing option 
>  when dnf-installing them; as I wrote above, that is not an issue for the use 
>  case I outlined: all non-boot packages can be built on-demand according to the 
>  local application settings; IMHO avoiding global package databases should be 
>  an uncontroversial practice. 
Are you proposing we deal with Haskell packages like Fedora Rust?
ie only ship sources?

>  2. Do the built binaries behave as expected? 
I am not sure of your point here: your local build is basically equivalent to the module
build.

>  So, I have a couple of questions: What are the reasons behind letting
>  
>  GHC-8.6.5 linger in non-modular repositories whereas a fresh compiler could 
>  take its place? Is there anything third parties could do to alleviate the 
>  burden Jens has certainly to shoulder? Would be there a way to avoid non-
>  modular users having to build GHC from sources every time there is a new 
>  release? 
I think I answered this above - I am tracking Stackage LTS, and it lags behind ghc major
releases.
Even today Stackage Nightly is still on ghc-8.8.3.

Just to clarify this isn't just only about ghc - it is about the ~500 haskell packages
we have in Fedora.

We also have stack now in Fedora, though it is not the latest version (like
cabal-install).

>  It goes without saying, I am very grateful to everything Jens has
> done and 
>  does for this SIG. 
Thank you I appreciate your words.

If you have more questions, do ask.

Jens

[hekkaidekapus]

Hello Jens,

>  Hi, not sure why this mail didn't read my inbox...
>  Tristan kindly pointed it out to me today. 
Thanks for taking time to reply. I do not know people around here, I assume
this helpful Tristan is the same as the one in the SIG’s meeting logs dated to
June 9, 2020 (UTC); anyway, thank you Tristan very much.

[…]
>  It does have one advantage that it allows canonical installation of
> a
>  different version of a package without any packaging tweaks or hacks though. 
[…]
>  So err, why not use the modules? 
>  I am not clear on your objection to using modules.
>  It seems to me the ghc:8.10 module already does exactly what you want.
>  What do you need from ghc-8.10 btw? 
[…]
>  I am tracking Stackage LTS releases.
>  Fedora 32 is based on LTS 14 […] 
Now, I understand how the decision about which packages and which versions to
ship is made.

>  Are you proposing we deal with Haskell packages like Fedora Rust? ie
> only
>  ship sources? […] 
I am suggesting to untie the release of GHC packages from other packages’, as
long as the latter are not in the “boot” set[^1]. Id est, if upstream releases
GHC version λ, my inquiry is about whether Fedora could release ghc-λ
regardless of which Stackage snapshots are based on ghc-λ.

Such a suggestion might be nonsense in face of how packaging is done in
Fedora, so please correct me if I am way off on the technical side. Currently,
my understanding is that the state of affairs is as it is because

* it is a great __convenience__ to have a source of assurance (Stackage) that
  hundreds of upstream packages will build successfully;
* most of those packages are taken care of in Fedora by a single person (Jens)
  who, by the way, is an upstream Stackager, thus relying on Stackage is
  “natural”.

To be clear, may the above be close to the reality, I do not object against 
any of it: people who put in the work make the choices in FOS.

As of why should I bother using the latest upstream GHC, the answer is the
same as the reason I use Fedora in the first place: I want “bleeding-edge” yet
solid software without _Gentoo-ing_ everything. So why don’t I switch to 
modularity streams? Parallel installs and partial upgrades is not my use case;
I upgrade the whole distribution at once.

It may be the case that properly doing something along the lines of my
suggestion would lead to mimicking what is done in Fedora Rust or Fedora Go, I
am unclear on the details. Is it so? For instance, the GCC maintainers upgrade
it independently of the release schedule of, say Boost[^2].

Regards,
h.


[^1]: `array`, `base`, `binary`, `bytestring`…
[^2]: Like all analogies, this one has flaws, too. GCC upgrade involves mass
rebuilds, side tags, etc. Maybe Stackage helps avoid churns like that.

[Jens Petersen]

Hi again,

I only saw your reply now somehow though I thought I had checked
sometimes...

On Wed, 10 Jun 2020 at 04:39, hekkaidekapus <cireyapmin(a)gmail.com&gt; wrote:

>  Thanks for taking time to reply. I do not know people around here, I
> assume
>  this helpful Tristan is the same as the one in the SIG’s meeting logs
>  dated to
>  June 9, 2020 (UTC); anyway, thank you Tristan very much.
>  
That's right


>  > So err, why not use the modules? 

You can `dnf module install ghc:8.10` today in Fedora to install the latest
ghc,
but then you lose access to all the prebuilt Haskell libraries, it's true.


>  > I am tracking Stackage LTS releases.
>  > Fedora 32 is based on LTS 14 […]
>
>  Now, I understand how the decision about which packages and which versions
>  to
>  ship is made.
>  
Okay, yes, I just updated Rawhide to ghc-8.8.4 last week for F33.

I am suggesting to untie the release of GHC packages from other packages’,
>  as
>  long as the latter are not in the “boot” set[^1]. Id est, if upstream
>  releases
>  GHC version λ, my inquiry is about whether Fedora could release ghc-λ
>  regardless of which Stackage snapshots are based on ghc-λ.
>  
You mean to use newer ghc than Stackage LTS, I see - I think that is not
generally possible.
Though it might work sometimes perhaps. ((8.8.4 is newer than current LTS
16 though.))
Typically we see considerable breakage in Stackage Nightly when moving to a
new ghc major version.
Mostly this is due to API or other major changes in ghc's libraries: eg
good recent example is MonadFail.

eg
https://gitlab.haskell.org/ghc/ghc/-/wikis/migration/8.8
https://gitlab.haskell.org/ghc/ghc/-/wikis/migration/8.10


>  Such a suggestion might be nonsense in face of how packaging is done
> in
>  Fedora, so please correct me if I am way off on the technical side.
>  Currently,
>  my understanding is that the state of affairs is as it is because
>  
Every major version release of ghc generally has some breaking changes.
I think shipping a ghc incompatible with the Stackage version will cause
problems
both for packaging and users.


>  * it is a great __convenience__ to have a source of assurance
> (Stackage)
>  that
>    hundreds of upstream packages will build successfully;
>  * most of those packages are taken care of in Fedora by a single person
>  (Jens)
>    who, by the way, is an upstream Stackager, thus relying on Stackage is
>    “natural”.
>
>  To be clear, may the above be close to the reality, I do not object
>  against
>  any of it: people who put in the work make the choices in FOS.
>
>  As of why should I bother using the latest upstream GHC, the answer is the
>  same as the reason I use Fedora in the first place: I want “bleeding-edge”
>  yet
>  solid software without _Gentoo-ing_ everything. 

I know this is popular having the latest and greatest ghc,
but with the breaking changes we see it is really quite impractical I think.

I have happily been using 8.6.5 (LTS 13-14) for many months without any
problems.
And just switch recently to developing with LTS 15: basically I prefer to
stay
one LTS release behind because I prefer just to use the last final minor
release: eg LTS 15.16.
I imagine that most users honestly would not notice the differences much,
unless
they want to play with the latest fancy "typery" or benchmarking perhaps.

Sure if Fedora was still on 8.4 or 8.2 say that would be an increasing
major concern,
but as long as we are incrementing Haskell versions even if a release behind
I feel it is not a huge issue, and modules (and stack) are there for those
that really
must have the latest ghc. But I am taking the general feedback from the
community
into account (even Stackage has had complaints about being too slow to move
ahead).
This time I was slightly more aggressive and went with newer minor ghc than
in Stackage
and latest LTS 16 which is still evolving rather than packaging the final
LTS 15.

So why don’t I switch to
>  modularity streams? Parallel installs and partial upgrades is not my
> use
>  case;
>  I upgrade the whole distribution at once.
>  
Modules are not parallel installs. Did you try it out?
Upgrades of modules between releases is also generally supported.
Eg you can update ghc:8.8 or ghc:8.10 from F31 to F32 to F33: it should
just work.


>  It may be the case that properly doing something along the lines of
> my
>  suggestion would lead to mimicking what is done in Fedora Rust or Fedora
>  Go, I
>  am unclear on the details. Is it so? For instance, the GCC maintainers
>  upgrade
>  it independently of the release schedule of, say Boost[^2].
>  
AIUI Fedora Rust basically only ships library sources as cargo cache src
files and builds them per executable.
Maybe we could fool cabal-install to use source tarballs instead, but I
don't find this approach attractive.
Rather my vision is that people should use cabal-rpm to build Haskell
projects on Fedora on top of our binary libraries.

Well I hope that helps a little - it is difficult to make everyone happy.
If we had "infinite resources" we could have modules for each major LTS
release, etc.
Maybe one day with greater automation, but we are just not there yet.

Thanks, Jens