Preparing for ocaml 4.11

[Jerry James]

This message is mostly for Richard Jones, but I'm sending it to the
list so that others who maintain OCaml packages can weigh in if they
choose.  I have BCCed a few of you that are affected by some
suggestions I make below.

I have spent the last several days doing mock builds to see if I can
safely update all the OCaml packages I maintain to their latest
versions, since some have new versions that are specifically for OCaml
4.11 compatibility.  The following remarks, in no particular order,
stem from that effort.

First the good news: I was able to build all the updated versions
successfully, although see below.

Some of the recently-added packages have boolean dependencies, which I
added at the reviewer's request in order to mimic the version
requirements in the opam files.  The ocaml-ppx-custom-printf spec
file, for example, has this:

BuildRequires:  (ocaml-base-devel >= 0.13 and ocaml-base-devel < 0.14)

It never occurred to me until now to ask if your (Richard's) ocaml
rebuild script can handle such dependencies.  Can it?  (And I lost the
link to the script.  Can you supply it again?)

The utop package has a new version, 2.6.0, that brings OCaml 4.11
compatibility.  However, it requires version 3.1.x of
ocaml-lambda-term, which has a new dependency, mew_vi, that we do not
have in Fedora.  The mew_vi module requires mew, which requires trie.
I have submitted review requests and am waiting for somebody to
respond to yesterday's request for review swaps:
https://bugzilla.redhat.com/show_bug.cgi?id=1850263
https://bugzilla.redhat.com/show_bug.cgi?id=1850264
https://bugzilla.redhat.com/show_bug.cgi?id=1850265

I successfully updated all of the Jane Street packages that are
currently on version 0.13.x to version 0.14.0.  However, one of them
is going to be somewhat problematic.  The ocaml-ppx-inline-test
package has a new dependency, time_now.  The time_now module has 11
dependencies that we do not already have in Fedora.  All 11, and
time_now itself, were on my roadmap for eventual inclusion in Fedora,
but I had not intended to pursue them just yet.  I haven't put any of
the 11 up for review yet, although I have written spec files and done
local mock builds for them all.  I need to go over them once more and
then I'll put them up for review.  If the timeframe to an ocaml 4.11
build is too short to wait for that, we may have to disable tests for
packages that currently consume ocaml-ppx-inline-test, and leave it in
a broken state until those reviews can be completed.

I would like to see some updates to other people's packages.  For all
of the following, I have the necessary spec file changes ready to show
to the maintainers should they like to see them.

Updating ocaml-bisect-ppx to its latest version requires updating
ocaml-ppx-tools-versioned to version 5.4.0 or later.

I suggest that we update to ocaml-ocplib-endian from version 1.0 to
1.1.  It includes fixes for a couple of issues
(https://github.com/OCamlPro/ocplib-endian/issues/20 and
https://github.com/OCamlPro/ocplib-endian/issues/21) that I think we
would like to have resolved.  This entails a change to building with
dune.  Unfortunately, a new issue makes some tests fail
(https://github.com/OCamlPro/ocplib-endian/issues/18), but I hope to
find a workaround for that soon.

I suggest an update to ocaml-result from version 1.2 to 1.5.  This
version makes Result an alias of Stdlib.Result on recent OCaml
versions.  Again, this entails a change to building with dune.

We could also update ocaml-seq to 0.2.2 just to drop the downstream
patches (this is yet another conversion to building with dune, by the
way), but I don't know that there is any other advantage to updating.

We could update ocaml-react to version 1.2.1 to drop the downstream
patch.  This entails a license change.

A rebuild seems like a good opportunity to update ocaml-lwt to version
5.3.0.  With this version, a number of spec file simplifications are
possible.

I've got changes ready to update ocaml-sedlex to version 2.2 AND add a
%check script.

I've been wondering about uchar and stdlib-shims.  We don't need them
in Fedora, but then we don't really need ocaml-result or ocaml-seq
either.  Should we add uchar and stdlib-shims, just so we can stop
modifying upstream dune and opam files?  That might also benefit
Fedora users who try to build using Fedora's ocaml ecosystem.

Maybe I should do COPR builds of all this so everybody can easily see
what I'm talking about?  I did a total of 63 package builds in mock,
many of them simple rebuilds, so it will take a little time to get
that going.  Do we know yet what the timeframe is for an ocaml 4.11
release?
-- 
Jerry James
http://www.jamezone.org/

[Richard W.M. Jones]

On Wed, Jun 24, 2020 at 12:39:24PM -0600, Jerry James wrote:
>  This message is mostly for Richard Jones, but I'm sending it to
> the
>  list so that others who maintain OCaml packages can weigh in if they
>  choose.  I have BCCed a few of you that are affected by some
>  suggestions I make below.
>  
>  I have spent the last several days doing mock builds to see if I can
>  safely update all the OCaml packages I maintain to their latest
>  versions, since some have new versions that are specifically for OCaml
>  4.11 compatibility.  The following remarks, in no particular order,
>  stem from that effort. 
About 4.11, I noticed that alpha #3 was released last week.  I'm still
waiting for the beta before I do a rebuild however, and I think that's
still a few weeks away.

>  First the good news: I was able to build all the updated versions
>  successfully, although see below.
>  
>  Some of the recently-added packages have boolean dependencies, which I
>  added at the reviewer's request in order to mimic the version
>  requirements in the opam files.  The ocaml-ppx-custom-printf spec
>  file, for example, has this:
>  
>  BuildRequires:  (ocaml-base-devel >= 0.13 and ocaml-base-devel < 0.14)
>  
>  It never occurred to me until now to ask if your (Richard's) ocaml
>  rebuild script can handle such dependencies.  Can it?  (And I lost the
>  link to the script.  Can you supply it again?) 
Yeah probably it won't, but I should be able to fix it.  The script
is:

http://git.annexia.org/?p=fedora-ocaml-rebuild.git;a=summary

>  The utop package has a new version, 2.6.0, that brings OCaml 4.11
>  compatibility.  However, it requires version 3.1.x of
>  ocaml-lambda-term, which has a new dependency, mew_vi, that we do not
>  have in Fedora.  The mew_vi module requires mew, which requires trie.
>  I have submitted review requests and am waiting for somebody to
>  respond to yesterday's request for review swaps:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1850263
>  https://bugzilla.redhat.com/show_bug.cgi?id=1850264
>  https://bugzilla.redhat.com/show_bug.cgi?id=1850265
>  
>  I successfully updated all of the Jane Street packages that are
>  currently on version 0.13.x to version 0.14.0.  However, one of them
>  is going to be somewhat problematic.  The ocaml-ppx-inline-test
>  package has a new dependency, time_now.  The time_now module has 11
>  dependencies that we do not already have in Fedora.  All 11, and
>  time_now itself, were on my roadmap for eventual inclusion in Fedora,
>  but I had not intended to pursue them just yet.  I haven't put any of
>  the 11 up for review yet, although I have written spec files and done
>  local mock builds for them all.  I need to go over them once more and
>  then I'll put them up for review.  If the timeframe to an ocaml 4.11
>  build is too short to wait for that, we may have to disable tests for
>  packages that currently consume ocaml-ppx-inline-test, and leave it in
>  a broken state until those reviews can be completed.
>  
>  I would like to see some updates to other people's packages.  For all
>  of the following, I have the necessary spec file changes ready to show
>  to the maintainers should they like to see them.
>  
>  Updating ocaml-bisect-ppx to its latest version requires updating
>  ocaml-ppx-tools-versioned to version 5.4.0 or later.
>  
>  I suggest that we update to ocaml-ocplib-endian from version 1.0 to
>  1.1.  It includes fixes for a couple of issues
>  (https://github.com/OCamlPro/ocplib-endian/issues/20 and
>  https://github.com/OCamlPro/ocplib-endian/issues/21) that I think we
>  would like to have resolved.  This entails a change to building with
>  dune.  Unfortunately, a new issue makes some tests fail
>  (https://github.com/OCamlPro/ocplib-endian/issues/18), but I hope to
>  find a workaround for that soon.
>  
>  I suggest an update to ocaml-result from version 1.2 to 1.5.  This
>  version makes Result an alias of Stdlib.Result on recent OCaml
>  versions.  Again, this entails a change to building with dune.
>  
>  We could also update ocaml-seq to 0.2.2 just to drop the downstream
>  patches (this is yet another conversion to building with dune, by the
>  way), but I don't know that there is any other advantage to updating.
>  
>  We could update ocaml-react to version 1.2.1 to drop the downstream
>  patch.  This entails a license change.
>  
>  A rebuild seems like a good opportunity to update ocaml-lwt to version
>  5.3.0.  With this version, a number of spec file simplifications are
>  possible.
>  
>  I've got changes ready to update ocaml-sedlex to version 2.2 AND add a
>  %check script.
>  
>  I've been wondering about uchar and stdlib-shims.  We don't need them
>  in Fedora, but then we don't really need ocaml-result or ocaml-seq
>  either.  Should we add uchar and stdlib-shims, just so we can stop
>  modifying upstream dune and opam files?  That might also benefit
>  Fedora users who try to build using Fedora's ocaml ecosystem.
>  
>  Maybe I should do COPR builds of all this so everybody can easily see
>  what I'm talking about?  I did a total of 63 package builds in mock,
>  many of them simple rebuilds, so it will take a little time to get
>  that going.  Do we know yet what the timeframe is for an ocaml 4.11
>  release? 
This is all great stuff, and I think it also makes me wonder how we
can make OCaml packaging simpler in Fedora.

RPM has a new %generate_buildrequires feature which we might utilise
(driven from the opam package metadata, I guess?).  I would also like
to drop the whole ocaml-pkg vs ocaml-pkg-devel split, it's confusing
and not necessary.  And maybe we could look at some kind of opam2spec
scripting, similar to cpanspec.  Of course the devil is in the details
because unless we can mostly automate packaging, we could end up with
something that's even more complicated to deal with the exceptional
cases.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Dan Čermák]

"Richard W.M. Jones" <rjones(a)redhat.com&gt; writes:

> > Maybe I should do COPR builds of all this so everybody can easily
> see
> > what I'm talking about?  I did a total of 63 package builds in mock,
> > many of them simple rebuilds, so it will take a little time to get
> > that going.  Do we know yet what the timeframe is for an ocaml 4.11
> > release?
>
>  This is all great stuff, and I think it also makes me wonder how we
>  can make OCaml packaging simpler in Fedora.
>
>  RPM has a new %generate_buildrequires feature which we might utilise
>  (driven from the opam package metadata, I guess?).  I would also like
>  to drop the whole ocaml-pkg vs ocaml-pkg-devel split, it's confusing
>  and not necessary.  And maybe we could look at some kind of opam2spec
>  scripting, similar to cpanspec.  Of course the devil is in the details
>  because unless we can mostly automate packaging, we could end up with
>  something that's even more complicated to deal with the exceptional
>  cases. 
Thanks for bringing this up Rich, I was actually thinking about this for
quite a while now after getting involved in OCaml packaging a bit.

I would suggest the following: we introduce a new Provides for ocaml
packages: Opam($pkg-name) = 1.2.3
I would propose that the ocaml provides generator takes care of adding
this to the devel package. This shouldn't be really hard to pull off, it
would just need to find the opam file and grep for the version.

%generate_buildrequires could be used to parse the opam file from the
tarball and extract the dependent opam packages from the depends:
array. E.g. the following:
--8<---------------cut here---------------start------------->8---
depends: [
  "ocaml" {>= "4.04.2"}
  "base"  {>= "v0.13" & < "v0.14"}
  "dune"  {>= "1.5.1"}
]
--8<---------------cut here---------------end--------------->8---
would translate to:
--8<---------------cut here---------------start------------->8---
BuildRequires: Opam(ocaml) >= 4.04.2
BuildRequires: Opam(base) >= 0.13 and Opam(base) < 0.14
BuildRequires: Opam(dune) >= 1.5.1
--8<---------------cut here---------------end--------------->8---

The hypothetical opam2spec would be certainly also a huge improvement,
as I think that most ocaml packages could be completely autogenerated.


But as you already said, the devil is most likely in the details, so
maybe we should start collecting ideas and potential pitfalls first? I
have therefore also cc'd Olaf, who takes care of Ocaml in
openSUSE. I would hope that we can coordinate our efforts, as Olaf has
some pretty nice ocaml macros that automatically generate file lists for
the subpackages (thereby already simplifying packaging quite a bit).


And at last I'd like to throw in a (probably quite controversial) idea:
how about we move ocaml into a module? Not to be able to have multiple
ocaml versions as streams, but so that we can use MBS to automatically
untangle the bootstrap loops for us. I was totally not aware of this
feature of MBS, but it was mentioned by Stephen in this email:
https://lists.fedoraproject.org/archives/list/[e-mail].o...


So, what do you think about all this?


Cheers,

Dan

[Richard W.M. Jones]

On Thu, Jun 25, 2020 at 12:15:51PM +0200, Dan Čermák wrote:
>  "Richard W.M. Jones" <rjones(a)redhat.com&gt; writes:
>  
>  >> Maybe I should do COPR builds of all this so everybody can easily see
>  >> what I'm talking about?  I did a total of 63 package builds in mock,
>  >> many of them simple rebuilds, so it will take a little time to get
>  >> that going.  Do we know yet what the timeframe is for an ocaml 4.11
>  >> release?
>  >
>  > This is all great stuff, and I think it also makes me wonder how we
>  > can make OCaml packaging simpler in Fedora.
>  >
>  > RPM has a new %generate_buildrequires feature which we might utilise
>  > (driven from the opam package metadata, I guess?).  I would also like
>  > to drop the whole ocaml-pkg vs ocaml-pkg-devel split, it's confusing
>  > and not necessary.  And maybe we could look at some kind of opam2spec
>  > scripting, similar to cpanspec.  Of course the devil is in the details
>  > because unless we can mostly automate packaging, we could end up with
>  > something that's even more complicated to deal with the exceptional
>  > cases.
>  
>  Thanks for bringing this up Rich, I was actually thinking about this for
>  quite a while now after getting involved in OCaml packaging a bit.
>  
>  I would suggest the following: we introduce a new Provides for ocaml
>  packages: Opam($pkg-name) = 1.2.3
>  I would propose that the ocaml provides generator takes care of adding
>  this to the devel package. This shouldn't be really hard to pull off, it
>  would just need to find the opam file and grep for the version. 
Does it need "Opam()", surely ocaml-${pkg-name} is sufficient?
(unless $pkg-name == ocaml in which case we special-case it)

>  %generate_buildrequires could be used to parse the opam file from
> the
>  tarball and extract the dependent opam packages from the depends:
>  array. E.g. the following:
>  --8<---------------cut here---------------start------------->8---
>  depends: [
>    "ocaml" {>= "4.04.2"}
>    "base"  {>= "v0.13" & < "v0.14"}
>    "dune"  {>= "1.5.1"}
>  ]
>  --8<---------------cut here---------------end--------------->8---
>  would translate to:
>  --8<---------------cut here---------------start------------->8---
>  BuildRequires: Opam(ocaml) >= 4.04.2
>  BuildRequires: Opam(base) >= 0.13 and Opam(base) < 0.14
>  BuildRequires: Opam(dune) >= 1.5.1
>  --8<---------------cut here---------------end--------------->8--- 
Agreed, if Opam() was replaced with ocaml-

>  The hypothetical opam2spec would be certainly also a huge
> improvement,
>  as I think that most ocaml packages could be completely autogenerated.
>  
>  But as you already said, the devil is most likely in the details, so
>  maybe we should start collecting ideas and potential pitfalls first? I
>  have therefore also cc'd Olaf, who takes care of Ocaml in
>  openSUSE. I would hope that we can coordinate our efforts, as Olaf has
>  some pretty nice ocaml macros that automatically generate file lists for
>  the subpackages (thereby already simplifying packaging quite a bit).
>  
>  And at last I'd like to throw in a (probably quite controversial) idea:
>  how about we move ocaml into a module? Not to be able to have multiple
>  ocaml versions as streams, but so that we can use MBS to automatically
>  untangle the bootstrap loops for us. I was totally not aware of this
>  feature of MBS, but it was mentioned by Stephen in this email:
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
I think I've suffered enough from having to deal with the virt module
in RHEL without having more modularity in my life.  In any case MBS
doesn't actually solve dependencies (in virt:rhel we have to list them
all explicitly, which is even worse than the current situation), and
I've already solved the dependency problem properly in my goals script.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Jerry James]

On Thu, Jun 25, 2020 at 4:16 AM Dan Čermák
<dan.cermak(a)cgc-instruments.com&gt; wrote:
>  %generate_buildrequires could be used to parse the opam file from
> the
>  tarball and extract the dependent opam packages from the depends:
>  array. E.g. the following: 
I've been thinking along those lines as well.  More automation would be great.

>  The hypothetical opam2spec would be certainly also a huge
> improvement,
>  as I think that most ocaml packages could be completely autogenerated. 
My experience is that the build dependencies in the opam files are of
high quality, but that test and documentation dependencies are often
omitted.  We could add those manually, so that may not be a big deal.

I have found myself copying lots of boilerplate from spec file to spec
file while adding packages.  I think I've only encountered a few
variations, which seem to be centered around the chosen build tool:
- dune
- topkg
- ocamlbuild
- ocamlc/ocamlopt

with increasing entropy as you go down the list. :-)

>  But as you already said, the devil is most likely in the details, so
>  maybe we should start collecting ideas and potential pitfalls first? I
>  have therefore also cc'd Olaf, who takes care of Ocaml in
>  openSUSE. I would hope that we can coordinate our efforts, as Olaf has
>  some pretty nice ocaml macros that automatically generate file lists for
>  the subpackages (thereby already simplifying packaging quite a bit). 
By all means, let's draw from the experiences of those who have
traveled this road already.  Olaf, thanks for chiming in.  I'm eager
to learn from your experiences.

I'm glad the idea of opam2spec was brought up.  I've been thinking
about that, and a couple of other tools:

Tool 1: visualize package dependencies

The idea is to pull dependency information from upstream sources [1],
then draw a graph representing the dependencies.  We distinguish
between different kinds of dependencies:
- build
- documentation
- test
and further distinguish each dependency as "required" or "optional".
The graph uses colors, line styles, or both to distinguish between the
various dependency types.

I want an option to highlight dependency loops (perhaps cycling
through them one at a time if there is more than one).  There sure are
a lot of those in the OCaml world (when testing and documentation
dependencies are included).

Finally, I want the tool to help me visualize the effects of various
strategies for breaking loops.  That is, I want the tool to
automatically find some way of breaking loops so that a directed
acyclic graph is produced.  For example, I might say that when loops
are broken, the edges to remove are chosen to minimize the number of
edge removals, and use this preference order on the edge types (from
most preferred to least preferred type to remove):
- optional test
- required test
- optional documentation
- required documentation
- optional build
- required build

The tool should make it easy for me to select different orderings of
the preferred types, and give me a count of the number of removed
edges in each case.  Finding a minimal set of edges to remove makes
this an expensive operation in the general case, but I expect that the
typical use of such a tool would be on a few hundred nodes at most,
which shouldn't be a big deal on modern computers.

This tool will help me decide which dependencies should be hidden
behind a %bcond_with of one kind or another.  In particular, I'm
interested in using %bcond_with bootstrap because of the next tool.

Tool 2: compute a build order

Such a tool has been mentioned on this mailing list a number of times.
Richard Jones has a script that does this for OCaml, but I think it
could be generalized to RPM packages in general with some effort.  It
would work something like this:

Stage 1: read spec files (possibly specified as a list of checked-out
git repositories, like Richard's script) to generate two lists per
input:
- BuildRequires when --with-bootstrap is given
- BuildRequires when --without-bootstrap is given
If the two lists are identical, then discard the --with-bootstrap list.

Stage 2: Depsolve to convert each list into a list of package names;
i.e. convert logical, file, and boolean dependencies into explicit
package names.  If there are two lists, check again if they are
identical (they shouldn't be, but let's be careful), and discard the
--with-bootstrap list if so.

Stage 3: Construct a graph with one node for each list, labeled with
the name of the package/spec file from which the list was generated.
If it is a --with-bootstrap list, then give the node type BOOTSTRAP,
otherwise give it type NORMAL.

Stage 4: For each list L and each package P in L, if there are no
nodes in the graph corresponding to P, then use repoquery to get a
list of --without-bootstrap BuildRequires for P.  Make a node for P
and give it type DONOTBUILD.

Stage 5: For each node N in the graph, if N's list of BuildREquires
contains another node M in the graph, then add an edge from M to N.
If there are both BOOTSTRAP and NORMAL nodes for M, then make the edge
start at the BOOTSTRAP node.

Stage 6: Run a cycle detector on the graph.  If any cycles are found,
report them and exit.  Somebody hasn't broken a cycle with %{with
bootstrap} and has to fix that before we can continue.  See tool 1.
If we have not exited, then the graph is a DAG (directed acyclic
graph).

Stage 7: For each BOOTSTRAP node N in the graph, if the NORMAL node M
with the same name is not reachable from N, then:
- Merge all BuildRequires listed on N into M.
- If M now has BuildRequires on both the BOOTSTRAP and NORMAL versions
of some package, then discard the BOOTSTRAP BuildRequires.
- Discard N.
The graph is still a DAG because otherwise M would have been reachable from N.

Stage 8: Traverse the graph in DAG order; i.e., starting from nodes
with no incoming edges.  For each DONOTBUILD node N visited:
- Merge all of N's incoming edges (if any) into the nodes on the other
side of its outgoing edges.
- Remove N and all outgoing edges from the graph.

At this point, we have a DAG giving a partial order on the packages to
be built, each one labeled either BOOTSTRAP or NORMAL.  Build them any
way you like that is consistent with the partial order and the node
types.

I may have overthought this a bit. :-)  If either tool exists
somewhere, even in an unfinished state, tell me!  Otherwise, since I'm
approximately the world's worst at choosing project names [2],
somebody do that and I'll chip in with some code.

Footnotes:
[1] We would pull from opam.ocaml.org in this case, but if the tool is
designed well, it could have "plugins" or "modules" that know how to
read from a variety of sources.  I'm interested in having this work
with the GAP packages I maintain, for example, which also have lots of
dependency loops involving test and documentation.
[2] For tool 1 ... dagomatic?  See, I told you!
-- 
Jerry James
http://www.jamezone.org/

[Stephen J. Turnbull]

Jerry James writes:

>  My experience is that the build dependencies in the opam files are
> of
>  high quality, but that test and documentation dependencies are often
>  omitted.  We could add those manually, so that may not be a big deal. 
If you add them manually, maybe upstream would accept a patch based on
that addition?  That could contribute to better support going forward.

Steve

[Richard W.M. Jones]

FWIW looks like OCaml 4.11 beta 1 was released earlier this week.

  https://discuss.ocaml.org/t/ocaml-4-11-first-beta-release/6042

So I will try a mass rebuild into a side tag soon, and see what the
fallout is.  I believe I've added all the new packages added recently
to my list.

I've no intention of making any changes to packaging standards or
dependencies at this time, it would just be a simple rebuild using the
new compiler.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v