FYI: New OCaml ARM code generator

[Richard W.M. Jones]

I'm in the process of backporting the new, rewritten OCaml ARM code
generator to OCaml 3.12.1.  This may necessitate recompiling all OCaml
packages in ARM, but I'll also start on that tomorrow.

The particular reason for doing this is support for natdynlink
(dynamic linking of OCaml code -- think of it as the equivalent of
'dlopen' for OCaml native code).  Having this will enhance a few OCaml
packages in Fedora/ARM and bring them up to par with their x86 cousins.

Here are the highlights of the new implementation taken from the
upstream commit message:

  - Support for both software and hardware floating-point (VFPv3).

  - Properly supports interworking with Thumb/Thumb-2 code for both
    OCaml and C code.

  - Supports dynamic linking and large memory models (PR#5049).

  - Optional support for position-independent code via a command line
    option -fPIC. This is disabled by default and not required for
    natdynlink.

  - Can emit both ARM and Thumb-2 code, with avg. code size savings of
    28% for Thumb-2 (quite close the optimal 30% advertised by ARM
    Ltd.).

  - Supports both AAPCS (armel) as well as extended VFP calling
    conventions (armhf).

  - Supports several special ARM instructions to reduce code size and
    latency.

  - Uses standard ARM EABI runtime functions instead of relying on GCC
    internals.

  - Supports exception backtraces.

  - Supports profiling using gprof.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw

[Peter Robinson]

Hi Richard,

All sounds really cool.

On Fri, Apr 27, 2012 at 10:46 PM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  I'm in the process of backporting the new, rewritten OCaml ARM
> code
>  generator to OCaml 3.12.1.  This may necessitate recompiling all OCaml
>  packages in ARM, but I'll also start on that tomorrow. 
Basically for this you need to push the changes to mainline and
koji-shadow will pull it all in and deal with it. It builds all
packages with at least the NVR of any packages it was built with so it
will handle all the build ordering etc. I can send you the details of
any failures of that if you like from the logs as it runs though.

Cheers,
Peter

>  The particular reason for doing this is support for natdynlink
>  (dynamic linking of OCaml code -- think of it as the equivalent of
>  'dlopen' for OCaml native code).  Having this will enhance a few OCaml
>  packages in Fedora/ARM and bring them up to par with their x86 cousins.
>
>  Here are the highlights of the new implementation taken from the
>  upstream commit message:
>
>   - Support for both software and hardware floating-point (VFPv3).
>
>   - Properly supports interworking with Thumb/Thumb-2 code for both
>     OCaml and C code.
>
>   - Supports dynamic linking and large memory models (PR#5049).
>
>   - Optional support for position-independent code via a command line
>     option -fPIC. This is disabled by default and not required for
>     natdynlink.
>
>   - Can emit both ARM and Thumb-2 code, with avg. code size savings of
>     28% for Thumb-2 (quite close the optimal 30% advertised by ARM
>     Ltd.).
>
>   - Supports both AAPCS (armel) as well as extended VFP calling
>     conventions (armhf).
>
>   - Supports several special ARM instructions to reduce code size and
>     latency.
>
>   - Uses standard ARM EABI runtime functions instead of relying on GCC
>     internals.
>
>   - Supports exception backtraces.
>
>   - Supports profiling using gprof.
>
>  Rich.
>
>  --
>  Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
>  New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
>  programs, test, and build Windows installers. Over 70 libraries supprt'd
>  http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
>  _______________________________________________
>  arm mailing list
>  arm(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/arm

[Richard W.M. Jones]

On Fri, Apr 27, 2012 at 11:26:01PM +0100, Peter Robinson wrote:
>  Hi Richard,
>  
>  All sounds really cool.
>  
>  On Fri, Apr 27, 2012 at 10:46 PM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  > I'm in the process of backporting the new, rewritten OCaml ARM code
>  > generator to OCaml 3.12.1.  This may necessitate recompiling all OCaml
>  > packages in ARM, but I'll also start on that tomorrow.
>  
>  Basically for this you need to push the changes to mainline and
>  koji-shadow will pull it all in and deal with it. It builds all
>  packages with at least the NVR of any packages it was built with so it
>  will handle all the build ordering etc. I can send you the details of
>  any failures of that if you like from the logs as it runs though. 
OK, but I'll probably do a few builds by hand to check this new
code generator works ...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

[Peter Robinson]

On Fri, Apr 27, 2012 at 11:40 PM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  On Fri, Apr 27, 2012 at 11:26:01PM +0100, Peter Robinson wrote:
> > Hi Richard,
> >
> > All sounds really cool.
> >
> > On Fri, Apr 27, 2012 at 10:46 PM, Richard W.M. Jones <rjones(a)redhat.com&gt;
> wrote:
> > > I'm in the process of backporting the new, rewritten OCaml ARM code
> > > generator to OCaml 3.12.1.  This may necessitate recompiling all OCaml
> > > packages in ARM, but I'll also start on that tomorrow.
> >
> > Basically for this you need to push the changes to mainline and
> > koji-shadow will pull it all in and deal with it. It builds all
> > packages with at least the NVR of any packages it was built with so it
> > will handle all the build ordering etc. I can send you the details of
> > any failures of that if you like from the logs as it runs though.
>
>  OK, but I'll probably do a few builds by hand to check this new
>  code generator works ... 
You can use scratch-builds without issue, the problem with doing
manual builds is that if there's a soname bump it all ends up in a
mess.

Peter

[Richard W.M. Jones]

[Just to clarify my previous email, the ARM work was done upstream by
Benedikt Meurer of the University of Siegen.  I'm merely backporting
that work to Fedora.]

I've now updated the OCaml compiler (ocaml) and ocaml-findlib packages.

The other packages will need to be recompiled, and I'm working my way
through them.  Unfortunately (but perhaps not surprisingly) if you mix
code compiled with the old and new compiler, it doesn't work.

If you want to use the new compiler, just install ocaml >= 3.12.1-3.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Peter Robinson]

On Sat, Apr 28, 2012 at 7:29 PM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  [Just to clarify my previous email, the ARM work was done upstream
> by
>  Benedikt Meurer of the University of Siegen.  I'm merely backporting
>  that work to Fedora.]
>
>  I've now updated the OCaml compiler (ocaml) and ocaml-findlib packages.
>
>  The other packages will need to be recompiled, and I'm working my way
>  through them.  Unfortunately (but perhaps not surprisingly) if you mix
>  code compiled with the old and new compiler, it doesn't work.
>
>  If you want to use the new compiler, just install ocaml >= 3.12.1-3. 
You've done it all wrong. You've built packages in F-18 on mainline
and then built those packages on F-17 on ARM without them in F-17
mainline. The packages built on ARM _MUST_ be identical to those
packages on mainline. Please resolve this and please in the future let
koji deal with it. Let me know once it's all cleaned up or if you need
assistance in cleaning it up. If you're going to bump builds please do
it on the mainline using the usual process and let koji-shadow
properly deal with building packages on ARM.

Regards,
Peter

[Richard W.M. Jones]

On Sun, Apr 29, 2012 at 01:26:11PM +0100, Peter Robinson wrote:
>  On Sat, Apr 28, 2012 at 7:29 PM, Richard W.M. Jones
> <rjones(a)redhat.com&gt; wrote:
>  > [Just to clarify my previous email, the ARM work was done upstream by
>  > Benedikt Meurer of the University of Siegen.  I'm merely backporting
>  > that work to Fedora.]
>  >
>  > I've now updated the OCaml compiler (ocaml) and ocaml-findlib packages.
>  >
>  > The other packages will need to be recompiled, and I'm working my way
>  > through them.  Unfortunately (but perhaps not surprisingly) if you mix
>  > code compiled with the old and new compiler, it doesn't work.
>  >
>  > If you want to use the new compiler, just install ocaml >= 3.12.1-3.
>  
>  You've done it all wrong. You've built packages in F-18 on mainline
>  and then built those packages on F-17 on ARM without them in F-17
>  mainline. The packages built on ARM _MUST_ be identical to those
>  packages on mainline. Please resolve this and please in the future let
>  koji deal with it. Let me know once it's all cleaned up or if you need
>  assistance in cleaning it up. If you're going to bump builds please do
>  it on the mainline using the usual process and let koji-shadow
>  properly deal with building packages on ARM. 
Calm down!  It's only a small matter of cherry picking the patches
back to F17, which will take a matter of minutes to fix.  But first,
and more importantly, how can I get email from koji-shadow as to
whether the builds it is doing are successful or not?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Peter Robinson]

On Sun, Apr 29, 2012 at 3:31 PM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  On Sun, Apr 29, 2012 at 01:26:11PM +0100, Peter Robinson wrote:
> > On Sat, Apr 28, 2012 at 7:29 PM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
> > > [Just to clarify my previous email, the ARM work was done upstream by
> > > Benedikt Meurer of the University of Siegen.  I'm merely backporting
> > > that work to Fedora.]
> > >
> > > I've now updated the OCaml compiler (ocaml) and ocaml-findlib packages.
> > >
> > > The other packages will need to be recompiled, and I'm working my way
> > > through them.  Unfortunately (but perhaps not surprisingly) if you mix
> > > code compiled with the old and new compiler, it doesn't work.
> > >
> > > If you want to use the new compiler, just install ocaml >= 3.12.1-3.
> >
> > You've done it all wrong. You've built packages in F-18 on mainline
> > and then built those packages on F-17 on ARM without them in F-17
> > mainline. The packages built on ARM _MUST_ be identical to those
> > packages on mainline. Please resolve this and please in the future let
> > koji deal with it. Let me know once it's all cleaned up or if you need
> > assistance in cleaning it up. If you're going to bump builds please do
> > it on the mainline using the usual process and let koji-shadow
> > properly deal with building packages on ARM.
>
>  Calm down!  It's only a small matter of cherry picking the patches
>  back to F17, which will take a matter of minutes to fix.  But first,
>  and more importantly, how can I get email from koji-shadow as to
>  whether the builds it is doing are successful or not? 
Richard, we have processes for doing builds for a reason, so it's
simple for you to say calm down when you don't have to clean up the
mess or deal with rel-eng when the repositories don't align and cause
us issues.

It won't build in koji-shadow until it's at least built and tagged
into either f18 or f-17 updates-testing or stable. So you get things
built and once it starts to filter through I can give you failures.

Peter

[Richard W.M. Jones]

On Sun, Apr 29, 2012 at 03:37:31PM +0100, Peter Robinson wrote:
>  It won't build in koji-shadow until it's at least built and
> tagged
>  into either f18 or f-17 updates-testing or stable. So you get things
>  built and once it starts to filter through I can give you failures. 
OK, but this is very indirect.  There are long chains of dependencies
in the OCaml packages, and it will take forever if I can't build
directly for ARM.  The process (for each layer of dependency) will be
something like:

(1) Do scratch builds for ARM until the build(s) works.

(2) Submit a build to Rawhide/x86 and/or F17/x86.

(2a) [After waiting for the x86 build ...] submit an update to F17/x86.

(3) Wait some undetermined time for koji-shadow to do something and to
get feedback from you as to whether the ARM build succeeded.

(4) Go back to (1).

I don't doubt that koji-shadow is great for the bulk of builds that
had to be done to get ARM going, but as I say it's very indirect where
a maintainer is interested in fixing specific packages for ARM.

I know koji-shadow handles dependencies, but looking at the code I'm
not convinced it's going to be safe to build OCaml packages with the
new compiler until I've tested each one, and we mustn't let old+new
code to be combined (how to express that in RPM deps? not sure it's easy).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/