ROS packages

[Till Hofmann]

Hi all,

I'm currently working on packaging the whole ROS stack by automatically 
generating SPEC files with the help of the rosinstall_generator. I think 
the project is far enough to be shared on this list, but it is still 
work in progress.

The SPEC files are generated with a python script. You can check it out 
here: https://pagure.io/ros

I've created a COPR for ROS Kinetic that contains all packages from 
desktop_full and move_base for F24+ (currently only for x86_64):
https://copr.fedorainfracloud.org/coprs/thofmann/ros/

The scripts should also work for ROS Lunar (and possibly other 
releases), but I've only tested Kinetic so far.

I decided to package each ROS package into a separate Fedora package, 
because imho that's the best mapping from ROS to Fedora: Each package 
has its own sources, different dependencies, and some packages require 
different build arguments. This way, it's also much easier to add new 
(ROS) packages to Fedora.

I picked /usr/lib64/ros as ROS_ROOT. Currently, this does not allow 
having two separate ROS installations. But moving it to e.g., 
/usr/lib64/ros/kinetic should be as simple as adapting the template 
file. Is that something we would want to have?

There are still some remaining tasks on my list:
- Provides: for ros-$pkgname (currently, the packages are all called 
ros-kinetic-$pkgname)
- Clean up dependencies, many packages have explicit runtime 
dependencies that they don't need because they are automatic 
dependencies on the sonames. Some packages also have devel packages as 
runtime dependency, which is of course incorrect.
- Splitting devel packages. I'm not sure how useful this would be, but I 
think it is required by the packaging guidelines.
- I thought about adding the ROS lib dir to /etc/ld.so.conf.d, but I'm 
not sure whether this is a good idea because most libraries have 
unversioned sonames. On the other hand, all the packages currently 
Provide: their (unversioned) libraries, which may have unexpected 
effects, e.g., a program linking (and requiring) a ros lib without 
sourcing the ROS environment.
- As it is, the ROS packages have a lot of bundled third-party 
libraries. This may require some work to separate if at all possible.
- I haven't really checked any existing Fedora ROS packages other than 
catkin_pkg.

In the long run, it would be awesome to get those packages into Fedora 
(one-by-one), maybe even with different modules for the different ROS 
versions - modularity could be very useful here!

In the meantime, I'd appreciate any feedback, test results, RFEs, 
package requests, etc. The best place for this would be the repository:
https://pagure.io/ros/issues

We (team Carologistics, RCLL) have successfully tested some of those 
packages on Fedora 25 at the last RoboCup in Japan, but of course more 
tests would be nice, as we only use a few packages.

Please let me know what you think!


Kind regards,
Till

[Tom Callaway]

On 08/16/2017 04:19 PM, Till Hofmann wrote:
>  Hi all,
>  
>  I'm currently working on packaging the whole ROS stack by automatically
>  generating SPEC files with the help of the rosinstall_generator. I think
>  the project is far enough to be shared on this list, but it is still
>  work in progress.
>  
>  The SPEC files are generated with a python script. You can check it out
>  here: https://pagure.io/ros
>  
>  I've created a COPR for ROS Kinetic that contains all packages from
>  desktop_full and move_base for F24+ (currently only for x86_64):
>  https://copr.fedorainfracloud.org/coprs/thofmann/ros/
>  
>  The scripts should also work for ROS Lunar (and possibly other
>  releases), but I've only tested Kinetic so far.
>  
>  I decided to package each ROS package into a separate Fedora package,
>  because imho that's the best mapping from ROS to Fedora: Each package
>  has its own sources, different dependencies, and some packages require
>  different build arguments. This way, it's also much easier to add new
>  (ROS) packages to Fedora.
>  
>  I picked /usr/lib64/ros as ROS_ROOT. Currently, this does not allow
>  having two separate ROS installations. But moving it to e.g.,
>  /usr/lib64/ros/kinetic should be as simple as adapting the template
>  file. Is that something we would want to have?
>  
>  There are still some remaining tasks on my list:
>  - Provides: for ros-$pkgname (currently, the packages are all called
>  ros-kinetic-$pkgname)
>  - Clean up dependencies, many packages have explicit runtime
>  dependencies that they don't need because they are automatic
>  dependencies on the sonames. Some packages also have devel packages as
>  runtime dependency, which is of course incorrect.
>  - Splitting devel packages. I'm not sure how useful this would be, but I
>  think it is required by the packaging guidelines.
>  - I thought about adding the ROS lib dir to /etc/ld.so.conf.d, but I'm
>  not sure whether this is a good idea because most libraries have
>  unversioned sonames. On the other hand, all the packages currently
>  Provide: their (unversioned) libraries, which may have unexpected
>  effects, e.g., a program linking (and requiring) a ros lib without
>  sourcing the ROS environment.
>  - As it is, the ROS packages have a lot of bundled third-party
>  libraries. This may require some work to separate if at all possible.
>  - I haven't really checked any existing Fedora ROS packages other than
>  catkin_pkg. 
Wow, thanks for taking this on. I haven't had any cycles to work on ROS
in a long time, but this is work I support.

* I do think that making -devel packages might be useful, but if you can
make the case as to why they aren't useful (as opposed to "harder to
script"), you can probably get an exception for ROS packages.

* I don't think you want ROS libs in the ld cache, that seems like a
recipe for disaster. You also probably want to disable the autoprovides
generation and instead do something like:

Provides: ros(library.so)

or

Provides: ros-kinetic(library.so)

* The unbundling is going to suck, except in the rare cases where the
bundled code is unmodified. Thankfully, you should be able to use the
bundled code as long as you follow the Fedora Packaging Guidelines
around that: https://fedoraproject.org/wiki/Bundled_Libraries

~tom

[Till Hofmann]

Thanks for the feedback!

On 08/16/2017 10:29 PM, Tom Callaway wrote:
>  * I do think that making -devel packages might be useful, but if you
> can
>  make the case as to why they aren't useful (as opposed to "harder to
>  script"), you can probably get an exception for ROS packages. 
My first thought was that in most use cases, people would use those ROS 
packages as base to build their own software on top of it, which would 
always require the devel package anyway. But on second thought, that's 
only for some of the packages, and people may want to just run ROS 
without a lot of custom software, so having separate devel packages is 
probably useful, I just haven't started working on the split.
>  
>  * I don't think you want ROS libs in the ld cache, that seems like a
>  recipe for disaster. You also probably want to disable the autoprovides
>  generation and instead do something like:
>  
>  Provides: ros(library.so)
>  
>  or
>  
>  Provides: ros-kinetic(library.so) 
That seems like a good idea, although I would have liked to use more of 
RPM's implicit dependency management instead of essentially re-adding 
all those Provides manually. Do you have an idea how we could add those 
Provides automatically, or at least in a scripted way without looking at 
the resulting packages first? So far I can generate all SPEC files 
without looking into the generated RPMs, (with the exceptions that have 
custom configs and/or templates) it would be nice if we could keep it 
that way.
>  
>  * The unbundling is going to suck, except in the rare cases where the
>  bundled code is unmodified. Thankfully, you should be able to use the
>  bundled code as long as you follow the Fedora Packaging Guidelines
>  around that: https://fedoraproject.org/wiki/Bundled_Libraries 
Yes, that's what I was aiming for. Unbundling would be ideal, but 
probably not feasible. But this is also a step that has to be done 
separately for each package, so far I've been trying to avoid this 
whenever possible.

Kind regards,
Till

[Tom Callaway]

On 08/16/2017 05:08 PM, Till Hofmann wrote:
>  Do you have an idea how we could add those Provides automatically, or
> at
>  least in a scripted way without looking at the resulting packages first?
>  So far I can generate all SPEC files without looking into the generated
>  RPMs, (with the exceptions that have custom configs and/or templates) it
>  would be nice if we could keep it that way. 
Hmm. It _might_ be possible to write a generic converter macro that
parsed the generated provides and replaced any Provides for unversioned
libraries with the special ros Provide.

Or, you could make something that is the ros equivalent of
/usr/lib/rpm/perl.prov. You can look through /usr/lib/rpm/ at all the
other .prov files for inspiration. (This seems like the better path forward)

~tom

[Till Hofmann]

On 08/16/2017 10:29 PM, Tom Callaway wrote:
>  * I don't think you want ROS libs in the ld cache, that seems
> like a
>  recipe for disaster. You also probably want to disable the autoprovides
>  generation and instead do something like:
>  
>  Provides: ros(library.so)
>  
>  or
>  
>  Provides: ros-kinetic(library.so) 
I've figured how to do that with a __ros_provides script (once I 
understood the RPM macro system, this was actually quite simple). But 
I'm not sure how to translate the Requires: accordingly. I have ROS libs 
that "Provides: ros(libfoo.so)", but I don't know how to create a 
"Requires: ros(libfoo.so)" in a package that uses libfoo.

So far I see two options:
1) Run /usr/lib/rpmdeps from my __ros_requires script and translate ROS 
libs accordingly. But this would mean I would have to check for each 
dependency whether it is part of a ROS package.
2) Don't use rpmdeps but instead run ldd manually and filter ldd's 
output. This makes the distinction between ROS libs and other libs 
easier, but I would have to recreate the behavior of rpmdeps, which 
sounds like a terrible idea.

So neither option seems really satisfactory. Any other ideas?

Kind regards,
Till

[Jose Luis Rivero]

Hello Till:

On 16/08/17 22:19, Till Hofmann wrote:
>  Hi all,
>  
>  I'm currently working on packaging the whole ROS stack by automatically
>  generating SPEC files with the help of the rosinstall_generator. I think
>  the project is far enough to be shared on this list, but it is still
>  work in progress.
>   
I just want to mention that there was a previous effort to package ROS
in Fedora in the past, mostly done by Rich Mattes:
- https://fedoraproject.org/wiki/SIGs/Robotics/ROS_Packaging

A similar work has been done in Debian recently:
- https://wiki.debian.org/DebianScience/Robotics/ROS/Motivation

You can check their approach which does not rely in ROS distribution
(ROS debian packages[1] has no ROS distribution in their names) and also
map one ROS package with one downstream distribution package.

Some patches in Debian and specially the work done in Fedora before
could be useful to you.

Good luck!

[1] https://wiki.debian.org/DebianScience/Robotics/ROS/Packages

-- 
Jose Luis Rivero <jrivero(a)osrfoundation.org&gt;

[Rich Mattes]

On Thu, Aug 17, 2017 at 5:20 PM, Jose Luis Rivero
<jrivero(a)osrfoundation.org&gt; wrote:
>  Hello Till:
>
>  On 16/08/17 22:19, Till Hofmann wrote:
> > Hi all,
> >
> > I'm currently working on packaging the whole ROS stack by automatically
> > generating SPEC files with the help of the rosinstall_generator. I think
> > the project is far enough to be shared on this list, but it is still
> > work in progress.
> >
>
>  I just want to mention that there was a previous effort to package ROS
>  in Fedora in the past, mostly done by Rich Mattes:
>  - https://fedoraproject.org/wiki/SIGs/Robotics/ROS_Packaging
>
>  A similar work has been done in Debian recently:
>  - https://wiki.debian.org/DebianScience/Robotics/ROS/Motivation
>
>  You can check their approach which does not rely in ROS distribution
>  (ROS debian packages[1] has no ROS distribution in their names) and also
>  map one ROS package with one downstream distribution package.
>
>  Some patches in Debian and specially the work done in Fedora before
>  could be useful to you.
>
>  Good luck!
>
>  [1] https://wiki.debian.org/DebianScience/Robotics/ROS/Packages
>  
Hi Till,

This is great, thanks for taking this on!  Expanding on what Jose
mentioned: there have been a couple efforts to support ROS packaging
in Fedora, none of which made it all the way to a usable product.  A
little bit of history:

As Jose said, I did try to package ROS in Fedora, around the fuerte
timeframe in 2012.  As a result of that effort, about half of the core
ROS stack as outlined on the robotics sig wiki page is packaged for
fedora, and the other half is sitting in review requests.  When I did
it, I took the approach that there would be one ros distro provided in
the core Fedora repositories, and that they would be installed like
any other system package (e.g. libs in /usr/lib<qual>, config in /etc,
data in /usr/share.)  I figured that if anyone needed support for a
separate ros distro, they could continue to build manually and install
to /opt.  I was also creating the specfiles manually, so that I could
create proper subpackages (-devel, -doc, etc.)  in each individual
case.  I think that this effort closely maps to the debian science
efforts, though they got a lot farther than I did.  I did recently
look at trying to bring the debian science patches to fedora to
restart the packaging process, but I haven't had the time to dedicate
to such an effort.

Around the 2014 timeframe, we started to investigate the possibility
of having the ros buildfarm generate fedora RPMs in parallel to the
Ubuntu debs, so that all of the ros distros would be available in a
manner similar to Ubuntu.  Scott Logan did the work there, and landed
patches to the ROS 'bloom' tool enabling it to generate spec files for
fedora.  That infrastructure is in place, but we got hung up in
working with the OSRF to enable RPM building in their buildfarm, and
then that fizzled out as well.

As it stands today, a lot of ROS dependencies are packaged in Fedora,
making it possible to build from source pretty easily.  I think that
to move forward safely, you need to decide for the long term whether
you want to support:
- ROS installation via OSRF resources
- ROS installation via copr
- ROS packages in the distribution proper

Supporting ROS as a copr gives you a lot of leeway to do things that
don't conform to the packaging guidelines, such as installing parallel
stacks to /opt, creating single rpms for each package, etc.  To bring
the packages into fedora proper, as Tom was getting at, you need to
reconcile how to support multiple software stacks within the package
guidelines (and possibly get some exemptions to the guidelines,) and
how to auto-generate specs that meet the packaging guidelines (proper
division of -devel packages, etc.)

In case that wasn't enough to worry about, there is also a
"Modularity" effort going on in Fedora.  I have not been able to
figure out what they're doing, but [1] claims that it's modules can
have different lifecycles than the base distribution.  It may provide
a model to maintain multiple ROS stacks in parallel, but I don't
really have any more information than that.

I'm happy to add you to the existing ROS packages that have been
reviewed, add you to the robotics-sig user group, and continue to
discuss on the mailing list.

As a side note, I've recently been poking at ros2 packaging[2].  The
ecosystem is a lot smaller at the moment, and it may be able to re-use
some of the concepts and tooling that get worked out for ros 1.


[1] https://docs.pagure.org/modularity/
[2] https://copr.fedorainfracloud.org/coprs/g/robotics-sig/ros2/

[Hedayat Vatankhah]

Hi

/*Rich Mattes*/ wrote on Sun, 20 Aug 2017 12:29:12 -0400:
>  <...>
>  In case that wasn't enough to worry about, there is also a
>  "Modularity" effort going on in Fedora.  I have not been able to
>  figure out what they're doing, but [1] claims that it's modules can
>  have different lifecycles than the base distribution.  It may provide
>  a model to maintain multiple ROS stacks in parallel, but I don't
>  really have any more information than that. 
Yes, it provides a model to maintain
multiple ROS stacks in parallel. 
However, currently, they are most likely assuming that only a single 
stack can be installed at a given time (if they just 'assume' it, it 
should be fine. But if they enforce it, you'll be unable to install 
multiple stacks simultaneously even if their packages doesn't conflict).

Regards,
Hedayat


>  <...>
>  [1] https://docs.pagure.org/modularity/
>

[Till Hofmann]

Hi all,

thanks again to everyone for the valuable feedback!

On 08/20/2017 06:29 PM, Rich Mattes wrote:
>  On Thu, Aug 17, 2017 at 5:20 PM, Jose Luis Rivero
>  <jrivero(a)osrfoundation.org&gt; wrote:
> > Hello Till:
> >
> > On 16/08/17 22:19, Till Hofmann wrote:
> >> Hi all,
> >>
> >> I'm currently working on packaging the whole ROS stack by automatically
> >> generating SPEC files with the help of the rosinstall_generator. I think
> >> the project is far enough to be shared on this list, but it is still
> >> work in progress.
> >>
> >
> > I just want to mention that there was a previous effort to package ROS
> > in Fedora in the past, mostly done by Rich Mattes:
> > - https://fedoraproject.org/wiki/SIGs/Robotics/ROS_Packaging
> >
> > A similar work has been done in Debian recently:
> > - https://wiki.debian.org/DebianScience/Robotics/ROS/Motivation
> >
> > You can check their approach which does not rely in ROS distribution
> > (ROS debian packages[1] has no ROS distribution in their names) and also
> > map one ROS package with one downstream distribution package.
> >
> > Some patches in Debian and specially the work done in Fedora before
> > could be useful to you.
> >
> > Good luck!
> >
> > [1] https://wiki.debian.org/DebianScience/Robotics/ROS/Packages
> >
>  
>  Hi Till,
>  
>  This is great, thanks for taking this on!  Expanding on what Jose
>  mentioned: there have been a couple efforts to support ROS packaging
>  in Fedora, none of which made it all the way to a usable product.  A
>  little bit of history:
>  
>  As Jose said, I did try to package ROS in Fedora, around the fuerte
>  timeframe in 2012.  As a result of that effort, about half of the core
>  ROS stack as outlined on the robotics sig wiki page is packaged for
>  fedora, and the other half is sitting in review requests.  When I did
>  it, I took the approach that there would be one ros distro provided in
>  the core Fedora repositories, and that they would be installed like
>  any other system package (e.g. libs in /usr/lib<qual>, config in /etc,
>  data in /usr/share.)  I figured that if anyone needed support for a
>  separate ros distro, they could continue to build manually and install
>  to /opt.  I was also creating the specfiles manually, so that I could
>  create proper subpackages (-devel, -doc, etc.)  in each individual
>  case.  I think that this effort closely maps to the debian science
>  efforts, though they got a lot farther than I did.  I did recently
>  look at trying to bring the debian science patches to fedora to
>  restart the packaging process, but I haven't had the time to dedicate
>  to such an effort. >
>  Around the 2014 timeframe, we started to investigate the possibility
>  of having the ros buildfarm generate fedora RPMs in parallel to the
>  Ubuntu debs, so that all of the ros distros would be available in a
>  manner similar to Ubuntu.  Scott Logan did the work there, and landed
>  patches to the ROS 'bloom' tool enabling it to generate spec files for
>  fedora.  That infrastructure is in place, but we got hung up in
>  working with the OSRF to enable RPM building in their buildfarm, and
>  then that fizzled out as well. 

Thanks for the summary, I've seen the Fedora wiki pages on ROS, although 
I didn't realize that many of those are already available in the 
repositories. I'm not seeing a list of the ROS packages packaged for 
Fedora, is that available somewhere? The build farm approach sounds like 
an interesting alternative, do you have a link where I can get further 
information about that?

I decided to start packaging from scratch rather than building on top of 
the existing packages for several reasons:
1. Since fuerte, there were quite some changes ROS in general and the 
build system in particular. I thought it would be easier to start from 
scratch, rather than building on top of the fuerte packages.
2. I want to avoid maintaining >200 SPEC files manually, and therefore 
came up with the SPEC file generation approach. Of course modifications 
may be necessary for some packages, but so far the number of 
package-specific changes is surprisingly small.
3. For me, the approach to put ROS packages in a separate directory 
(e.g., /usr/lib<qual>/ros) seemed like a better idea for now, because of 
missing SO versions and because it seems like this is still expected by 
ROS in many places. As an example it's not possible to provide separate 
install paths for the different target types such as binaries, 
libraries, devel files and so on. Additionally, in my understanding, 
packaging ROS in /usr/lib<qual>/ros is not prohibited by the packaging 
guidelines. Also, even if we package ROS in /usr/lib<qual>/ros, it 
provides several benefits over a local build in /opt: It is much easier 
to set up, especially if you have to manage multiple robots, it provides 
automatic updates, you can easily select and un-select packages that you 
want to have installed, you don't need to build locally, and there are 
probably many more reasons. In fact, I started packaging ROS because I 
was annoyed by managing 6 ROS installations on our robots, and only 
afterwards realized that the approach may be useful for distro-wide 
packages.

Anyway, I'd like to have a closer look at the existing packages, because 
I assume that they carry some patches that may still be useful, even 
when generating SPEC files.

>  
>  As it stands today, a lot of ROS dependencies are packaged in Fedora,
>  making it possible to build from source pretty easily.  I think that
>  to move forward safely, you need to decide for the long term whether
>  you want to support:
>  - ROS installation via OSRF resources
>  - ROS installation via copr
>  - ROS packages in the distribution proper
>  
>  Supporting ROS as a copr gives you a lot of leeway to do things that
>  don't conform to the packaging guidelines, such as installing parallel
>  stacks to /opt, creating single rpms for each package, etc.  To bring
>  the packages into fedora proper, as Tom was getting at, you need to
>  reconcile how to support multiple software stacks within the package
>  guidelines (and possibly get some exemptions to the guidelines,) and
>  how to auto-generate specs that meet the packaging guidelines (proper
>  division of -devel packages, etc.) 
In the long run, my goal is to provide proper distribution packages for 
ROS, with COPR as a playing ground for development and testing. 
Splitting devel files into a sub-package should be fairly easy. Since 
the packaging guidelines regarding bundled libraries changed recently 
(last year?) and now allow bundled libraries as long as the package 
provides the respective bundled(library)s, unbundling all libraries is 
also not necessary anymore. Finding out all bundled libraries may result 
in a lot package-specific work, but at least we have an easy way to add 
these to the generated SPEC files. I think these are the main problems 
regarding the packaging guidelines.

>  
>  In case that wasn't enough to worry about, there is also a
>  "Modularity" effort going on in Fedora.  I have not been able to
>  figure out what they're doing, but [1] claims that it's modules can
>  have different lifecycles than the base distribution.  It may provide
>  a model to maintain multiple ROS stacks in parallel, but I don't
>  really have any more information than that. 
I think modularity only helps us, and we could just decide to ignore it 
for now. In my understanding, it could provide us an easy way to support 
multiple ROS releases for the same Fedora release, e.g., having both 
kinetic and lunar modules for F26 without the hack of having the version 
in the name. That would mean that there is no ros-kinetic-desktop 
package, but instead two ros-desktop packages in separate modules.
>  
>  I'm happy to add you to the existing ROS packages that have been
>  reviewed, add you to the robotics-sig user group, and continue to
>  discuss on the mailing list. 
Yes, that would be nice as it would already give me more insight in the 
current status of ROS in Fedora. Please also add me to the user group.

I can grant access to the ROS pagure and my ROS COPR to anyone interested.

>  
>  As a side note, I've recently been poking at ros2 packaging[2].  The
>  ecosystem is a lot smaller at the moment, and it may be able to re-use
>  some of the concepts and tooling that get worked out for ros 1. 
I was totally unaware of ros2, thanks for the hint!


Kind regards,
Till