A need for build triggers & automatic rebuilds

[Krzysztof Daniel]

Hello everyone,

I'm responsible for packaging Eclipse, which is *very* vulnerable to the
change of certain subset (10-15 or so) of its dependencies. The point is
that those dependencies, when rebuilt, frequently change their version,
and Eclipse is unable to start in most cases.

I'd like to be able to define a build trigger for Eclipse, so when a
"vital" dependency changes, Eclipse is rebuild, BR/R are updated, and a
bodhi update is performed. And I get a notification :-).

Is there a way to do so?

-- 
Krzysztof Daniel <kdaniel(a)redhat.com&gt;
Red Hat

[Florian Weimer]

On 06/19/2013 10:07 AM, Krzysztof Daniel wrote:
>  I'm responsible for packaging Eclipse, which is *very* vulnerable
> to the
>  change of certain subset (10-15 or so) of its dependencies. The point is
>  that those dependencies, when rebuilt, frequently change their version,
>  and Eclipse is unable to start in most cases. 
Are these failures artificially introduced by OSGi?  If yes, the version 
checks should be relaxed to match the RPM dependencies (or the RPM 
dependencies tightened to match the OSGi dependencies).  This would help 
with automated tracking.

-- 
Florian Weimer / Red Hat Product Security Team

[Nicolas Mailhot]

Le Mer 19 juin 2013 10:07, Krzysztof Daniel a écrit :
>  Hello everyone, 
>  I'd like to be able to define a build trigger for Eclipse, so
> when a
>  "vital" dependency changes, Eclipse is rebuild, BR/R are updated, and a
>  bodhi update is performed. And I get a notification :-).
>
>  Is there a way to do so? 
You don't need a special trigger. You need your requirements to match the
constraints declared at the rpm level, then incompatibilities will be
caught at this level by normal distro tooling.

Regards,

-- 
Nicolas Mailhot

[Eric Smith]

On Wed, Jun 19, 2013 at 4:23 AM, Nicolas Mailhot
<nicolas.mailhot(a)laposte.net&gt; wrote:
>  You don't need a special trigger. You need your requirements to
> match the
>  constraints declared at the rpm level, then incompatibilities will be
>  caught at this level by normal distro tooling. 
How does that work?  A pointer to the docs is fine; My own search
apparently hasn't turned up the right stuff.

Eric

[Adam Williamson]

On Wed, 2013-06-19 at 10:00 -0600, Eric Smith wrote:
>  On Wed, Jun 19, 2013 at 4:23 AM, Nicolas Mailhot
>  <nicolas.mailhot(a)laposte.net&gt; wrote:
>  > You don't need a special trigger. You need your requirements to match the
>  > constraints declared at the rpm level, then incompatibilities will be
>  > caught at this level by normal distro tooling.
>  
>  How does that work?  A pointer to the docs is fine; My own search
>  apparently hasn't turned up the right stuff. 
I'm not sure I'd concur with Nicolas that our mechanisms in this area
are entirely sufficient.

If you set up your RPM dependencies to match the actual code
dependencies, then when someone submits an update for one of the
dependencies that breaks the chain, the 'depcheck' AutoQA test ought to
fail, and I think the 'updates-testing report' mail sent to this list
should highlight the issue. But neither of those are 'enforced'
mechanisms: neither would prevent the update being pushed anyway. They
are both advisory. So it would still certainly be the case that this
could wind up out of sync.

Also, putting on my Update Nerd hat for a moment, our current setup not
only *doesn't encourage* coherent Bodhi updates, it more or less
*actively discourages* them. It's very difficult for packagers to edit
each other's updates - even provenpackagers - though pp's can edit
anyone's spec file at the drop of a hat. So if someone else submits an
update for one of Eclipse's dependencies which requires Eclipse to be
updated too, it's very unlikely that the Eclipse maintainers will be
able to do a new Eclipse build and edit it into the same update. Even
though that's how it's supposed to be: updates are supposed to be
internally consistent. If A depends on B, then when B is updated in a
way that breaks A, the Bodhi update should contain builds of both A and
B. There should not be separate Bodhi updates for A and B. But it is
actually very difficult to make this happen, given the way our
permissions system and Bodhi currently work. You have to really
carefully arrange things and have a very high degree of co-operation
between the packagers of A and B: basically, you have to make sure
someone has the necessary rights to be able to submit updates for both A
and B, and no-one else ever submits updates for B, or else even that
person can't fix things. It kind of stinks.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Krzysztof Daniel]

On Wed, 2013-06-19 at 12:23 +0200, Nicolas Mailhot wrote:
>  Le Mer 19 juin 2013 10:07, Krzysztof Daniel a écrit :
>  > Hello everyone,
>  
>  > I'd like to be able to define a build trigger for Eclipse, so when a
>  > "vital" dependency changes, Eclipse is rebuild, BR/R are updated, and a
>  > bodhi update is performed. And I get a notification :-).
>  >
>  > Is there a way to do so?
>  
>  You don't need a special trigger. You need your requirements to match the
>  constraints declared at the rpm level, then incompatibilities will be
>  caught at this level by normal distro tooling.
>  
>  Regards,
>  
>  -- 
>  Nicolas Mailhot
>   
Not really. I can set up BR/R in two ways: 
(greater or equal) or exact match.

The problem with greater or equal is that Eclipse internally reads a
bundle manifest, and records a version, which sometimes can change after
plain rebuild (because a timestamp is added).

Plain Eclipse example:

Eclipse requires jetty, and it symlinks a jetty bundles.
 org.eclipse.jetty.http_9.0.3.v20130506.jar
-> /usr/share/java/jetty/jetty-http.jar

OSGI records that there is a file
org.eclipse.jetty.http_9.0.3.v20130506.jar that holds a plugin with
version 9.0.3.v20130506. That version goes at the build time in a couple
of places (including metabundle).

If jetty is rebuilt, and it changes let's say to 9.0.4 (minor update),
Eclipse metadata gets out of sync. Eclipse sees a jar
org.eclipse.jetty.http_9.0.3.v20130506.jar, but version inside the
manifest is 9.0.4. Since at the build time Eclipse used 9.0.3, a lot of
meta bundles will not be resolved.

At this point I can only rebuild Eclipse (yes, the build works even if
Eclipse is not working).

Exact match can't be used at all, because if jetty is updated, then it
will be impossible to install Eclipse.

-- 
Krzysztof Daniel <kdaniel(a)redhat.com&gt;
Red Hat

[Florian Weimer]

On 06/21/2013 08:28 AM, Krzysztof Daniel wrote:

>  OSGI records that there is a file
>  org.eclipse.jetty.http_9.0.3.v20130506.jar that holds a plugin with
>  version 9.0.3.v20130506. That version goes at the build time in a couple
>  of places (including metabundle). 
Such exact dependencies are fundamentally broken and do not scale.  We 
cannot rebuild the whole world just for minor (say, security) updates. 
Lying about the version number (so that the new version looks like the 
old one to OSGi) doesn't strike me as a good idea, either, because it 
will confuse developers and other tools.

I tried to bring this up on the Project Jigsaw mailing list a couple of 
years ago, but I'm not sure if I brought across this point.  From my 
point of view, these Java module frameworks refuse to acknowledge that 
there is extensive experience with distro-level release engineering. 
(Basically, exact dependencies and multiple versions of the same code 
might be convenient now, but will seriously hurt you down the road.)

>  Exact match can't be used at all, because if jetty is updated,
> then it
>  will be impossible to install Eclipse. 
Well, if it doesn't work with the old version, that's the right thing to do.

I believe Debian relaxes the OSGi-generated dependencies on system 
libraries.  Fedora should do the same thing in its Eclipse packaging.

-- 
Florian Weimer / Red Hat Product Security Team

[Aleksandar Kurtakov]

----- Original Message -----
>  From: "Florian Weimer" <fweimer(a)redhat.com&gt;
>  To: devel(a)lists.fedoraproject.org
>  Sent: Friday, June 21, 2013 3:00:36 PM
>  Subject: Re: A need for build triggers & automatic rebuilds
>  
>  On 06/21/2013 08:28 AM, Krzysztof Daniel wrote:
>  
>  > OSGI records that there is a file
>  > org.eclipse.jetty.http_9.0.3.v20130506.jar that holds a plugin with
>  > version 9.0.3.v20130506. That version goes at the build time in a couple
>  > of places (including metabundle).
>  
>  Such exact dependencies are fundamentally broken and do not scale.  We
>  cannot rebuild the whole world just for minor (say, security) updates.
>  Lying about the version number (so that the new version looks like the
>  old one to OSGi) doesn't strike me as a good idea, either, because it
>  will confuse developers and other tools.
>  
>  I tried to bring this up on the Project Jigsaw mailing list a couple of
>  years ago, but I'm not sure if I brought across this point.  From my
>  point of view, these Java module frameworks refuse to acknowledge that
>  there is extensive experience with distro-level release engineering.
>  (Basically, exact dependencies and multiple versions of the same code
>  might be convenient now, but will seriously hurt you down the road.)
>  
>  > Exact match can't be used at all, because if jetty is updated, then it
>  > will be impossible to install Eclipse.
>  
>  Well, if it doesn't work with the old version, that's the right thing to do.
>  
>  I believe Debian relaxes the OSGi-generated dependencies on system
>  libraries.  Fedora should do the same thing in its Eclipse packaging. 
Just to clarify a bit - this is not a limitation of OSGi. It's the Eclipse
implementation being faulty of this. And when I say Eclipse implementation I mean Eclipse
platform because the actual OSGi runtime in Eclipse (called Equinox) works perfectly with
version ranges and doesn't suffer from this problem at all. It's the Eclipse IDE
that suffers from this due to legacy reasons.
This shows another problem though: expressing version range in a spec file e.g there is no
easy way to say I want a package that provides osgi(smth) [1.0.0, 1.5.0). If you try to
express it with RPM requires as two statements:
Requires: osgi(smth) >= 1.0.0
Requires: osgi(smth) < 1.5.0 
it's not exactly the same  as having packages providing osgi(0.9) and osgi(1.6.0) will
satisfy this requirements too. I know this is hypothetical but it shows the inability to
require version range in RPM terms reliably. And the way to get that but not fail-proof is
too verbose now.


Alexander Kurtakov
Red Hat Eclipse team

>  
>  --
>  Florian Weimer / Red Hat Product Security Team
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel

[Nicolas Mailhot]

Le Lun 24 juin 2013 11:59, Aleksandar Kurtakov a écrit :

>  This shows another problem though: expressing version range in a spec
> file
>  e.g there is no easy way to say I want a package that provides osgi(smth)
>  [1.0.0, 1.5.0). If you try to express it with RPM requires as two
>  statements:
>  Requires: osgi(smth) >= 1.0.0
>  Requires: osgi(smth) < 1.5.0
>  it's not exactly the same  as having packages providing osgi(0.9) and
>  osgi(1.6.0) will satisfy this requirements too. I know this is
>  hypothetical but it shows the inability to require version range in RPM
>  terms reliably. And the way to get that but not fail-proof is too verbose
>  now. 
You can express it reliably :
Requires: osgi(smth) >= 1.0.0
Conflicts: osgi(smth) >= 1.5.0

The drawback being you can not have two generations of osgi(smth)
on-system (but having two generations will confuse lots of apps anyway)

-- 
Nicolas Mailhot

[Florian Weimer]

On 06/24/2013 11:59 AM, Aleksandar Kurtakov wrote:

> > I believe Debian relaxes the OSGi-generated dependencies on
> system
> > libraries.  Fedora should do the same thing in its Eclipse packaging.
>
>  Just to clarify a bit - this is not a limitation of OSGi. It's the Eclipse
> implementation being faulty of this. And when I say Eclipse implementation I mean Eclipse
> platform because the actual OSGi runtime in Eclipse (called Equinox) works perfectly with
> version ranges and doesn't suffer from this problem at all. It's the Eclipse IDE
> that suffers from this due to legacy reasons. 
Yes, thanks for the clarification.  As I said, I believe that Debian 
patches the generated dependencies to be more lenient.  At least they 
did some time ago—this particular feature might have been lost along 
with Eclipse build system changes.

>  This shows another problem though: expressing version range in a spec
> file e.g there is no easy way to say I want a package that provides osgi(smth) [1.0.0,
> 1.5.0). If you try to express it with RPM requires as two statements:
>  Requires: osgi(smth) >= 1.0.0
>  Requires: osgi(smth) < 1.5.0
>  it's not exactly the same  as having packages providing osgi(0.9) and osgi(1.6.0)
> will satisfy this requirements too. I know this is hypothetical but it shows the inability
> to require version range in RPM terms reliably. And the way to get that but not fail-proof
> is too verbose now. 
Conflicts with future versions which do not exist yet are a bit 
questionable, IMHO.  In most cases, you don't know that 1.6.0 will 
actually be incompatible with your code because it hasn't been released yet.

-- 
Florian Weimer / Red Hat Product Security Team

[Michael Schwendt]

Krzysztof,

unrelated to this thread, but I've noticed the "Face" header in your
email, which exceeds the maximum size of such headers. As such, it breaks
the hard limits of some servers as well as clients, such as Claws Mail.

http://quimby.gnus.org/circus/face/


-- 
Fedora release 19 (Schrödinger’s Cat) - Linux 3.9.6-301.fc19.x86_64
loadavg: 0.05 0.11 0.11