Errors in constructing SPEC from local source.

[Allen S. Rout]

In R2spec-3.0.3-1, if you set a local source (-s <source>) then the URL
and source0 fields in the spec will remain empty.  Such specs fail to
build.

There are a variety of ways to address this, I'm not sure that what I've
done is really a superior solution.  


in RPackage.py, down the branch of "I've only got a source, not a
source0", I added

                self.source0 = source
                self.URL = source

(see below).

This means that URL is populated with a local file path, but that's
really all that's available. 



....

All I'm really trying to do right now is get ggplot to one of my
customers.  I wasn't gearing up for contributing much code, but I can,
if that's what Pierre-Yves wants. 

I have to say, this is being a distinctly frustrating process.

....

"below"... 


    def setSource(self, source0, source):
        ''' From the url retrieve the name of the source '''
        if source != None and source0 != None:
            if source != source0.rsplit('/',1)[1] :
                sys.exit("Do not give 2 contradictory sources of information")
        elif source != None:
            try:
                self.source = source.rsplit('/',1)[1]
                self.path = source.rsplit('/',1)[0]
                self.source0 = source
                self.URL = source
            except IndexError, err:
                self.source = source
                self.path = '../'
        elif source0 != None:
            self.source = source0.rsplit('/',1)[1]
            self.path = '../'
            self.source0 = source0
        else:
            sys.exit("The source are not correct ")



- Allen S. Rout

[Allen S. Rout]

asr-hnhdhkBXzx8(a)public.gmane.org (Allen S. Rout) writes:

>  In R2spec-3.0.3-1, if you set a local source (-s <source>) then
> the URL
>  and source0 fields in the spec will remain empty.  Such specs fail to
>  build.
>
>  There are a variety of ways to address this, I'm not sure that what I've
>  done is really a superior solution.  
>
>
>  in RPackage.py, down the branch of "I've only got a source, not a
>  source0", I added
>
>                  self.source0 = source
>                  self.URL = source
>
>  (see below).
>
>  This means that URL is populated with a local file path, but that's
>  really all that's available.  

So, I see from your previous replies, Pierre-Yves, that this is more or
less expected behavior.  Understood.

I have the opinion that the current manner of failing doesn't
communicate what's going on very well.  Would you consider documenting
this requirement in the usage message, or even failing with an
informative message?

I also think that there are other sources of packages than the various
repos.  It seems incorrect to me to insist that we name one.



- Allen S. Rout

[Pierre-Yves Chibon]

On Tue, 2011-04-12 at 16:57 -0400, Allen S. Rout wrote:
>  
>  I have the opinion that the current manner of failing doesn't
>  communicate what's going on very well.  Would you consider documenting
>  this requirement in the usage message, or even failing with an
>  informative message? 
As in print the error message in the console rather than in the build
file ? Or do you have something else in mind ?

>  I also think that there are other sources of packages than the
> various
>  repos.  It seems incorrect to me to insist that we name one. 
For this I had few ideas but I do not know how far I went with them:
- One idea was to be able to add repositories on the ~/.R2spec.
Now that I am thinking about it, I guess something like:
[repository]
bioc = http://bioconductor.org
cran = http://cran.r-project.org
mine = http://pingoured.fr
where one could then call R2spec/R2rpm with --repo=mine or --repo=bioc
or something like this.

- Another idea might be to just specify the repository via an argument
--URL="http://pingoured.fr"

R2spec was designed to easily create spec file which would then be
reviewed and corrected by human. It has evolved to include rpm building
but I have the feeling that it might need a good redesign, maybe a
version 3.0.0 ;-)

Best regards,
Pierre

[Allen S. Rout]

Pierre-Yves Chibon <pingou-E11Oz7VxvVOXCRStZZN3OA(a)public.gmane.org&gt;
writes:
>  On Tue, 2011-04-12 at 16:57 -0400, Allen S. Rout wrote:
> > 
> > I have the opinion that the current manner of failing doesn't
> > communicate what's going on very well.  Would you consider documenting
> > this requirement in the usage message, or even failing with an
> > informative message?
>
>  As in print the error message in the console rather than in the build
>  file ? Or do you have something else in mind ? 
Well, the error message is something along the lines of "Blank URL
field".  This doesn't lead me to assume "I should specify a repo, for
this file that I have locally". 

On the one hand, for the -source case, you could specify a file:/// URL
 (and source0) if the user doesn't give one.

If you don't like that, you could detect "Oh, they set a source but no
repo", and give a diagnostic like "Must set a repo; please use one of
--[... yadda yadda " 





> > I also think that there are other sources of packages than the
> various
> > repos.  It seems incorrect to me to insist that we name one.
>
>  For this I had few ideas but I do not know how far I went with them:
>  - One idea was to be able to add repositories on the ~/.R2spec.
>  Now that I am thinking about it, I guess something like:
>  [repository]
>  bioc = http://bioconductor.org
>  cran = http://cran.r-project.org
>  mine = http://pingoured.fr
>  where one could then call R2spec/R2rpm with --repo=mine or --repo=bioc
>  or something like this.
>
>  - Another idea might be to just specify the repository via an argument
>  --URL="http://pingoured.fr" 

I understand your repo-centric perspective, but I would encourage you to
let people build from files without forcing them to invent a repo.  But
this is an aesthetic call; If you permit an escape valve like --URL, and
an error message to point to its use, then the usability concerns are
well answered.



>  R2spec was designed to easily create spec file which would then be
>  reviewed and corrected by human. It has evolved to include rpm building
>  but I have the feeling that it might need a good redesign, maybe a
>  version 3.0.0 ;-) 
The version I got from EPEL was 3.0.3.. ? 


I would be delighted to help if you are interested in it.  My previous
wave of changes did not meet with your approval, and I inferred that
this meant you look at the problem differently than I do.  I am not
offended by this, it's your project.  But I did stop trying to
contribute.

I definitely look at the problem from the perspective of "Automatic
production of entire dependency chains".  If you think that is a
worthwhile direction to go, I would be happy to help get there.



- Allen S. Rout

[Pierre-Yves Chibon]

On Tue, 2011-04-12 at 17:34 -0400, Allen S. Rout wrote:
>  
>  
>  > R2spec was designed to easily create spec file which would then be
>  > reviewed and corrected by human. It has evolved to include rpm
>  building
>  > but I have the feeling that it might need a good redesign, maybe a
>  > version 3.0.0 ;-)
>  
>  The version I got from EPEL was 3.0.3.. ?  
Sorry I meant 4.0.0

>  I would be delighted to help if you are interested in it.  My
> previous
>  wave of changes did not meet with your approval, and I inferred that
>  this meant you look at the problem differently than I do.  I am not
>  offended by this, it's your project.  But I did stop trying to
>  contribute.
>  
>  I definitely look at the problem from the perspective of "Automatic
>  production of entire dependency chains".  If you think that is a
>  worthwhile direction to go, I would be happy to help get there. 
R2spec remains a project designed to make the life of Fedora's R
packager's easier, so it has to remain guideline compliant. That is a
reason why putting the source0 as URL also is not *in my opinion* a good
approach (while I actually do like the --URL option).

There was a time when I thought I would try to get an "automatic cran
builder" working but atm I have other priorities.

Pierre

[Allen S. Rout]

Pierre-Yves Chibon <pingou-E11Oz7VxvVOXCRStZZN3OA(a)public.gmane.org&gt;
writes:

>  R2spec remains a project designed to make the life of Fedora's R
>  packager's easier, so it has to remain guideline compliant. That is a
>  reason why putting the source0 as URL also is not *in my opinion* a good
>  approach (while I actually do like the --URL option). 
Well, like I said, from my perspective it's aesthetic, so I'd be
delighted with it.


>  There was a time when I thought I would try to get an "automatic
> cran
>  builder" working but atm I have other priorities. 
Are you willing to record your preferences on this, especially with
respect to how my previous work didn't fit your picture?  If you did,
then if I get around to it I could work in ways aligned with your plans.



- Allen S. Rout

[Pierre-Yves Chibon]

On Wed, 2011-04-13 at 15:06 -0400, Allen S. Rout wrote:
>  >
>  > R2spec remains a project designed to make the life of Fedora's R
>  > packager's easier, so it has to remain guideline compliant. That is 
>  > a reason why putting the source0 as URL also is not *in my opinion* 
>  > a good approach (while I actually do like the --URL option).
>  
>  Well, like I said, from my perspective it's aesthetic, so I'd be
>  delighted with it. 
I'll try to push this on the git by the end of the week-end then :)

>  > There was a time when I thought I would try to get an
> "automatic 
>  > cran  builder" working but atm I have other priorities.
>  
>  Are you willing to record your preferences on this, especially with
>  respect to how my previous work didn't fit your picture?  If you did,
>  then if I get around to it I could work in ways aligned with your
>  plans. 
Well at the moment I see more R2rpm/R2spec has a tool to generate rpm or
spec individually as one would have to do to incorporate them into
Fedora. It is a spec generator tool which push the work up to building
the rpm (which it needs to fill the %file section anyway).
Specs needs manual inspection and curation sometime, one needs to run
rpmlint and so on.

My thoughts were that there was space for a tool to build the whole
dependency tree and run R2spec/R2rpm in the right order.
In my attempts to build CRAN I used your R script for that. I could this
way create 7 files and then I was just running these 7 files on after
the other.
My problem was more or less:
- Takes time to build (~1 week for the whole run)

- Lack of resources (although I improved on these but not enough)

- I did it the dirty way by doing : build all R packages in fileX in
mock - Install all rpm - build all R packages in fileX+1 in mock -
install all rpm.... - restart for another arch/release
I actually wonder, if one wants to give a shot at such project in a long
term, if looking into koji would not be a nice alternative: build all
the R packages from fileX for all arch/release - populate the repo as
they are build - build all the R packages from fileX+1 for all
arch/release... And this way at the end you have a repo which you know
works.
I always had the wish to support at least i686 and x86_64 on Fedora n
and n-1 RHEL 5 and 6, so 6 to 8 (if I added rawhide) build for each
packages. For this I though koji would make sense, but I never got
around to try to install it and configure vm builders.

- Not all the packages build, sometime because of the lack of outside
dependencies.
For this reason all the packages that fails have to be inspected by
hand.

- I have a script to update existing spec file but it does not check for
changes in the dependency tree and cran as well as bioconductor are so
good at changing the dependency tree at every releases...
So for every release of bioconductor, we have to update all the spec
files for most of their elements but the changelog basically (sounds
easy when I say it this way ^^).

Basically I would keep R2spec at a single package level and build on the
top a cran2rpm or so.


All this makes me think that I still have few thousands spec files
hanging somewhere and that I should actually make them accessible
somewhere (with a warning that a significant part of it is probably
outdated).


These were my thoughts when I was looking into it.

Hope this help,

Pierre

[José Abílio Matos]

On Wednesday 13 April 2011 22:10:21 Pierre-Yves Chibon wrote:
>  All this makes me think that I still have few thousands spec files
>  hanging somewhere and that I should actually make them accessible
>  somewhere (with a warning that a significant part of it is probably
>  outdated).
>  
>  
>  These were my thoughts when I was looking into it.
>  
>  Hope this help,
>  
>  Pierre 
It would be nice to have a plan for this.

A boostrap stage where the dependencies are reduced just to build 
dependencies.

The identification of core packages required by other packages.

Debian has done it before, Dirk can provide feedback for specific issues we 
get.

-- 
José Abílio

[Pierre-Yves Chibon]

On Wed, 2011-04-13 at 23:10 +0200, Pierre-Yves Chibon wrote:
>  On Wed, 2011-04-13 at 15:06 -0400, Allen S. Rout wrote:
>  > > R2spec remains a project designed to make the life of Fedora's R
>  > > packager's easier, so it has to remain guideline compliant. That
>  is 
>  > > a reason why putting the source0 as URL also is not *in my
>  opinion* 
>  > > a good approach (while I actually do like the --URL option).
>  > 
>  > Well, like I said, from my perspective it's aesthetic, so I'd be
>  > delighted with it.
>  
>  I'll try to push this on the git by the end of the week-end then :) 
A bit late, but I just pushed the changes. Testers are welcome :)

Pierre

[Allen S. Rout]

José Matos <jamatos-BxEiAy0842I(a)public.gmane.org&gt; writes:

>  It would be nice to have a plan for this.
>
>  A boostrap stage where the dependencies are reduced just to build 
>  dependencies. 
So here 

http://article.gmane.org/gmane.linux.redhat.fedora.r.devel/177

is the farthest I've gotten in formalizing that path.  I've waved that
around a few times; it doesn't seem to pick up a lot of traction.  If
the topic were simple, I'd tl;dr it, but I don't think it can really be
done.


>  The identification of core packages required by other packages.
I think that's what I'd call the "Narrow dependency graph"; that is,
"packages which are needed to install other packages", as opposed to
"Packages which are needed to CMD CHECK other packages".


So; if we take R2rpm as a building block, then I suggest the following
pseudocode, to build a repo:


build_a_repo()

Make a list from CRAN, consult some local metadata about busted
packages.

Pick an unbuilt package at random, and build it, until done. 

-----

build_a_package()

 generate the narrow and wide dependency graphs for your package. 
 
 generate the narrow graph of your wide dependencies.  
 (everything you need to install to run (everything you need to check (package p ))) 
 I'll call that G'.

 Remove from G' everything already installed. 

 Build everything in G' , disabling CHECK and ignoring Suggests,
 Enhances.  [ Can we also avoid the build-it-once, check %files, build
 it again step at this stage? ]

 Install everything in G'. 

 Build everything in G', enabling CHECK.   

 Packages that build and CHECK may be signed, and removed from the list
 of packages to be built.

-----


I split it up this way because it permits also the "Build this package
and its depgraph", which is where I came in. :)

This process also suggests some additional caching, at least the package
metadata from all the repos; It's going to be necessary to trace deps
across repos, of course.  If anyone has opinions about where that ought
to go, fire away.  I figure it'd go in /var/tmp/[something] , and not
expire for a week or more.  Since the build process is going to be
multi-day, anthing shorter than that will just be insane.



- Allen S. Rout