Koji "hidden" packages proposal

[Mike Bonnet]

I've written up a brief proposal about how "hidden" packages may be
supported in Koji.  The objective of this is to enable building EPEL
packages in Koji.  I wrote this up fairly quickly, and I'm sure I
haven't thought through all the issues, but I wanted to get the ball
rolling.  Let me know if you have any questions/comments/ideas/issues
relating to this proposal.

http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html

Thanks,
Mike

[Dennis Gilmore]

On Friday 14 March 2008, Mike Bonnet wrote:
>  I've written up a brief proposal about how "hidden"
> packages may be
>  supported in Koji.  The objective of this is to enable building EPEL
>  packages in Koji.  I wrote this up fairly quickly, and I'm sure I
>  haven't thought through all the issues, but I wanted to get the ball
>  rolling.  Let me know if you have any questions/comments/ideas/issues
>  relating to this proposal.
>
>  http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html
>
>  Thanks,
>  Mike 
the tree will have to be nothing like /mnt/koji/packages 

instead it will have to be like  
http://download.fedora.redhat.com/pub/fedora/linux/  so that you can use 
rsync  and repo mirroring tools to sync the content and keep trees in sync.   
ill leave it up to Seth to explain more but hit micro repository option he is 
working on would allow us to pull the existing repodata intothe new repodata.

We really only need to have a command that can suck the repodata into the 
database.  so we know about the packages.

Dennis

[Stephen John Smoogen]

On Fri, Mar 14, 2008 at 6:55 PM, Dennis Gilmore <dennis(a)ausil.us&gt; wrote:
>  On Friday 14 March 2008, Mike Bonnet wrote:
>   > I've written up a brief proposal about how "hidden" packages may be
>   > supported in Koji.  The objective of this is to enable building EPEL
>   > packages in Koji.  I wrote this up fairly quickly, and I'm sure I
>   > haven't thought through all the issues, but I wanted to get the ball
>   > rolling.  Let me know if you have any questions/comments/ideas/issues
>   > relating to this proposal.
>   >
>   > http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html
>   >
>   > Thanks,
>   > Mike
>   the tree will have to be nothing like /mnt/koji/packages
>
>   instead it will have to be like
>   http://download.fedora.redhat.com/pub/fedora/linux/  so that you can use
>   rsync  and repo mirroring tools to sync the content and keep trees in sync.
>   ill leave it up to Seth to explain more but hit micro repository option he is
>   working on would allow us to pull the existing repodata intothe new repodata.
>
>   We really only need to have a command that can suck the repodata into the
>   database.  so we know about the packages.
>  
Another big issue will be that EL is split in so many 'interesting'
ways. You need to be able to either have the build system know of
every seperate channels from RHN for each variation EL-3,4,5 or have
RHN somehow give a conglomerate channel to you of all the packages in
one big bucket. The code from DAG's mrepo sort of does this but would
need some work on getting it to play nicely


-- 
Stephen J Smoogen. -- CSIRT/Linux System Administrator
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"

[Mike Bonnet]

On Fri, 2008-03-14 at 19:17 -0600, Stephen John Smoogen wrote:
>  On Fri, Mar 14, 2008 at 6:55 PM, Dennis Gilmore
> <dennis(a)ausil.us&gt; wrote:
>  > On Friday 14 March 2008, Mike Bonnet wrote:
>  >  > I've written up a brief proposal about how "hidden" packages may
> be
>  >  > supported in Koji.  The objective of this is to enable building EPEL
>  >  > packages in Koji.  I wrote this up fairly quickly, and I'm sure I
>  >  > haven't thought through all the issues, but I wanted to get the ball
>  >  > rolling.  Let me know if you have any questions/comments/ideas/issues
>  >  > relating to this proposal.
>  >  >
>  >  > http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html
>  >  >
>  >  > Thanks,
>  >  > Mike
>  >  the tree will have to be nothing like /mnt/koji/packages
>  >
>  >  instead it will have to be like
>  >  http://download.fedora.redhat.com/pub/fedora/linux/  so that you can use
>  >  rsync  and repo mirroring tools to sync the content and keep trees in sync.
>  >  ill leave it up to Seth to explain more but hit micro repository option he is
>  >  working on would allow us to pull the existing repodata intothe new repodata.
>  >
>  >  We really only need to have a command that can suck the repodata into the
>  >  database.  so we know about the packages.
>  >
>  
>  Another big issue will be that EL is split in so many 'interesting'
>  ways. You need to be able to either have the build system know of
>  every seperate channels from RHN for each variation EL-3,4,5 or have
>  RHN somehow give a conglomerate channel to you of all the packages in
>  one big bucket. The code from DAG's mrepo sort of does this but would
>  need some work on getting it to play nicely 
We're not going to be pulling packages straight from RHN, we'll be
manually importing the packages.  The proposal allows for multiple
alternate directory trees, so it will be possible to have a tree to
which we import the RHEL-4 binaries, a tree for RHEL-5 binaries, etc.
When a new RHEL update is released, it'll be up to an admin to download
the new packages and import them into the proper tree.

[Mike Bonnet]

On Fri, 2008-03-14 at 19:55 -0500, Dennis Gilmore wrote:
>  On Friday 14 March 2008, Mike Bonnet wrote:
>  > I've written up a brief proposal about how "hidden" packages may be
>  > supported in Koji.  The objective of this is to enable building EPEL
>  > packages in Koji.  I wrote this up fairly quickly, and I'm sure I
>  > haven't thought through all the issues, but I wanted to get the ball
>  > rolling.  Let me know if you have any questions/comments/ideas/issues
>  > relating to this proposal.
>  >
>  > http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html
>  >
>  > Thanks,
>  > Mike
>  the tree will have to be nothing like /mnt/koji/packages 
>  
>  instead it will have to be like  
>  http://download.fedora.redhat.com/pub/fedora/linux/  so that you can use 
>  rsync  and repo mirroring tools to sync the content and keep trees in sync.   
>  ill leave it up to Seth to explain more but hit micro repository option he is 
>  working on would allow us to pull the existing repodata intothe new repodata. 
I'm not really sure what you're talking about.  *No one* will be
mirroring the hidden packages in the case we're talking about (building
EPEL in Koji), these will be RHEL binaries, and only available to the
builders.

>  We really only need to have a command that can suck the repodata into
> the 
>  database.  so we know about the packages. 
We have a command that can suck data about packages in to the database.
It's "koji import".  The proposal is designed to work within the
framework of that we already have in Koji, to try to minimize the number
of changes and thus the time it'll take to implement.

The micro repository option Seth has talked about sounds interesting,
but it doesn't really help with the issue of making packages available
to Koji builders without making them available to the world.

[Dennis Gilmore]

On Saturday 15 March 2008, Mike Bonnet wrote:
>  On Fri, 2008-03-14 at 19:55 -0500, Dennis Gilmore wrote:
>  > On Friday 14 March 2008, Mike Bonnet wrote:
>  > > I've written up a brief proposal about how "hidden" packages may
> be
>  > > supported in Koji.  The objective of this is to enable building EPEL
>  > > packages in Koji.  I wrote this up fairly quickly, and I'm sure I
>  > > haven't thought through all the issues, but I wanted to get the ball
>  > > rolling.  Let me know if you have any questions/comments/ideas/issues
>  > > relating to this proposal.
>  > >
>  > > http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html
>  > >
>  > > Thanks,
>  > > Mike
>  >
>  > the tree will have to be nothing like /mnt/koji/packages
>  >
>  > instead it will have to be like
>  > http://download.fedora.redhat.com/pub/fedora/linux/  so that you can use
>  > rsync  and repo mirroring tools to sync the content and keep trees in
>  > sync. ill leave it up to Seth to explain more but hit micro repository
>  > option he is working on would allow us to pull the existing repodata
>  > intothe new repodata.
>
>  I'm not really sure what you're talking about.  *No one* will be
>  mirroring the hidden packages in the case we're talking about (building
>  EPEL in Koji), these will be RHEL binaries, and only available to the
>  builders. 
What im talking about was the proposal put forward  when we had the
buildsys 
meeting  the proposal that  we all said sounded like the way to move forward.  
This is not that proposal.   importing packages is the one thing that hurts 
koji from getting wider use outside of fedora.  Some people will import the 
data.  Most do not want to.  Fedora will be mirroring RHEL content from RHN,   
people outside of fedora will be mirroring fedora and building on top of 
that.   

>  > We really only need to have a command that can suck the repodata
> into the
>  > database.  so we know about the packages.
>
>  We have a command that can suck data about packages in to the database.
>  It's "koji import".  The proposal is designed to work within the
>  framework of that we already have in Koji, to try to minimize the number
>  of changes and thus the time it'll take to implement. 

sure koji import --repodata  --url=<http|ftp|file>://<host>/path/to/repodata

>  The micro repository option Seth has talked about sounds
> interesting,
>  but it doesn't really help with the issue of making packages available
>  to Koji builders without making them available to the world. 
sure it does,  we have
an internal only tree  thats on the phx internal 
network and not available outside of phx,   just as we do now. 

Dennis

[Seth Vidal]

On Sat, 2008-03-15 at 21:58 -0500, Dennis Gilmore wrote:
>  What im talking about was the proposal put forward  when we had the
> buildsys 
>  meeting  the proposal that  we all said sounded like the way to move forward.  
>  This is not that proposal.   importing packages is the one thing that hurts 
>  koji from getting wider use outside of fedora.  Some people will import the 
>  data.  Most do not want to.  Fedora will be mirroring RHEL content from RHN,   
>  people outside of fedora will be mirroring fedora and building on top of 
>  that.   
>   
I'm not sure I see the problem here, necessarily.

okay here's how I see what Mike's proposal does and then what we
eventually want to do:

Mike's Proposal:
 - gives us a way of building from the reposync'd rhel-trees that we now
have on puppet1 w/o distributing these pkgs out to the world. Yay
 - gives us a way of building epel in koji so we can finally disable
plague, also yay.


The eventual future:
 -  have a way of koji to build from an arbitrary set of remote repos
by:
     1. syncing down the repodata from those remote repos and 'faking'
an import 
     2. creating its own local repo out of the remote repos (or
micro-repos) for external use.

There's nothing mutually exclusive about these two items and, afaict,
there's no requirement that they be done at the same time, either.


I guess in short:
 - Dennis: I think your suggestion is one we should do but we do not
require it immediately.

 - Mike: The proposal sounds round enough to me. We've already got all
of the production repos(no betas) in rhn for rhel4 and 5 for i386,
x86_64 and ppc* synced over on puppet1. They resync every night. Is
there anything else you need that I missed?

-sv

[Mike McLean]

Dennis Gilmore wrote:
>  What im talking about was the proposal put forward  when we had the
> buildsys 
>  meeting  the proposal that  we all said sounded like the way to move forward.  
>  This is not that proposal.   importing packages is the one thing that hurts 
>  koji from getting wider use outside of fedora.  Some people will import the 
>  data.  Most do not want to.  Fedora will be mirroring RHEL content from RHN,   
>  people outside of fedora will be mirroring fedora and building on top of 
>  that.    
All that was really concluded at the meeting was that Mike and/or I 
would create a proposal. As I said in that meeting there are basically 
two approaches: import and hide or find a sane way to use external 
repos. Mike's proposal is the former.

[Stephen John Smoogen]

On Fri, Mar 14, 2008 at 5:33 PM, Mike Bonnet <mikeb(a)redhat.com&gt; wrote:
>  I've written up a brief proposal about how "hidden"
> packages may be
>   supported in Koji.  The objective of this is to enable building EPEL
>   packages in Koji.  I wrote this up fairly quickly, and I'm sure I
>   haven't thought through all the issues, but I wanted to get the ball
>   rolling.  Let me know if you have any questions/comments/ideas/issues
>   relating to this proposal.
>
>   http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html
>
>   Thanks,
>   Mike 
What can I do to help on this?


-- 
Stephen J Smoogen. -- CSIRT/Linux System Administrator
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"