Grepping through all Fedora specfiles?

[Tony Breeds]

On Wed, Jul 24, 2013 at 12:14:53PM +0200, Ralf Corsepius wrote:
> On 07/24/2013 02:33 AM, Tony Breeds wrote:
> >On Tue, Jul 23, 2013 at 10:01:51PM +0300, Ville Skyttä wrote:
> >>On 2013-07-23 08:04, Ralf Corsepius wrote:
> >>>
> >>>For similar purposes, I have been using the CSV table returned by
> >>>"https://admin.fedoraproject.org/pkgdb/lists/bugzilla?tg_format=plain"
> >>>
> >>>I am downloading the CSV-table and then use scripts to filter the CSV
> >>>into lists and subsequently to feed them into further scripts for
> >>>further processing.
> >>
> >>I've used that approach too sometime and it certainly works, but it'll
> >>take ages to clone every package from git (even though the cloning can
> >>be trivially parallelized with xargs -P or parallel).
> >
> >Sorry to be dense but if you know the package name and you only care
> >about the specfile can't you do something like:
> The problem is that you don't know the package names ;)

But you and Ville, both indictated that you're using the URL above and
post-processing it to get the names.

So can't you just combine the 2 things to do:

---
#!/bin/bash

full_clone=0
base_url='http://pkgs.fedoraproject.org'

set -ex

[ -e pkgdb_list ] || {
   wget \
    -c https://admin.fedoraproject.org/pkgdb/lists/bugzilla\?tg_format\=plain \
    -O pkgdb_list
}

pkgs=$( awk -F\| '/\}/ {print $2}' pkgdb_list )
for pkg in $pkgs ; do
	echo "Grabbing the specfile for $pkg"
	(
	cd ~/fedora-specs
	[ -e $pkg.spec  ] || {
		wget -o ~/tmp/$pkg.log -O $pkg.spec \
			"$base_url/cgit/${pkg}.git/plain/${pkg}.spec?id=HEAD"
	}
	)

	[ $full_clone == 0 ] && continue

	echo "Grabbing the git repo for $pkg"
	(
	cd ~/fedora-git
	[ -d $pkg  ] || {
		git clone git://pkgs.fedoraproject.org/git/${pkg}.git >&1
	) > ~/tmp/${pkg}-git.log &
done
---
and then wait a long time ;P

As an aside, Any guesses as to how big the ~/fedora-git dir would be
after running the script above?

Yours Tony
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20130725/2ab16200/attachment.sig>