Code Search for Fedora

[Michael Stapelberg]

On Tue, Nov 18, 2014 at 2:12 PM, Kevin Fenzi <kevin at scrye.com> wrote:
> On Tue, 18 Nov 2014 13:24:02 -0800
> Michael Stapelberg <michael+fedora at stapelberg.ch> wrote:
>
>> Thanks for your quick reply!
>
> No problem. ;)
>
> ...snip...
>
>> I’ve had very quick glance only so far, but the general idea sounds
>> reasonable. I’m not sure who’d want to work with me on the project,
>> but perhaps we can find someone who’s interested.
>
> Yeah, we just want to make sure at least a few people know how to debug
> and fix things. Then it's not all on one persons shoulders.
>
> ...snip...
>
>> That’s a bummer. How many IOPS do your SAS disks provide? Is there any
>> chance that you could get some SSDs in the near to mid term future?
>
> I'd have to test them. ;)
You could install fio(1) and run it with this config file:

[global]
bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10
rw=randread
direct=1
blocksize=4k
size=500m
ioengine=libaio
iodepth=1
#write_bw_log
#write_lat_log
numjobs=1

[sda]
filename=/dev/xvdb

Then, compare the results to http://p.nnev.de/4763 — if they are
roughly comparable, that might work.

>
> It's possible, but not sure how possible. We are putting together
> budgets for next year now, but not sure if we would be able to get
> SSD's for this.
>
> I'm not sure also how big things are. ie, all the unpacked package
> sources. We could find out, but will take some investigation.
Agreed.

>
>> > I think before we go looking into hardware requirements, we should
>> > discuss the software? Whats it written in? Is there a bunch of
>> > people who work on it? or just you?
>> It’s written in Go, and mostly I’m working on it, with a few random
>> contributions from other people from time to time.
>
> We are very heavily a python shop here, I don't know if any of our
> applications folks have really done much with go, but I'm sure they
> will chime in if they have. ;)
I see. Go is pretty easy to pick up, and the vast majority of people
in my filter bubble who gave it a try found it pleasant to work with.

>
>> > We would want it packaged up as rpms for deployment, preferably for
>> > epel7 (to work on rhel7 hosts).
>> Yeah, I’ve heard about that, and it shouldn’t be a problem, I think. I
>> assume the Go compiler is in EPEL7.
>
> Yeah, I think so.
>
>> > Would you be open to changes in code/architecture to meet our setup
>> > better?
>> Of course, yeah.
>
> There would also be questions around processing the source... in debian
> do you unpack all the source debs? or ?
In Debian we operate on all source packages of the main distribution’s
unstable (“sid”) version.

>
> Do you follow just some branches of packages? Or all of them?
We only use sid, mostly because it was simple to implement Code Search
with just a single corpus. I’ve been asked once whether I could also
provide other suites, but that’s a project for the future, if at all
necessary. The rationale so far is that we’re mostly interested in
code that’s in the development version, because code that is _only_ in
the older versions is likely obsolete in some way.

> Or just some arches?
Since we’re talking about source packages, they are not
architecture-specific at all :).

>
> Is this just upstream source? or the source + any local patches?
We’re unpacking the upstream source and then applying the Debian diff
so that we have the package indexed the same way our build daemons see
them. So, yes, this includes applying local patches.

>
> Lots of questions... hope I'm not asking too many dumb ones. ;)
Perfectly reasonable questions so far :). I hope my reply makes things
clearer, but don’t hesitate to ask if you have further questions.