On Mon, 6 Apr 2020 at 08:58, Timothée Floure <timothee.floure(a)posteo.net>
> How does the indexing works ?
You point Yacy to a domain or list of URLs
in this case), and it takes care of
everything. There is also an advanced crawler panel in the UI allowing
you to filter content (e.g. HTML classes) from pages, which would be
useful if we do not want to index everything (e.g. dependencies).
I am not familiar with the maths used by Yacy for indexing.
Me neither :D
> And what would it take to add more info for each package ?
I wrote a quick script (https://paste.gnugen.ch/raw/4JAC
package metadata from PDC+mdapi for testing, but it is ways too slow to
scale to the whole package set.
Cool, yeah the current indexing takes hours (I think around 4-5 hours)
there are more than 80 000s packages and sub-packages. I think we can run
this once a day so speed is not super super critical I would say.
MDAPI will have to be replaced by local SQLite to increase performance.
I think we could generate most of the content from the repositories'
metadata (last N Fedora + EPEL) but I need to find where the SQL files
lives. A privileged endpoint to dist-get to fetch the package ->
maintainer mapping bypassing pagination would be convenient.
You can look at how mdapi grabs these sqllite files (
). For the
maintainer mapping you should be able to find that here -->
We can use Yacy's JSON API to build a sexy fedora-branded search
but I think it's a late-stage optimization.
+1, would be interesting to see with msuchy if that can be easily
integrated with the work he was doing.
infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
To unsubscribe send an email to
Fedora Code of Conduct:
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines