On Mon, Jan 18, 2021 at 05:41:35PM -0800, Kevin Fenzi wrote:
On Mon, Jan 18, 2021 at 04:25:09PM +0100, Pierre-Yves Chibon wrote:
> Good Morning Everyone,
> Currently our ideas are:
> - for datanommer:
> - port it to fedora-messaging
> - adjust it to whichever solution we chose to replace datagrepper
> - for datagrepper:
> - keep it as is
> - Replace by
> - postgres https://postgrest.org/
> - prest https://github.com/prest/prest
> - kinto https://docs.kinto-storage.org/en/stable/
> - Swagger/OpenAPI https://swagger.io/
Doing any of those means existing queries no longer work right?
Thats kind of a pain. ;(
Yes, that will need to be taken into account when a decision is made on which of
these solutions to pick.
> - Add support for Graphql
> - for the postgresql server
> - Split messages per year in different table
> - Unite them using a postgresql view
I've long wanted to do this. ;)
It might be worth making it non default to query more than the most
recent year. Most queries won't need anything that old...
> - Kick out the old messages per year
> - Keep the current year + n-1 in the current DB
> - Kick the other to another DB?
> - Kick the other to a tarball somewhere?
I would prefer to keep it queryable... there's some things that may want
to query the entire backhistory.
Again, the idea was to list every ideas here and then we can weight them again
> - Output the database daily dump to file / year
> - TimescaleDB a postgresql plugin for time-series data
> - https://dev.t-matix.com/blog/postgresql-as-a-time-series-database/
> - https://docs.timescale.com/latest/introduction
> - Make the msg field in the message table be a JSON field
If you do this would you convert all the old messages?
> Would you have any other ideas of things we could look at?
I'm not sure if there's any compression we could use here, but it would
be nice if the data took up less room. :)
Worth looking into as well.
Thanks for looking into this!
Hopefully it'll be productive and successful!