[fedora-commops] Issue #105 `GSoC Idea - Centralized metrics for Fedora`

[Sachin S Kamath]

skamath reported a new issue against the project: `fedora-commops` that you are
following:
``
## Problem

Right now, metrics collection in CommOps is not very efficient and requires a lot of
manual work. Metrics for various events/FAS groups/users are collected using scripts which
query [datagrepper](https://apps.fedoraproject.org/datagrepper/) and return results. This
process is very time consuming and writing scripts each time is a very tedious process.
Also, querying the datagrepper to get data everytime is redundant and time-consuming.

Example of current statistics generation : https://github.com/bee2502/fedora-stats-tools
(Lots of hack-y scripts)


## Proposed Solution

Hack on  [statscache](https://github.com/fedora-infra/statscache) to build a central
metrics generation system for Fedora with handy features to pull statistics. Statscache
consumes all the messages and does not query datagrepper every single time thereby
increasing the efficiency. By building on top of statscache, we can significantly reduce
the number of scripts required to gather metrics to almost 0. 

Nice to have features : 

* Statistics by FAS group.
* Per user statistics.
* Statistics of users holding a badge (Useful for event statistics)
* Exporting of stats in various formats (JSON, HTML, CSV, etc)
* Date based filtering for all statistics (Useful for generating reports)
* Graphs for statistics

I added the above-mentioned features based on datagrepper in the
[tool](https://pagure.io/gsoc-stats) I had developed last summer. This can be used as a
base for statscache integration.

Sample data generated using the tool :
[fedstats-data](https://github.com/sachinkamath/fedstats-data)

## Final Deliverables

* A webapp to run queries (with a nice, minimal interface - based on statscache)
* API for all the queries functions
* Well-Documented code.


This is my initial proposal and we can definitely build upon this. Thoughts?
CC: @bee2502 @bex


``

To reply, visit the link below or just reply to this email
https://pagure.io/fedora-commops/issue/105