Draft for talk about statscache in Flock 2015

[Ratnadeep Debnath]

Hi all,

I want to propose a talk on our ongoing project: statscache[1] in
Flock 2015. I am sharing a draft of the proposal for your review:

Title
====
Statscache, a near realtime intelligent caching layer empowering
faster data representation

Summary
=======
Fedmsg and fedmsg based tools: datanommer, datagrepper, etc. have
indeed taken communication and coordination across various components
in the Fedora ecosystem to an all new level. Currently, we are using
the raw datagrepper APIs to render various widgets and data
visualization. However, due to the massive amount of data stored in
datagrepper backend, searches tend to be slower. Also, consuming the
raw datagrepper API leads to increased data payload over the network
and writing complex client side logic to format the data.

Statscache is an attempt to solve the above problems and also to
create a generic framework to empower powerful data visualization. It
achieves this by listening to fedmsg events and storing the
denormalized and clean data in desired format for faster retrieval. It
also aims to become an intelligent caching layer based on "Single
source of truth" model and publishing REST and websocket API around it
to enable faster and realtime data visualizations. This could become
the backbone for empowering various widgets in Fedora hub.

Topology diagram:
https://raw.githubusercontent.com/fedora-infra/statscache/develop/docs/di...

[1]: https://github.com/fedora-infra/statscache

Regards,
rtnpro
-- 
Ratnadeep Debnath,
https://www.waartaa.com
GPG Fingerprint: 033C 8041 A0E9 CDBA 2E02  B785 2119 5486 F245 DFD6

[Ralph Bean]

On Tue, Apr 28, 2015 at 12:46:14AM +0530, Ratnadeep Debnath wrote:
>  Hi all,
>  
>  I want to propose a talk on our ongoing project: statscache[1] in
>  Flock 2015. I am sharing a draft of the proposal for your review:
>  
>  Title
>  ====
>  Statscache, a near realtime intelligent caching layer empowering
>  faster data representation
>  
>  Summary
>  =======
>  Fedmsg and fedmsg based tools: datanommer, datagrepper, etc. have
>  indeed taken communication and coordination across various components
>  in the Fedora ecosystem to an all new level. Currently, we are using
>  the raw datagrepper APIs to render various widgets and data
>  visualization. However, due to the massive amount of data stored in
>  datagrepper backend, searches tend to be slower. Also, consuming the
>  raw datagrepper API leads to increased data payload over the network
>  and writing complex client side logic to format the data.
>  
>  Statscache is an attempt to solve the above problems and also to
>  create a generic framework to empower powerful data visualization. It
>  achieves this by listening to fedmsg events and storing the
>  denormalized and clean data in desired format for faster retrieval. It
>  also aims to become an intelligent caching layer based on "Single
>  source of truth" model and publishing REST and websocket API around it
>  to enable faster and realtime data visualizations. This could become
>  the backbone for empowering various widgets in Fedora hub.
>  
>  Topology diagram:
>
> https://raw.githubusercontent.com/fedora-infra/statscache/develop/docs/di...
>  
>  [1]: https://github.com/fedora-infra/statscache 
+1 from me :)