Fedora Notifications System recent issues and Fixes

[Kevin Fenzi]

Greetings. 

The Fedora Notifications System, available at: 
https://apps.fedoraproject.org/notifications/
Is a way for interested parties to be notified when events occur in
Fedora. This system currently allows for IRC and/or Email notifications
of events, as well as "digesting" a specified number of messages or any
messages in a specified time. 

Recently the digest processing had a number of issues: 

* Some users had selected very large time frames or number of messages
  and the resulting digest messages were too large to be accepted by
  our mail server. 

* Over sized digests also resulted in no processing of additional
  digests in the queue. 

* Digest processing was interrelated with the non digest processing,
  making it difficult to debug or test changes.

* When the digest queue was very large, the IRC handler couldn't get
  enough processing time to join the network. 

* When the queue was very large testing fixes required processing all
  the queue and only then producing digests. 

This resulted in no digests being sent at all.

We have taken the following steps: 

* Digests that are 'too large' are split into chunks that are
  acceptable to our mail server. If they are still too large for some
  users email providers, they will be dropped after retries. 

* Digest processing is now separated from non digest processing
  allowing for debugging and fixes that don't affect the real time
  notifications, as well as not blocking IRC workers from joining the
  network.

Unfortunately, in landing our fixes a bug was introduced that split
digests too soon, resulting in incomplete digests after a few
characters. This has been corrected and any new digest notifications
should be complete. 

We are sorry for any inconvenience these issues may have caused. 

Please report any issues you find with the notification system to 
https://pagure.io/fedora-infrastructure (our config or deployment)
or
https://github.com/fedora-infra/fmn (upstream bugs/issues). 

Thanks, 

kevin