From d524e7cae19eec440a4ad14467ec31a219e11a85 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Miroslav=20Such=C3=BD?= <msuchy@redhat.com>
Date: Wed, 17 May 2017 11:09:16 +0200
Subject: [PATCH] added retrace SOP

---
 docs/sysadmin-guide/sops/retrace.rst | 127 +++++++++++++++++++++++++++++++++++
 1 file changed, 127 insertions(+)
 create mode 100644 docs/sysadmin-guide/sops/retrace.rst

diff --git a/docs/sysadmin-guide/sops/retrace.rst b/docs/sysadmin-guide/sops/retrace.rst
new file mode 100644
index 0000000..bef3dc9
--- /dev/null
+++ b/docs/sysadmin-guide/sops/retrace.rst
@@ -0,0 +1,127 @@
+.. title: Retrace SOP
+.. slug: infra-retrace
+.. date: 2017-15-17
+.. taxonomy: Contributors/Infrastructure
+
+==========
+retrace SOP
+==========
+
+  Retrace server - provides complete tracebacks for unhandled crashes and
+  show aggregated information for developers
+
+Contact Information
+-------------------
+
+Owner:
+  Fedora QA Devel, Fedora Infrastructure Team, ABRT team
+Contact:
+  #abrt, #fedora-admin, #fedora-noc
+Servers:
+  retrace*, faf*
+
+Purpose:
+  Provides complete tracebacks for unhandled crashes and
+  show aggregated information for developers.
+
+Description
+-----------
+
+The physical server runs two main servers: retrace-server and FAF.
+
+Retrace-server
+==============
+
+The upstream for retrace server lives at:
+
+  https://github.com/abrt/retrace-server
+
+When user has ABRT client installed and some software crash with unhandled
+exception (traceback or core dump). User can send the request to retrace-server,
+which will install the same set of package plus debuginfo and return to user
+traceback where instead of plain pointers are names of functions. This is more
+useful for further debugging.
+
+Retrace-server can allow uploading of user coredump via WebUI, but this has been
+disabled in Fedora instance.
+
+FAF
+===
+
+When user decide that he want to report his crash then problem is sent to FAF.
+ABRT can be also configured to send microreports automatically without user
+intervention.
+FAF can agreggate the data and similar reports group into one entity (called
+Problems). FAF provides nice WebUI for developers who can see crashes in their
+packages. It lives at:
+
+  https://retrace.fedoraproject.org/faf/
+
+Playbook
+--------
+
+Playbook is splitted into several roles. There are two main roles
+
+ * abrt/faf
+ * abrt/retrace
+
+These are copy from upstream. You should never update it directly.
+The new version can be fetched from upstram using:
+
+ # cd ansible/abrt
+ # rm -rf faf retrace
+ # ansible-galaxy install -f -r requirements.yml --ignore-errors -p ./
+
+You should review the new differences and commit and push.
+
+Then there are some roles, which are local for our instance:
+
+ * abrt/faf-local - This is run *before* abrt/faf.
+ * abrt/retrace-local - This is run *after* abrt/retrace.
+ * abrt/retrace-local-pre - This is run *before* abrt/retrace.
+
+Services
+--------
+
+FAF and retrace-server are web applications. So you just need httpd running.
+
+Cron
+----
+
+Both FAF and retrace-server has bunch of cron tasks. They are *not* under
+/etc/cron*, but are user crons. They are under users: faf and retrace.
+
+You can list those crons using:
+
+ * sudo -u faf crontab -l
+ * sudo -u retrace crontab -l
+
+All cronjobs should be Ansible managed. Just make sure if you delete some
+cron from Ansible that it does not remain on the server. (not always possible
+with state=absent)
+
+Directories
+-----------
+
+- /srv/ssd - fast disk, used for PostgreSQL storage
+- /srv - big fat disk, used for storing packages. Mainly:
+  - /srv/faf/lob
+  - /srv/retrace
+- /srv/faf/db-backup/ - Daily backups of DB. No rotating yet. Needs to be
+    manually deleted occasionally.
+- /srv/faf/lob/InvalidUReport/ - Invalid reports, can be pretty big.
+    No automatic removal too. Need to be purged manually occasionally.
+
+Front-page
+----------
+
+The main web page is handled by package abrt-server-info-page, which can be
+controlled using:
+ /usr/lib/python2.7/site-packages/abrt-server-info-page/config.py
+
+DB
+--
+
+Only FAF use DB. We use our own instance of PostgreSQL. You can connect to it
+using:
+  sudo -u faf psql faf
-- 
2.13.0

