The State of Publishing Documentation in Fedora

[Pete Travis]

Hi all,

I've given a lot of thought to our publishing situation.  The current
solution, using an unmaintained version of Publican on an unsupported
Fedora release, is untenable. The site we have been maintaining in
web.git is incompatible with current versions of publican, besides being
massive and prone to breakage.  We have been working towards a
replacement, slowly.

The proposed successor to web.git is an RPM based system where Publican
creates SRPMs of the guides, Fedora's koji buildsystem builds packages
from the RPMs, and a VM instance installs the package to build the
site.  Creating the SRPM would be analagous to the `publican build
--embedtoc ...` from the old process, and when the VM installs the
package, it does the `publican install-book` step of the process.  We
currently have the koji infrastructure in place, a mostly viable way for
newly built packages to automatically be installed on the VM, and a
functional but mostly vanilla site frontend.  Along the way, I've
identified a few caveats to this method, to which I'll add some general
observations:

  - Each language of each release of each guide is a unique package.  To
publish anything new, we must coordinate with releng to have the package
defined in koji.  Releng is already overburdened.
  - Each language must have a separately maintained revision history.
  - The buildroot shared by koji and the VM required updated versions of
publican and publican's dependencies, and now must be maintained there
(which is currently not happening afaik).
  - None of the breakage problems we've dealt with in web.git will go
away.  Unusual language codes, bad draft procedures, wonky *_Info.xml
issues, and whatever black magic that ends with my manually running
sqlite invocations against the site db - that's all still there.
  - The site frontend needs to be completely redesigned from the ground
up, and we have not demonstrated the motivation to see that through. 
There are intricacies here that we didn't discover from a
straightforward following of the publican user's guide, and many
subjectively frustrating surprises.
  - A publican website can only publish publican-friendly content.  To
the best of my knowledge, that means docbook only.  Our contributor base
is declining, and prospective contributors have consistently
demonstrated a lack of interest in writing docbook.
  - A publican website does not provide an effective presentation of
many smaller articles.  I've been disagreed with on this point, and will
concede that it's definitely possible to have 200 things under that F21
category, but I maintain that presentation in that way would be inimical
to reader browsing.

With all this in mind, I've decided to be honest with myself, and you,
in saying that I'm not motivated to make the publican website frontend
work.  Something must be done to deal with our site's incompatibility
with current Publican, but from a usage perspective, I feel like there's
a very high upfront investment - and a moderate continuing investment in
maintaining the el6-docs koji repo - with no substantial gain in the
product delivered.  For normal usage, we've simply replaced `publican
build;publican install-book` with `publican package;koji build`.  Rudi
had initially volunteered to do *all* of the setup to make this work,
but he's a busy guy and it really isn't fair to expect him to deal with
that level of implementation for us.

We've tossed around ideas in #fedora-docs, and in discussing it, I've
developed a wishlist for an improved publishing platform:

  - We use release-based branches to maintain content for a given Fedora
release.  Anything committed to these branches is intended for
publication, so that should happen automatically.
  - It should be simpler to make drafts available.  If we can automate
publishing on release branches, we can do the same with master.
  - We don't do as good of a job as we should with pushing and pulling
strings from translation.  Pushes and pulls could be automated.
  - There should be an effective way to publish and organize sundry
articles.
  - The publishing platform should be able to process arbitrary markup
formats.  I think docbook is great, and publican does a good job with
it, but there will be room for more contributions if that isn't mandatory.
  - The frontend of the site should be adaptable.  docs.fp.o would
ideally use the same design elements as other official Fedora sites for
a more unified appearance.
  - Some validation on commits would be nice.

The working theory at this point is to use a continuous integration
system like Jenkins to automate builds.  I've had a Jenkins instance
running locally this week, with new builds triggered via fedmsg signals
generated by our commits.  This part works smoothly, with the exception
of a fedwatch bug that I've crudely worked around, but triggers using
python-fedmsg really wouldn't be too difficult.  The infra folks are
also working on a Jenkins setup, so there's potential to share
experience and buildslaves.  Turning that into a browseable frontend is
more immediately viable than you might think;
http://sourceforge.net/projects/jenkinscat/ is designed just for that
and can be customized for our needs.  Jenkins plugins to cycle strings
from Zanata, or some other method, could come down the road.  There's
room for other enhancements, too.

The CI idea is something I'm excited about, and motivated to work on. 
I'd love to spend that time working with others, if you're interested.

--Pete

-- 
-- Pete Travis
 - Fedora Docs Project Leader
 - 'randomuser' on freenode
 - immanetize at fedoraproject.org