docs.fedoraproject.org redesign -- very long!

[Ruediger Landmann]

For at least the last year or so, we've been looking for ways to make 
docs.fedoraproject.org (d.fp.o) more usable for our readers and less 
onerous for our contributors. At the last docs meeting,[0] nb outlined a 
proposal to replace the current CVS mechanism, and I presented a brief 
demo of what publishing looks like with the forthcoming Publican 2.0.[1]

To further the discussion, I thought I'd present the options a bit more 
completely here.

Sorry this is so very wordy! I threw together some graphics to accompany 
this post that hopefully might make it a bit more digestible.

Please have a read and share your thoughts.

Cheers
Rudi


================================
OPTION 0 -- What we have now -- CVS + PHP
================================

1. Writers[2] pull down the source code of a doc from a repo (mostly 
git) and at least some part of d.fp.o site as stored in a CVS repo. 
Unless you're working on multiple docs, you only need the directory that 
corresponds to your  book, not the entire site.

2. Writers build the book from source on their local machines, and copy 
the Publican output into their local copy of the d.fp.o directory for 
that book. They also hand-edit PHP files as needed to provide the index 
pages.

3. Writers perform CVS voodoo to add and commit the files back up to the 
CVS repo, and tag the files LIVE. Within a few hours, the web server 
picks up any files tagged LIVE in the CVS repo and publishes them.

PROBLEMS:
1. Editing the PHP index files by hand is onerous and seriously prone to 
error.  Everyone who has worked on this has routinely made mistakes 
keeping track of which book in which format in which language goes where.

2. The CVS process is notoriously temperamental, unforgiving, and prone 
to error. Mistakes are frequent, and correcting them is difficult and 
frustrating (and in some cases, not even possible).


================================
OPTION 1 -- Proposal from Infra -- git + static HTML
================================

1. Writers pull down the source code of a doc from a repo (mostly git) 
and a git repo containing the entire d.fp.o website.

2. Writers build the book from source on their local machines, and copy 
the Publican output into the local copy of the git directory for that 
book. They also hand-edit html files as needed to provide the index pages.

3. Writers commit the files back up to the d.fp.o git repo. Within a few 
hours,  the web server synchronises with the repo and publishes the new 
files.

ADVANTAGES:
1. Replacing CVS with git eliminates the single greatest source of 
failure and frustration.

2. The tools available to edit static html index files are more powerful 
and  more user-friendly than editing PHP in a text editor, so mistakes 
are less likely  to occur.

DISADVANTAGES:
1. The initial git clone is massive -- something like 2 GB -- and this 
will only grow bigger over time.

2. Indexes are still built by hand.


================================
OPTION 2 -- Publican
================================
Note technology demonstration available at 
http://publictest8.fedoraproject.org/fedoradocs/public_html -- hurried, 
and slightly broken in places

1. Writers ssh into a remote system (let's call it the "build system" 
and pull  down the source code of a doc from a repo (mostly git) onto 
that system.

2. Writers build the book with Publican on the build system, and run 
the  "publican install_book" command. Publican adds the book to a local 
copy of d.fp.o and automatically rebuilds the site index.

3. At regular intervals, any new or changed files in the version of 
d.fp.o on the build server are copied to the web server, using scp or 
rsync or  something similar, initiated either on the build server end or 
the web server end.[3]

ADVANTAGES:
1. Writers are finally completely free from the two most onerous and  
error-prone parts of the publication process: using a version-control 
system as a publishing tool, and having to rebuild indexes by hand. Even 
if a book itself is misconfigured, screwups are contained to a single 
book in a single version in a single language, unlike some incidents 
we've had with hand-edited PHP.

DISADVANTAGES:
None of which I'm aware, however we'll need some new infrastructure in 
place  and we'll all no doubt make mistakes while getting used to a new 
way of  doing things.

================================
OPTION FOR TOMORROW -- Publican + Koji
================================

1. Writers pull down the source code of a doc from a repo (mostly git)

2. Writers run "publican package" on their local systems with an option 
that  tells Publican to upload the SRPM package to a Koji instance. Koji 
builds the package and places it in a repo.

3. At regular intervals, the web server pulls any new docs packages from 
the  repo and installs them. Documentation packages built with Publican  
automatically install into subdirectories of /var/www/html/ and 
regenerate any Publican site index present.

ADVANTAGES:
1. Publishing is completely automated. This is actually the job that 
Publican was designed to do and pretty much how it's been running 
internally within Red Hat for  years. Writers and translators can focus 
on writing and translating and leave everything else to the machines.

2. Since the documentation is going to a repo, Fedora users world-wide 
could yum install docs packages from this repo. We might also be able to 
take advantage of the new langpacks feature in yum to help users find 
documentation in their own language.

DISADVANTAGES:
No disadvantages of which I'm aware, but the current stumbling blocks to 
implementing such a system right now are:

1. Publican doesn't yet support sending packages to arbitrary Koji 
instances! So this would require some code changes to Publican. Since 
Publican already supports Red Hat's Brew system, I suspect these are 
relatively minor.

2. We would need a Koji instance on which to build docs. Writers and 
translators who are trusted to publish docs would need to be approved as 
packagers on that Koji instance. In a perfect world, this would be the 
same Koji instance that the Fedora Project uses already to build 
software packages. However, most writers and translators do  not need to 
or want to package software. We would need some administrative changes 
within the project that would approve certain writers and translators to 
build docs packages only. We would also likely need a different  package 
approval process for docs, as docs packages would quickly overwhelm the 
existing system.[4] Failing to get these admin changes, we would need a 
separate Koji instance somewhere, and need Infra to maintain it for us.


================================
Notes
================================

[0] minutes -- 
http://lists.fedoraproject.org/pipermail/docs/2010-April/012130.html

[1] Publican 2.0 is still under heavy development, but I have (unsigned) 
packages for Publican 1.99 and its web component available from my 
fedorapeople page: http://rlandmann.fedorapeople.org/publican/ and you 
can find instructions for the new web stuff on Jeff Fearn's page: 
http://jfearn.fedorapeople.org -- please download and experiment! 
However, if you're involved in building or publishing F13 docs, please 
do not use Publican 1.99 for this, and in particular, do *not* update 
POT or PO files for current work with Publican 1.99!

[2] I say "writers" here, but the same applies to translators, although 
to date, translators have not generally been involved in publishing 
their work. One advantage of rethinking how we publish is to allow 
translators and language teams direct control over their work, without 
making them rely on writers as "gatekeepers" for their docs.

[3] Technically, there's no reason why the build server and web server 
need to be two separate machines, or why the "publican install_book" 
command couldn't write directly to a publicly accessible directory on 
the web server. I've suggested separating them mainly out of paranoia, 
but maybe I'm being overly cautious?

[4] Hopefully in the context of this whole workflow, Publican's 
insistence on packaging languages one-at-a-time and on including version 
numbers in package names makes more sense -- it's these distinctions 
that produce the table of contents for the site. It also illustrates why 
a different package approval process would be necessary for docs -- 15 
books in up to 12 regularly-translated languages is up to 180 new 
packages for each Fedora release...