The State of Publishing Documentation in Fedora

Thu Feb 12 00:38:44 UTC 2015

On 02/11/2015 03:27 PM, Brian (bex) Exelbierd wrote:
> Hi,
>
> On Feb 11, 2015, at 3:53 PM, Pete Travis <me at petetravis.com> wrote:
>
>> We've tossed around ideas in #fedora-docs, and in discussing it, I've
>> developed a wishlist for an improved publishing platform:
>>
>>  - We use release-based branches to maintain content for a given Fedora
>> release.  Anything committed to these branches is intended for
>> publication, so that should happen automatically.
> Yes.
>
>>  - It should be simpler to make drafts available.  If we can automate
>> publishing on release branches, we can do the same with master.
> Addressed below with Jenkinscat :)  I’d also like to see an easy way for individuals to see their non-master committed work in a stage environment.  I don’t think the answer lies in local builds though as they aren’t shareable.  Pulling this off will almost certainly require us to be willing to consider a modification to our branching strategy.  I have ideas here, if there is interest.
Interest, yes! I don't see a way around "release" branches, but I do
like the idea of shared work branches, if that's what you're getting at.
>
>>  - We don't do as good of a job as we should with pushing and pulling
>> strings from translation.  Pushes and pulls could be automated.
> Can this be tied to the publishable branching above?
Yes, it seems that we can simply add the appropriate commands/actions
for the zanata client to the Jenkins build job.  If the build passes,
push the strings.  Pull the strings, and if the build passes, merge the
new strings into the release branch.
>
>>  - There should be an effective way to publish and organize sundry
>> articles.
> I would encourage us to think about a single publication system here.  Therefore the challenge is how to present the articles, not how to publish them.  The articles can all live in a single repo and all be republished when one changes.  The right publication means this isn’t a problem.  I am doing this with some non-Fedora docs now and the runtime is trivial (markdown -> html).
Right, the presentation is what needs the most work, the front end that
users will browse through to get to the individual articles and guides. 
There are both tooling and design challenges there.  but, erm... single
repo? like web.git?  Maybe I'm misunderstanding you, but nobody wants to
keep that around :P
>>  - The publishing platform should be able to process arbitrary markup
>> formats.  I think docbook is great, and publican does a good job with
>> it, but there will be room for more contributions if that isn't mandatory.
> This is the biggest challenge, after design, IMHO.
>
> I have not yet found a great multi-input publishing mechanism.  Pandoc comes close, but has some serious shortcomings in some input formats.  Additionally, being written in Haskell, it is harder to find people who can extend it (in my experience).  Additionally, even if that works you still have to deal with branding.  Asciidoctor is supposed to be better, and rst via sphinx and others is also great.  Lastly, there are engines like Jekyll which might be a good fit too.  
>
> Here I think we need to either make some arbitrary decisions (i.e. we support only x and y, or we only support this subset of markup z, etc.) or risk having to support a lot of conversion engines.  My suggestion for today is to try to define the minimal markup needs for our publication chain and i18n and then choose two markups (Docbook and ???) and move forward.
>
> As a start on minimal markup needs, it sounds like we need entity support, possibly xinclude-style insertions, and a way to flag material that shouldn’t be translated.  We also need to decide whether we will allow non-semantic markup.  If we can ensure strong reviews and/or gating on publication, non-semantic markup can be made to work.
>
> I’d also like to see us consider a way to easily enable drive-by contribution and editing.  I’ve been working on an architecture for a different project that has similar requirements.  It currently exists mostly in my mind, but I hope to get it into writing and demo mode soon.
>

I really don't want to get hung up on support for additional formats at
this stage.  Given tooling to dynamically build a front end, it
shouldn't be a problem to add support for whatever format.  Jenkins can
probably handle whatever we throw at it, we can work out the tooling to
build other formats after the core solution in place.

>>  - The frontend of the site should be adaptable.  docs.fp.o would
>> ideally use the same design elements as other official Fedora sites for
>> a more unified appearance.
> Yes.  Anything we can do to offload branding and design is a huge massive win!
>
>>  - Some validation on commits would be nice.
> Take a look at https://github.com/emender it is a small but growing validator for integration into CI.  It isn’t commit level directly, but with the right kind of architecture could get you close.
>
>> The working theory at this point is to use a continuous integration
>> system like Jenkins to automate builds.  I've had a Jenkins instance
>> running locally this week, with new builds triggered via fedmsg signals
>> generated by our commits.  This part works smoothly, with the exception
>> of a fedwatch bug that I've crudely worked around, but triggers using
>> python-fedmsg really wouldn't be too difficult.
> This is fantastic.  I read this as you suggesting we can do both stage and publish on this platform.
Sure, we could definitely have ie drafts.docs.fedoraproject.org that's
built from master, or with navigable content from feature work branches,
etc.   For now, it is probably best limit the scope to a public-facing
solution, then iterate.
>> The infra folks are
>> also working on a Jenkins setup, so there's potential to share
>> experience and buildslaves.  Turning that into a browseable frontend is
>> more immediately viable than you might think;
>> http://sourceforge.net/projects/jenkinscat/ is designed just for that
>> and can be customized for our needs.  
> Leverage CI: +1
>
>> Jenkins plugins to cycle strings
>> from Zanata, or some other method, could come down the road.  There's
>> room for other enhancements, too.
> I’d love to hear from the i18n folks how they’d like to see this.  Do they want continuous updates or to work on a cadence?
I wrote plugins, then actually started looking into it and talked to
#zanata, and all we need to do is use the normal zanata cli tools as
part of the Jenkins build job. The Jenkins git plugin has some features
to merge-on-success we could leverage.

 The loudest voice here has been Jerome Fenal from the very active
French team, and he advocates a continuous flow.  In a practical sense,
we can set the master branch to push to master on Zanata, the F21 branch
for F21 on zanata, etc; translators that want the continuous flow can
work on master, those that want something more curated can work on the
release branches.  If Zanata's translation memory works like it did for
Transifex, the translations for identical strings will automatically be
available for all branches, no redundant translation required.
>> The CI idea is something I'm excited about, and motivated to work on. 
>> I'd love to spend that time working with others, if you're interested.
> I’d like to work on this, after March 15.  I am booked until then.  I’d love to push some container tech in here because it is cool and probably a good fit, but that isn’t a requirement.
>
> regards,
>
> bex
Sounds great! Keep the theory coming, until you have time for
implementation.

-- 
-- Pete Travis
 - Fedora Docs Project Leader
 - 'randomuser' on freenode
 - immanetize at fedoraproject.org