Weblate transition & integration for upstream projects

[Daniel P. Berrangé]

I'm the maintainer for various upstream projects [1] which are currently
using Fedora's Zanata for translation services and I saw the recent
change proposal for Fedora to shift to Weblate in the next release
cycle.

First I'm wondering what sort of time frame is being targetted for move
of the upstream projects which are independant of Fedora's release cycle ?


Next, and more importantly, I'm trying to understand how we will integrate
with Weblate....


With Zanata the maintainers just run "zanata push" and "zanata pull"
when
desired and we have rules integrated into our build system to do various
processing steps to build the .pot file and .po files as needed.

Looking at what was done for ibus-typing-booster, IIUC Weblate is given
commit access directly to the git repository.


This is tricky for us as, AFAICT, it assumes the project is storing the
.pot files and .po files in git which is not the case for an increasing
number of our virt related projects - libvirt, libvirt-glib & virt-viewer.
It is also not appealing from a security POV to grant commit access to
a 3rd party service.

The .pot file is never stored as it is an huge auto-generated file which
would be out of date as soon as it was committed. We would always rebuild
this from master source files whenever it is needed so it is fully updated.

We also recently stopped storing full .po files [2], again because they are
huge files with a large amount of redundant information in them. Instead
we store what we call ".mini.po" files. This is the same file format as
a normal ".po" file, but with

 - all filenames + line numbers stripped
 - all non-translated msgids stripped
 - msgids sorted alphabetically

This is loss-less - all the removed information can be automatically
re-added by processing with msgmerge & the latest .pot file.

This cut down size of stored /po directory from 100 MB to just 18 MB.

It also means that when we push updates with new translations, the
git patch diffs have high signal/noise ratio and are very small.
eg a git commit refreshing translations is now approx 700 KB, instead
of the ~80 MB we saw when storing full .po files.

This can also makes distributed tar.xz files smaller as we only need
include the .mini.po files.

We have build rules from convert from .po files to .mini.po files which
are run when we pull down new translations from zanata, and the inverse
to convert from .mini.po  files to .po files which we run at build time
when creating the binary .mo files for installation into /usr/share/locale

All this means though that Weblate can't simply operate directly on our
git repo. There's no .pot for it to find, nor can it push normally formatted
.po files.

Thus I'm trying to understand how we'll go about integrating with Weblate.

I'm guessing there's no way to control how it interacts with git - eg no
way to tell it to run meson to generate a .pot file, nor any way to tell
it to run meson to generate .mini.po files

The 'wlc download' and 'wlc upload' commands appear to be similar in
purpose to zanata's  'pull' and 'push' commands, but when I try to
run
'wlc download ibus-typing-booster/app' it reports "Error: Not
supported".
Essentially if there's a way to push a .pot file and pull a .po file
from Weblate's REST API we'll be able to continue our current workflow.

Finally in Weblate after authenticating with FAS, I still don't have the
ability to create new projects. Is this intentionally restricted ? I was
hoping to do some testing with Weblate & one of the virt apps like
virt-viewer to try to understand its operation better before any formal
transition period.

Regards,
Daniel

[1] libvirt, libvirt-glib, virt-viewer, entangle, osinfo-db, osinfo-db-tools,
    libosinfo, virt-manager, libvirt-sandbox
[2]
https://www.berrange.com/posts/2018/11/29/improved-translation-po-file-ha...
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Mike FABIAN]

Daniel P. Berrangé <berrange(a)redhat.com&gt; さんはかきました:

>  Looking at what was done for ibus-typing-booster, IIUC Weblate is
> given
>  commit access directly to the git repository. 
It does not commit directly to the ibus-typing-booster github repo,
it does pull requests like this:

https://github.com/mike-fabian/ibus-typing-booster/pull/59

>  The 'wlc download' and 'wlc upload' commands appear
> to be similar in
>  purpose to zanata's  'pull' and 'push' commands, but when I try to
> run
>  'wlc download ibus-typing-booster/app' it reports "Error: Not
> supported".
>  Essentially if there's a way to push a .pot file and pull a .po file
>  from Weblate's REST API we'll be able to continue our current workflow. 
I could  not get that working either, I would like to find out
what is wrong.

-- 
Mike FABIAN <mfabian(a)redhat.com&gt;
睡眠不足はいい仕事の敵だ。

[Jean-Baptiste Holcroft]

Le 2019-11-05 12:15, Daniel P. Berrangé a écrit :
>  I'm the maintainer for various upstream projects [1] which are 
>  currently
>  using Fedora's Zanata for translation services and I saw the recent
>  change proposal for Fedora to shift to Weblate in the next release
>  cycle. 

Thank you for coming here and engaging this discussion!

>  First I'm wondering what sort of time frame is being targetted
> for move
>  of the upstream projects which are independant of Fedora's release 
>  cycle ? 
Wish: by February, all projects are using the new translation platform.
Reality: there is no hard timeframe, but the translators won't go on 
multiple translation platform, and because most important packages will 
have migrate, it may lower a lot the number of contributions to non 
migrated projects.

>  Next, and more importantly, I'm trying to understand how we will
>
>  integrate
>  with Weblate....
>  
>  The .pot file is never stored as it is an huge auto-generated file 
>  which
>  would be out of date as soon as it was committed. We would always 
>  rebuild
>  this from master source files whenever it is needed so it is fully 
>  updated. 
Suboptimal is the pot file format, but also is the best we have.
Next generation may be the "Fluent" file format from Mozilla, wait and 
see.

>  We also recently stopped storing full .po files [2], again because
> they 
>  are
>  huge files with a large amount of redundant information in them. 
>  Instead
>  we store what we call ".mini.po" files. This is the same file format as
>  a normal ".po" file, but with
>  
>  [...]
>  
>  Thus I'm trying to understand how we'll go about integrating with 
>  Weblate. 
It's an interesting thing you did!

I see a few solutions:
* you create a repository dedicated to pot and po files,
* you interact with Weblate using the command line interface or one of 
the APIs.

>  Finally in Weblate after authenticating with FAS, I still don't
> have 
>  the
>  ability to create new projects. Is this intentionally restricted ? I 
>  was
>  hoping to do some testing with Weblate & one of the virt apps like
>  virt-viewer to try to understand its operation better before any formal
>  transition period. 
Yes, project creation is locked on purpose and won't be open.
Correct configuration isn't so easy and it requires some interaction 
with localization team.
The way it worked with Zanata was way too loose, nobody was informed 
when a new project came or when a project created a new document/branch.


I would love to collaborate with you to find a correct workflow using 
either method of your choice (and your help with DNF and anaconda team 
would be much appreciated), but first: _we need approval from Brian 
Exelbierd or Ben Cotton to continue deployment_. Red Hat and Weblate 
should finish the paperwork (billing + terms of use). Brian is the one 
responsible for this action.


Note for readers: I will always suggest to keep it simple and include 
pot/po files in repository for little projects. Don't invest time in 
automation as it will add maintenance work and lower the proximity 
between upstream source code and the translation platform.

Jean-Baptiste

[Daniel P. Berrangé]

On Tue, Nov 05, 2019 at 01:54:24PM +0100, Jean-Baptiste Holcroft wrote:
>  Le 2019-11-05 12:15, Daniel P. Berrangé a écrit :
>  > I'm the maintainer for various upstream projects [1] which are currently
>  > using Fedora's Zanata for translation services and I saw the recent
>  > change proposal for Fedora to shift to Weblate in the next release
>  > cycle.
>  
>  
>  Thank you for coming here and engaging this discussion!
>  
>  > First I'm wondering what sort of time frame is being targetted for move
>  > of the upstream projects which are independant of Fedora's release cycle
>  > ?
>  
>  Wish: by February, all projects are using the new translation platform.
>  Reality: there is no hard timeframe, but the translators won't go on
>  multiple translation platform, and because most important packages will have
>  migrate, it may lower a lot the number of contributions to non migrated
>  projects. 
Yeah, we want our upstream projects to be where the translaters are
most active.

Assuming we can figure out the workflow, I don't see a big problem
with moving the projects I'm involved with by Feb timeframe.

>  > We also recently stopped storing full .po files [2], again
> because they
>  > are
>  > huge files with a large amount of redundant information in them. Instead
>  > we store what we call ".mini.po" files. This is the same file format as
>  > a normal ".po" file, but with
>  > 
>  > [...]
>  > 
>  > Thus I'm trying to understand how we'll go about integrating with
>  > Weblate.
>  
>  It's an interesting thing you did!
>  
>  I see a few solutions:
>  * you create a repository dedicated to pot and po files,
>  * you interact with Weblate using the command line interface or one of the
>  APIs. 
The CLI/API option is the least disruptive to us - separate repo
is probably an undesirable burden to maintainers.

>  > Finally in Weblate after authenticating with FAS, I still
> don't have the
>  > ability to create new projects. Is this intentionally restricted ? I was
>  > hoping to do some testing with Weblate & one of the virt apps like
>  > virt-viewer to try to understand its operation better before any formal
>  > transition period.
>  
>  Yes, project creation is locked on purpose and won't be open.
>  Correct configuration isn't so easy and it requires some interaction with
>  localization team.
>  The way it worked with Zanata was way too loose, nobody was informed when a
>  new project came or when a project created a new document/branch.
>  
>  
>  I would love to collaborate with you to find a correct workflow using either
>  method of your choice (and your help with DNF and anaconda team would be
>  much appreciated), but first: _we need approval from Brian Exelbierd or Ben
>  Cotton to continue deployment_. Red Hat and Weblate should finish the
>  paperwork (billing + terms of use). Brian is the one responsible for this
>  action. 
I'd probably want start off by doing a PoC with the moving the virt-viewer
project as its fairly small & low traffic. I can look at it whenever things
are ready from POV of the Fedora weblate deployment.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Mike FABIAN]

Daniel P. Berrangé <berrange(a)redhat.com&gt; さんはかきました:

>  On Tue, Nov 05, 2019 at 01:54:24PM +0100, Jean-Baptiste Holcroft
> wrote:
> > 
> > It's an interesting thing you did!
> > 
> > I see a few solutions:
> > * you create a repository dedicated to pot and po files,
> > * you interact with Weblate using the command line interface or one of the
> > APIs.
>
>  The CLI/API option is the least disruptive to us - separate repo
>  is probably an undesirable burden to maintainers. 
I would also like to use the command line interface if possible.
The pull requests I currently get are fine, I like it, but it
would be nice if I also could force weblate to update the pull request
immediately (Maybe “wlc commit” would do that??). Currently
I only get the error message

    Error: Object not found on the server (maybe operation is not supported on the
server)

whenever I try to use wlc.

-- 
Mike FABIAN <mfabian(a)redhat.com&gt;
睡眠不足はいい仕事の敵だ。

[Jean-Baptiste Holcroft]

Le 2019-11-05 14:21, Mike FABIAN a écrit :
>  I would also like to use the command line interface if possible.
>  The pull requests I currently get are fine, I like it, but it
>  would be nice if I also could force weblate to update the pull request
>  immediately (Maybe “wlc commit” would do that??). Currently
>  I only get the error message
>  
>      Error: Object not found on the server (maybe operation is not
>  supported on the server)
>  
>  whenever I try to use wlc. 
which command do you type?

[Jean-Baptiste Holcroft]

Le 2019-11-05 14:51, Jean-Baptiste Holcroft a écrit :
>  Le 2019-11-05 14:21, Mike FABIAN a écrit :
> > I would also like to use the command line interface if possible.
> > The pull requests I currently get are fine, I like it, but it
> > would be nice if I also could force weblate to update the pull request
> > immediately (Maybe “wlc commit” would do that??). Currently
> > I only get the error message
> > 
> >     Error: Object not found on the server (maybe operation is not
> > supported on the server) 
Weblate team answered me, if you want to manually download, your shoud 
explicitly write the language name: wlc download 
ibus-typing-booster/app/fr
They will improve the error message

but to really interact with Weblate in your workflow (pot and po files 
are in you repo):

# Commits changes in Weblate object (translation, component or project).
wlc commit

# Pushes changes in Weblate object into remote repository (translation, 
component or project).
wlc push

Jean-Baptiste

[Mike FABIAN]

Jean-Baptiste Holcroft <jean-baptiste(a)holcroft.fr&gt; さんはかきました:

>  Le 2019-11-05 14:51, Jean-Baptiste Holcroft a écrit :
> > Le 2019-11-05 14:21, Mike FABIAN a écrit :
> >> I would also like to use the command line interface if possible.
> >> The pull requests I currently get are fine, I like it, but it
> >> would be nice if I also could force weblate to update the pull request
> >> immediately (Maybe “wlc commit” would do that??). Currently
> >> I only get the error message
> >>     Error: Object not found on the server (maybe operation is not
> >> supported on the server)
>
>  Weblate team answered me, if you want to manually download, your shoud
>  explicitly write the language name: wlc download 
>  ibus-typing-booster/app/fr
>  They will improve the error message 
wlc download ibus-typing-booster/app/fr

works for me ☺.

>  but to really interact with Weblate in your workflow (pot and po
> files
>  are in you repo):
>
>  # Commits changes in Weblate object (translation, component or project).
>  wlc commit
>
>  # Pushes changes in Weblate object into remote repository
>    (translation, component or project).
>  wlc push 
These two commands do not work for me:

$ wlc commit
Error: You don't have permission to access this object
mfabian@taka:/local/mfabian/src/ibus-typing-booster (release-2.7.3 $%)
$ wlc push
Error: You don't have permission to access this object
$

I am also a bit confused what they actually mean.
Does “wlc push” update the push request in my ibus-typing-booser github repo?

-- 
Mike FABIAN <mfabian(a)redhat.com&gt;
睡眠不足はいい仕事の敵だ。

[Jean-Baptiste]

Le 05/11/2019 à 12:15, Daniel P. Berrangé a écrit :
>  Next, and more importantly, I'm trying to understand how we will
> integrate
>  with Weblate....
>  
Hello Daniel,

let's integrate your projects to the new translation platform.

where are the public bug trackers of your projects?

I have a group of question for each repository: 
https://fedoraproject.org/w/index.php?title=L10N_Move_to_Weblate/Message_...

thanks for your help

[Daniel P. Berrangé]

On Mon, Dec 09, 2019 at 09:11:35PM +0100, Jean-Baptiste wrote:
>  Le 05/11/2019 à 12:15, Daniel P. Berrangé a écrit :
>  > Next, and more importantly, I'm trying to understand how we will integrate
>  > with Weblate....
>  > 
>  Hello Daniel,
>  
>  let's integrate your projects to the new translation platform.
>  
>  where are the public bug trackers of your projects? 
For libvirt related projects currently we use

  https://bugzilla.redhat.com/enter_bug.cgi?product=Virtualization%20Tools

>  I have a group of question for each repository:
>
> https://fedoraproject.org/w/index.php?title=L10N_Move_to_Weblate/Message_...
So that I can gain a understanding of the weblate integration, lets start
with only the libvirt-glib project first

https://libvirt.org/git/?p=libvirt-glib.git;a=log
https://fedora.zanata.org/project/view/libvirt-glib/?dswid=478

 * which branch is your development branch?

   master, but note weblate won't be allowed to either push or pull,
   we must manually upload & download .pot/.po files

 * have you merged latest translation from Zanata and locked the project?

   I've hit the "make this project readonly" assuming that's what
   you mean by 'locked'.

 * Weblate will handle updates when pot file changes, don't edit po files for this
[2]

   As above, we can't have weblate directly accessing git

 * what is the license of translation? (please use a code from
https://spdx.org/licenses/)

    LGPLv2-or-later

 * do you have any announcement/warning you would like to display to the translators? (it
will be displayed in Weblate)

   Questions/clarifications about translations should be sent to the
   libvir-list(a)redhat.com mailing list. For further information
   consult https://libvirt.org

 * do you need us to activate any specific checks? (this is a setting per component [3])

   Any .po file validation rules relevant to the C language.

   Especially validating that printf format specifiers have
   not been re-ordered/removed/changed, since we see that
   screw up alot.


 * do you need us to automatically detect new translation files? (typical usecase: website
translation with one translation file per page)

   Nope



Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Jean-Baptiste]

Le 12/12/2019 à 17:12, Daniel P. Berrangé a écrit :
>  On Mon, Dec 09, 2019 at 09:11:35PM +0100, Jean-Baptiste wrote:
> > Hello Daniel,
> > let's integrate your projects to the new translation platform.
> >
> > where are the public bug trackers of your projects?
>  For libvirt related projects currently we use
>
>     https://bugzilla.redhat.com/enter_bug.cgi?product=Virtualization%20Tools
>  
Thanks for your answer, I've created a bug there: 
https://bugzilla.redhat.com/show_bug.cgi?id=1783579

let's discuss all of this on bugzilla, and let interested people to 
register to the conversation