Stats (not) updating correctly

[Dimitris Glezos]

I managed to find some time today to look into why our stats weren't
updating correctly. I just made a small tweak to our update script that
I believe has fixed the bug. yay.

  http://translate.fedoraproject.org/

Please check your language's statistics and report any problems at bug
report #248260.

Only modules that weren't using intltool for their i18n were affected.
Damned Lies is designed to support intltool-enabled modules, ie. ones
with the files POTFILES.in, LINGUAS, etc and build the POT file
on-the-fly. Modules with POT files provided by the developer, ie ones
not using intltool, will most likely *always* have more problems than
the other ones, simply because upstream is not focused on maintaining
support for them. I've tried to make this point clear from early on. :)

Note that DL doesn't automatically merge POT changes into PO files, so
better compare the stats with your checked-out copy instead of
i18n.redhat.com, which merges before showing the statistics.

-d


-- 
Dimitris Glezos
Jabber ID: glezos(a)jabber.org, GPG: 0xA5A04C3B
http://dimitris.glezos.com/


"He who gives up functionality for ease of use
loses both and deserves neither." (Anonymous)
--

[Domingo Becker]

2007/9/12, Dimitris Glezos <dimitris(a)glezos.com&gt;:
>  I managed to find some time today to look into why our stats weren't
>  updating correctly. I just made a small tweak to our update script that
>  I believe has fixed the bug. yay.
>
>   http://translate.fedoraproject.org/
>
>  Please check your language's statistics and report any problems at bug
>  report #248260.
>  
Great.
Checking...

>  Note that DL doesn't automatically merge POT changes into PO
> files, so
>  better compare the stats with your checked-out copy instead of
>  i18n.redhat.com, which merges before showing the statistics.
>  
This will be a problem.
Is it possible to have a cron job to merge POT changes into PO files ?

thanks

Domingo Becker (es)

[Diego Búrigo Zacarão]

>  This will be a problem.
>  Is it possible to have a cron job to merge POT changes into PO files ? 


AFAIK it happens every four hours.

Regards

-- 
Diego Búrigo Zacarão
Linux User #402589
USE SOFTWARE LIVRE

[Dimitris Glezos]

Στις 12-09-2007, ημέρα Τετ, και ώρα 18:14 -0300, ο/η Domingo Becker
έγραψε:
>  Is it possible to have a cron job to merge POT changes into PO files
> ? 
I think it is possible, and in fact even someone from us could do it
every now and then, right?

If we get to build something that can be run from one person and get it
to become a bit mature and work right, then a cron could run it too, and
Infra will be more willing to try it out than writing it from
scratch. :)

On the other hand, this might not be optimal for a number of reasons. It
throws a big weight on Infra to maintain it, for example, and it should
work across VCSs with the ability to commit. From this point of view, it
sounds like something Transifex could do: Transifex is a translator-side
tool (in contrast to a maintainer or administrator-side tool) *with*
access to a lot of VCSs. It would be trivial to ask transifex "which
modules need merging?" and have it do a for loop and commit since most
of the code is there. Anyone willing to write a patch? :)

Also, to give another dimension to the issue: an even more efficient
(and logical) approach would be to do it right from the start (the
PO/POT creation): urge more developers to use intltool, and for those
modules that intlool can't be used (for a variety of reasons), the
person/script that updates the POT should update the POs as well.

My 2 cents.

-d


-- 
Dimitris Glezos
Jabber ID: glezos(a)jabber.org, GPG: 0xA5A04C3B
http://dimitris.glezos.com/


"He who gives up functionality for ease of use
loses both and deserves neither." (Anonymous)
--

[Ankit Patel]

Dimitris Glezos લખ્યું:
>  Στις 12-09-2007, ημέρα Τετ, και ώρα 18:14 -0300, ο/η Domingo Becker
>  έγραψε:
>    
> > Is it possible to have a cron job to merge POT changes into PO files ?
> >     
>
>  I think it is possible, and in fact even someone from us could do it
>  every now and then, right?
>
>  If we get to build something that can be run from one person and get it
>  to become a bit mature and work right, then a cron could run it too, and
>  Infra will be more willing to try it out than writing it from
>  scratch. :)
>
>     
1. Why can't we run the existing automerge script running on 
i18n.redhat.com through cronjob?
>  On the other hand, this might not be optimal for a number of reasons.
> It
>  throws a big weight on Infra to maintain it, for example, and it should
>  work across VCSs with the ability to commit. From this point of view, it
>  sounds like something Transifex could do: Transifex is a translator-side
>  tool (in contrast to a maintainer or administrator-side tool) *with*
>  access to a lot of VCSs. It would be trivial to ask transifex "which
>  modules need merging?" and have it do a for loop and commit since most
>  of the code is there. Anyone willing to write a patch? :)
>
>     
2. I could give a try, definitely take a longer time for me as out of 
touch now. OR someone who's expert in programming can jump in...
>  Also, to give another dimension to the issue: an even more efficient
>  (and logical) approach would be to do it right from the start (the
>  PO/POT creation): urge more developers to use intltool, and for those
>  modules that intlool can't be used (for a variety of reasons), the
>  person/script that updates the POT should update the POs as well.
>
>     
3. You are right. That's the best option. Developers/Package maintainers 
are not doing it for some reason i don't know.
>  My 2 cents.
>
>  -d
>     
I think the preference order for these solutions should be 3, 1, 
2...What do you say?

-- 
Regards,
Ankit Patel

[Bart Couvreur]

Op donderdag 13-09-2007 om 12:34 uur [tijdzone +0530], schreef
Ankitkumar Rameshchandra Patel:
>  Dimitris Glezos લખ્યું: 
>  > Στις 12-09-2007, ημέρα Τετ, και ώρα 18:14 -0300, ο/η Domingo Becker
>  > έγραψε:
>  >   
>  > > Is it possible to have a cron job to merge POT changes into PO files ?
>  > >     
>  > 
>  > I think it is possible, and in fact even someone from us could do it
>  > every now and then, right?
>  > 
>  > If we get to build something that can be run from one person and get it
>  > to become a bit mature and work right, then a cron could run it too, and
>  > Infra will be more willing to try it out than writing it from
>  > scratch. :)
>  > 
>  >   
>  1. Why can't we run the existing automerge script running on
>  i18n.redhat.com through cronjob? 
Well we could keep that one up and running, just for the modules on
i18n.r.c, but for the others that have moved this could become a bit
problematic, we would need to have a script for each VCS. The automerge
script works great for CVS, if it could be ported to git, hg, svn, ...
it would be a start. Although that would make us responsible for keeping
it working, keeping the weight off Infra

>  > On the other hand, this might not be optimal for a number of
> reasons. It
>  > throws a big weight on Infra to maintain it, for example, and it should
>  > work across VCSs with the ability to commit. From this point of view, it
>  > sounds like something Transifex could do: Transifex is a translator-side
>  > tool (in contrast to a maintainer or administrator-side tool) *with*
>  > access to a lot of VCSs. It would be trivial to ask transifex "which
>  > modules need merging?" and have it do a for loop and commit since most
>  > of the code is there. Anyone willing to write a patch? :)
>  > 
>  >   
>  2. I could give a try, definitely take a longer time for me as out of
>  touch now. OR someone who's expert in programming can jump in... 
Looks like a very logical thing to do. OTOH this would include some
extra work for translators before they can get started doing any real
translation: check with Transifex *if* the PO is up to date, if not run
the sync, get the file, translate, commit, ...

>  > Also, to give another dimension to the issue: an even more
> efficient
>  > (and logical) approach would be to do it right from the start (the
>  > PO/POT creation): urge more developers to use intltool, and for those
>  > modules that intlool can't be used (for a variety of reasons), the
>  > person/script that updates the POT should update the POs as well.
>  > 
>  >   
>  3. You are right. That's the best option. Developers/Package
>  maintainers are not doing it for some reason i don't know. 
+1, scratch out the middle-man

>  > My 2 cents.
>  > 
>  > -d
>  >   
>  I think the preference order for these solutions should be 3, 1,
>  2...What do you say? 
Yeah, fair enough

>  -- 
>  Regards,
>  Ankit Patel 
Regards,
Bart

-- 
Bart <couf(a)fedoraproject.org&gt; <couf(a)skynet.be&gt;
key fingerprint: 6AAB 544D 3432 D013 776D  3602 ADB6 6B2A D93F 0F93

[Jeremy Katz]

On Thu, 2007-09-13 at 10:50 +0200, Bart Couvreur wrote:
>  Op donderdag 13-09-2007 om 12:34 uur [tijdzone +0530], schreef
>  Ankitkumar Rameshchandra Patel:
>  > Dimitris Glezos લખ્યું: 
>  > > Στις 12-09-2007, ημέρα Τετ, και ώρα 18:14 -0300, ο/η Domingo Becker
>  > > έγραψε:
>  > > > Is it possible to have a cron job to merge POT changes into PO files ?
>  > > 
>  > > I think it is possible, and in fact even someone from us could do it
>  > > every now and then, right?
>  > > 
>  > > If we get to build something that can be run from one person and get it
>  > > to become a bit mature and work right, then a cron could run it too, and
>  > > Infra will be more willing to try it out than writing it from
>  > > scratch. :)
>  > >   
>  > 1. Why can't we run the existing automerge script running on
>  > i18n.redhat.com through cronjob?
>  
>  Well we could keep that one up and running, just for the modules on
>  i18n.r.c, but for the others that have moved this could become a bit
>  problematic, we would need to have a script for each VCS. The automerge
>  script works great for CVS, if it could be ported to git, hg, svn, ...
>  it would be a start. Although that would make us responsible for keeping
>  it working, keeping the weight off Infra 
While the script "works", I wouldn't go so far as to say it works great.
Especially from the point of view of a maintainer.

The automerge currently ends up doing one commit per update which is
pretty annoying from the perspective of looking at the commit history of
the project.  With CVS, it matters slightly less since revision history
is only really per-file, but with a sane SCM, having 50 changesets in a
row that are "automerge translations" is going to be a quick way to
making a lot of people unhappy.

And really, the automerges probably shouldn't even be committed.  They
are zero difference as far as the output .mo file is concerned which
means that the changes are a lot of noise as far as the source control
is concerned :/

Jeremy

[Domingo Becker]

2007/9/13, Jeremy Katz <katzj(a)redhat.com&gt;:
>  While the script "works", I wouldn't go so far as to
> say it works great.
>  Especially from the point of view of a maintainer.
>
>  The automerge currently ends up doing one commit per update which is
>  pretty annoying from the perspective of looking at the commit history of
>  the project.  With CVS, it matters slightly less since revision history
>  is only really per-file, but with a sane SCM, having 50 changesets in a
>  row that are "automerge translations" is going to be a quick way to
>  making a lot of people unhappy.
>
>  And really, the automerges probably shouldn't even be committed.  They
>  are zero difference as far as the output .mo file is concerned which
>  means that the changes are a lot of noise as far as the source control
>  is concerned :/
>  
Damn !
You're right about this.
I haven't thought about it.
As a programmer I use CVS revision history and it is important to me.

Other ideas to make transtators life easier ?

Domingo Becker (es)

[Jeremy Katz]

On Thu, 2007-09-13 at 11:21 -0300, Domingo Becker wrote:
>  2007/9/13, Jeremy Katz <katzj(a)redhat.com&gt;:
>  > While the script "works", I wouldn't go so far as to say it works
> great.
>  > Especially from the point of view of a maintainer.
>  >
>  > The automerge currently ends up doing one commit per update which is
>  > pretty annoying from the perspective of looking at the commit history of
>  > the project.  With CVS, it matters slightly less since revision history
>  > is only really per-file, but with a sane SCM, having 50 changesets in a
>  > row that are "automerge translations" is going to be a quick way to
>  > making a lot of people unhappy.
>  >
>  > And really, the automerges probably shouldn't even be committed.  They
>  > are zero difference as far as the output .mo file is concerned which
>  > means that the changes are a lot of noise as far as the source control
>  > is concerned :/
>
>  Damn !
>  You're right about this.
>  I haven't thought about it.
>  As a programmer I use CVS revision history and it is important to me.
>  
>  Other ideas to make transtators life easier ? 
I suspect that the right answer is to do a merge on fetch, perhaps with
some caching.  So when you go to transifex and get the file, it checks
to see if the pot file is newer than the po.  If so, it does a merge and
then hands you the merged po.  When the translator commits back, then,
the commit is the merged file.  If the translator is instead using the
VCS directly, then I think that doing a msgmerge probably isn't the
worse thing (and could be scripted as well)

Similarly, the status pages would need to do the same.  And to avoid the
pain of lots of repeated msgmerge'ing, you may want to keep stats cached
with an associated .pot and .po modification time.  Then you only have
to regenerate the stat if the modification times have changed

Jeremy

[Domingo Becker]

>  > Other ideas to make transtators life easier ?
>
>  I suspect that the right answer is to do a merge on fetch, perhaps with
>  some caching.  So when you go to transifex and get the file, it checks
>  to see if the pot file is newer than the po.  If so, it does a merge and
>  then hands you the merged po.  When the translator commits back, then,
>  the commit is the merged file.  If the translator is instead using the
>  VCS directly, then I think that doing a msgmerge probably isn't the
>  worse thing (and could be scripted as well)
>  
Why not get a process launched when transifex gets a pot update?
If transifex gets a pot update, then would look at the existing po
files and updates them.
We would be taking the merge to the maintainer side.
Then the status page would show the new untranslated strings.

>  Similarly, the status pages would need to do the same.  And to avoid
> the
>  pain of lots of repeated msgmerge'ing, you may want to keep stats cached
>  with an associated .pot and .po modification time.  Then you only have
>  to regenerate the stat if the modification times have changed
>  
Domingo Becker