FIltering out languages via kickstart

[Sayamindu Dasgupta]

Hello,

OLPC's earlier build system (based on Pilgrim) has the option to
specify a list of languages, and translations for all languages other
than the specified ones would be filtered out. Is there anything
equivalent in Kickstart ? I ask this because I have a patch for glibc
(based on a patch used in Ubuntu) which lets one specify (via a
configuration file) an alternate locale directory for updated
translations which can be installed later on (at the site of
deployment, for example). However, the current patch in its current
form will not work if there are pre-existing translations in
/usr/share/locale - so I was wondering if shipping a build without any
preexisting translation, and then using the customization key to
install the most up to date translations may be viable.

Thank you,
Sayamindu


-- 
Sayamindu Dasgupta
[http://sayamindu.randomink.org/ramblings]

[Daniel Drake]

On Thu, 2009-07-09 at 14:54 +0530, Sayamindu Dasgupta wrote:
>  Hello,
>  
>  OLPC's earlier build system (based on Pilgrim) has the option to
>  specify a list of languages, and translations for all languages other
>  than the specified ones would be filtered out. Is there anything
>  equivalent in Kickstart ? I ask this because I have a patch for glibc
>  (based on a patch used in Ubuntu) which lets one specify (via a
>  configuration file) an alternate locale directory for updated
>  translations which can be installed later on (at the site of
>  deployment, for example). However, the current patch in its current
>  form will not work if there are pre-existing translations in
>  /usr/share/locale - so I was wondering if shipping a build without any
>  preexisting translation, and then using the customization key to
>  install the most up to date translations may be viable. 
Yes, we already specify which locales/languages we support:

%packages --excludedocs --instLangs
en:es:ar:pl:pt_BR:pt:it:fr:ht:el:mn:mr_IN:th:am_ET:km_KH:ne_NP:ur_PK:rw:ja:de:tr:te:ps:fa_AF:si

but I don't particularly like the idea of excluding them all and then
forcing every deployment to run the customization stick on every laptop.
We need to do better than that.

Daniel

[Sayamindu Dasgupta]

On Thu, Jul 9, 2009 at 3:45 PM, Daniel Drake<dsd(a)laptop.org> wrote:
>  On Thu, 2009-07-09 at 14:54 +0530, Sayamindu Dasgupta wrote:
> > Hello,
> >
> > OLPC's earlier build system (based on Pilgrim) has the option to
> > specify a list of languages, and translations for all languages other
> > than the specified ones would be filtered out. Is there anything
> > equivalent in Kickstart ? I ask this because I have a patch for glibc
> > (based on a patch used in Ubuntu) which lets one specify (via a
> > configuration file) an alternate locale directory for updated
> > translations which can be installed later on (at the site of
> > deployment, for example). However, the current patch in its current
> > form will not work if there are pre-existing translations in
> > /usr/share/locale - so I was wondering if shipping a build without any
> > preexisting translation, and then using the customization key to
> > install the most up to date translations may be viable.
>
>  Yes, we already specify which locales/languages we support:
>
>  %packages --excludedocs --instLangs
> en:es:ar:pl:pt_BR:pt:it:fr:ht:el:mn:mr_IN:th:am_ET:km_KH:ne_NP:ur_PK:rw:ja:de:tr:te:ps:fa_AF:si
>  

Superb...

>  but I don't particularly like the idea of excluding them all and
> then
>  forcing every deployment to run the customization stick on every laptop.
>  We need to do better than that.
>  
Well, we can have an initial set of language packs pre-installed
(taking the list from --instLangs). I am trying to set up a repository
with RPMs generated from Pootle (similar to what we have today, but
instead of self extracting archives, the packs will be RPMs). Does
that sound like a viable option ?
Thanks,
Sayamindu


>  Daniel
>
>
>  


-- 
Sayamindu Dasgupta
[http://sayamindu.randomink.org/ramblings]

[Daniel Drake]

On Thu, 2009-07-09 at 16:23 +0530, Sayamindu Dasgupta wrote:
>  Well, we can have an initial set of language packs pre-installed
>  (taking the list from --instLangs). I am trying to set up a repository
>  with RPMs generated from Pootle (similar to what we have today, but
>  instead of self extracting archives, the packs will be RPMs). Does
>  that sound like a viable option ? 
I can still see some problems...

You're working towards having these installable from a customization
stick, right?

On the customization stick, we would then want some way of
differentiating language pack RPMs from other RPM packages - at least I
don't think we want to give the ability to install RPMs in that fashion
where we haven't before.

Also, do these RPMs write outside of /home? If so, you break the fast,
incremental olpc-update method.

But one advantage of using RPMs is that you could push the following
process to deployments:
 1. install F11 on an x86 computer
 2. run some simple commands to install livecd-creator and download our
spec file
 3. modify the spec file, adding in your language pack RPM
 4. run a command to make a build

and then they have their own *clean* build with their own language pack
as a result.

One disadvantage is that deployments (with secured laptops) that are not
able to sign their own builds would be excluded..but this has always
been the case. And I think the OLPC position at the moment is "big
deployments should sign their own builds; small deployments should have
security disabled"

Might be worth a larger discussion with Chris and Ed.

Daniel

[Sayamindu Dasgupta]

CC += devel

On Thu, Jul 9, 2009 at 5:38 PM, Daniel Drake<dsd(a)laptop.org> wrote:
>  On Thu, 2009-07-09 at 16:23 +0530, Sayamindu Dasgupta wrote:
> > Well, we can have an initial set of language packs pre-installed
> > (taking the list from --instLangs). I am trying to set up a repository
> > with RPMs generated from Pootle (similar to what we have today, but
> > instead of self extracting archives, the packs will be RPMs). Does
> > that sound like a viable option ?
>
>  I can still see some problems...
>
>  You're working towards having these installable from a customization
>  stick, right?
>
>  On the customization stick, we would then want some way of
>  differentiating language pack RPMs from other RPM packages - at least I
>  don't think we want to give the ability to install RPMs in that fashion
>  where we haven't before.
>  
I'm not sure how we can differentiate RPMs - can we use a different
signing key, to sign the RPMs, for instance ?

>  Also, do these RPMs write outside of /home? If so, you break the
> fast,
>  incremental olpc-update method.
>  
Hmm - I do agree that this breaks upgrade.

>  But one advantage of using RPMs is that you could push the following
>  process to deployments:
>   1. install F11 on an x86 computer
>   2. run some simple commands to install livecd-creator and download our
>  spec file
>   3. modify the spec file, adding in your language pack RPM
>   4. run a command to make a build
>
>  and then they have their own *clean* build with their own language pack
>  as a result.
>
>  One disadvantage is that deployments (with secured laptops) that are not
>  able to sign their own builds would be excluded..but this has always
>  been the case. And I think the OLPC position at the moment is "big
>  deployments should sign their own builds; small deployments should have
>  security disabled"
>
>  Might be worth a larger discussion with Chris and Ed.
>  
> From what it seems (except for the advantages you have described) - it
may make sense to use the older system of zip files (or have, even
translation bundles) and install it under /home/olpc (that can be
easily done, since the patch allows to specify the directory, instead
of hardcoding it, as in Ubuntu). This will

a) Make translations persist across updates
b) Require minimal changes to the existing customization key mechanism
c) Require minimal changes to the existing language pack building mechanism
d) Let users modify/enhance the translations themselves, via a Sugar activity


Of course, we would miss the bells and whistle that come with RPMs,
but I think this is better than overwriting and moving around existing
MO files (which is what we do now).

Thanks,
Sayamindu


-- 
Sayamindu Dasgupta
[http://sayamindu.randomink.org/ramblings]

[Alexander Boström]

I'm completely ignorant regarding how translations work even in plain
Fedora and even more so for the OLPC systems, but pray tell, why can't
these translations be shipped as regular Fedora updates?

/abo

[Alexander Boström]

I'm completely ignorant regarding how translations work even in plain
Fedora and even more so for the OLPC systems, but pray tell, why can't
these translations be shipped as regular Fedora updates?

/abo

[Daniel Drake]

On Fri, 2009-07-10 at 21:58 +0200, Alexander Boström wrote:
>  I'm completely ignorant regarding how translations work even in
> plain
>  Fedora and even more so for the OLPC systems, but pray tell, why can't
>  these translations be shipped as regular Fedora updates? 
Because we have no way of distributing such updates that is suitable for
deployments.

Also, for various reasons, many deployments run software that is no
longer maintained. All of the XO deployments run sugar-0.82 on Fedora 9
but nobody is doing releases of that any more.

OLPC has historically done releases every 6 months, and usually as it
happens, the deployments receive the releases before creating a
sufficiently sized translator team, so the translations come later.

And in the past, only OLPC could make and sign builds. Now deployments
can do this too, but it remains to be seen how realistic this is in the
field. Unfortunately, a lot of deployments do not have the resources or
know-how in order to do this.

Daniel

[Alexander Boström]

fre 2009-07-10 klockan 21:08 +0100 skrev Daniel Drake:
>  
>  OLPC has historically done releases every 6 months, and usually as it
>  happens, the deployments receive the releases before creating a
>  sufficiently sized translator team, so the translations come later.
>  
>  And in the past, only OLPC could make and sign builds. Now deployments
>  can do this too, but it remains to be seen how realistic this is in the
>  field. Unfortunately, a lot of deployments do not have the resources or
>  know-how in order to do this. 
I see, thanks for the explanation.

I wish I could suggest a pretty solution, but I know it's not that
simple. :)

/abo

[Sayamindu Dasgupta]

Hi,

2009/7/11 Alexander Boström <abo(a)root.snowtree.se&gt;:
>  I'm completely ignorant regarding how translations work even in
> plain
>  Fedora and even more so for the OLPC systems, but pray tell, why can't
>  these translations be shipped as regular Fedora updates? 
There are quite a few reasons. For example, our official, stable build
(version 8.2.1) is based on Fedora 9 (which has been officially
end-of-lifed recently), and runs Sugar 0.82, which has not seen any
package upgrade in at least the 6 months (I'm not blaming the Sugar
devs here, they are resource-starved, and need to prioritize
accordingly). I don't see the official build changing before the last
quarter of this year, and I know for sure, that at least one large
scale pilot (>1000 machines) is going to be deployed for the first
time a certain region (translations for at least one of the languages
to be enabled in that pilot did not exist a few months back). Doing
new package releases is not feasible in this scenario.

Even when we have an up to date system (eg F11 based), translation is
usually typically done in many cases via a set of translation sprints
at the very last moment. Submitting those translations upstream, then
chasing down each and every package maintainer (upstream as well as
distro) to do new releases within a very short timeframe is not
something that is realistically possible.
Hence the need for decoupling our translation process from the release
and packaging process.

Thanks,
Sayamindu



-- 
Sayamindu Dasgupta
[http://sayamindu.randomink.org/ramblings]

[Michael Stone]

Sayamindu,

I like your answer but I think that it leaves some important goals unstated --
most notably, click2trans, horizontal distribution of translations, and
translation undo.

See http://cscott.net/Publications/OLPC/fudcon-i18n.pdf and
http://lists.laptop.org/pipermail/devel/2008-June/015838.html for the writeups and
http://dev.laptop.org/git/users/cscott/click2trans for the prototype code.

Thoughts?

Regards,

Michael

[Sayamindu Dasgupta]

Hi Michael,

On Sat, Jul 11, 2009 at 7:29 AM, Michael Stone<michael(a)laptop.org> wrote:
>  Sayamindu,
>
>  I like your answer but I think that it leaves some important goals unstated
>  --
>  most notably, click2trans, horizontal distribution of translations, and
>  translation undo.
>
>  See http://cscott.net/Publications/OLPC/fudcon-i18n.pdf and
>  http://lists.laptop.org/pipermail/devel/2008-June/015838.html for the
>  writeups and
>  http://dev.laptop.org/git/users/cscott/click2trans for the prototype code.
>
>  Thoughts?
>  
One of the main requirements of the features outlined in Scott's
presentation (especially click2trans) is the ability to have
translations which are editable by the non-root user. I think storing
translations in $HOME addresses that.

The ability to edit will definitely necessitate a "Undo" function, and
the first step towards that would be store a copy of the original
translation pack somewhere locally (version control is another
possibility, but I would probably try to avoid more than what I can
chew at the moment :-).
Another very important aspect is to try to ensure the validity of
individual messages as they get translated (eg: making sure that "%s
foo" is translated as "%s bar", and not simply as "bar" or
"%z bar").
This validation can be done by libgettextpo, and I have been slowly
working on python bindings for this
(http://code.google.com/p/pygettextpo/).

Regarding distribution of translations, apart from click2trans, we can
also have an activity like Develop which would enable users to
translate activities, as well as share translations and translation
memories. In this regard, Google's recently announced translator
toolkit might provide some inspirations :-)
(http://googleblog.blogspot.com/2009/06/translating-worlds-information-wit...)


Thanks,
Sayamindu


-- 
Sayamindu Dasgupta
[http://sayamindu.randomink.org/ramblings]